How to do fuzz testing on Chrome OS

The target audience for this document are developers in Chrome OS who work on packages. It explains how to write a fuzz test target for a package, which will then be automatically picked up and run regularly by ClusterFuzz. It assumes some basic familiarity with the Chrome OS development environment.

What is fuzz testing?

Fuzz testing (or “fuzzing”) is the process of testing an application by feeding invalid, malformed, or malicious inputs to a target program in an attempt to crash it. It is particularly useful for testing APIs because it can play the part of an attacker sending malicious input to the API.

The input data is created by randomly mutating other inputs. Sometimes the initial inputs are provided by the user in what is known as a seed corpus, otherwise the fuzzer will generate inputs from scratch. Coverage-guided fuzz testing is an extension of fuzz testing that explicitly attempts to increase code coverage when it generates new tests, adding the new tests to its test corpus when it finds tests that increase coverage, i.e. when a new test executes a new code path in the target program.

LLVM has a built-in fuzzing engine, libFuzzer, that is provided as part of LLVM's sanitizer instrumentation. Google has a cross-platform fuzzing infrastructure, ClusterFuzz, that works with libFuzzer and aids developers by providing an end-to-end pipeline that automatically picks up new (and existing) fuzz targets, runs fuzz testing on these targets, reports bugs (assigning them to appropriate owners), and even verifies fixes. Chromium on Linux and MacOS has been using libFuzzer and ClusterFuzz for years, and they have proved very useful, finding thousands of security bugs and non-security bugs.

A fuzz test target is nothing more than a piece of code that can take a chunk of bytes and do something with them by calling the code to be tested. For the remainder of this document we will use the word fuzzer to refer to a fuzz test target that gets linked against libFuzzer to produce an executable that exercises the code under test. This is what ClusterFuzz will run.

Chrome OS has tooling to support writing fuzzers for Chrome OS packages. This document will walk you through the steps necessary to get your fuzzers up and running on ClusterFuzz.

Quickstart guide

This section of the document describes the basic steps needed to write a fuzzer in Chrome OS and have ClusterFuzz pick it up. For more detailed instructions, see the Detailed instructions section.

If you are working on a platform package (one that is in the platform or platform2 source directory and whose ebuild inherits from the platform eclass), see the Build a fuzzer for a platform package section.

Steps to create a new fuzzer in Chrome OS

  • Write a new test program in your package, whose name ends in _fuzzer (it would be a good idea, but not required, to prepend your package name to your fuzzer as well, e.g. ‘cryptohome__fuzzer’) and which defines a function LLVMFuzzerTestOneInput with the following signature:

    #include <stddef.h>
    #include <stdint.h>
    
    extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
        <your test code goes here>
        return 0;
    }
    

    Ensure LLVMFuzzerTestOneInput calls the function you want to fuzz.

  • Update the build system for your package to build your *_fuzzer binary.

  • Update your package's ebuild file:

    1. Add fuzzer to the IUSE flags list.
    2. Build you new fuzzer (conditioned on use fuzzer), with the appropriate flags:
      1. Inherit cros-fuzzer and cros-sanitizers eclasses
      2. Set up flags: call sanitizers-setup-env in src_configure
      3. USE flags: fuzzer
    3. Install your binary in /usr/libexec/fuzzers/
    4. Build the libraries your fuzzer depends on with the appropriate -fsanitize flags (optional).
  • Build and test your new fuzzer locally. Commit your changes.

  • Add the package dependency to the chromium-os-fuzzers ebuild, and commit the change.

That's it! The continuously running fuzzer builder on the Chrome OS waterfall will automatically detect your new fuzzer, build it and upload it to ClusterFuzz, which will start running it. For more details on what you can do with ClusterFuzz, see the Using ClusterFuzz section.

Detailed instructions

This section goes over the steps mentioned in the Quickstart guide in more detail. If you're working on a platform package, read Build a fuzzer for a platform package. Otherwise, read Build a fuzzer for any other package.

Build a fuzzer

It might be counterintuitive to start by building a fuzzer before one is even written, but the process of actually writing the fuzzer will be easier with a fast compile-test cycle, for which we need to be able to compile the fuzzer.

Start with a dummy fuzzer:

// Copyright 2018 The Chromium OS Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#include <stddef.h>
#include <stdint.h>

class Environment {
 public:
  Environment() {
    // Set-up code.
  }
};

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
  static Environment env;
  // Fuzzing code. Empty for now.
  return 0;
}

Build a fuzzer for a platform package

These steps assume that you have at least a dummy fuzzer to compile. Check out the previous section for an example.

  1. Update the build system for your package to build your *_fuzzer binary.

    • For packages that are built with GYP files, update your GYP file to build the fuzzer binary:

      {
        'targets': [
          ...
        ],
        'conditions': [
          # Fuzzer target.
          ['USE_fuzzer == 1', {
            'targets': [
              {
                'target_name': 'your_fuzzer',
                'type': 'executable',
                'includes': [
                  '../common-mk/common_fuzzer.gypi',
                ],
                'dependencies': [
                  # This could be an intermediate static library target in your
                  # package.
                ],
                'variables': {
                  'deps': [
                    'libchrome-test-<(libbase_ver)',  # For FuzzedDataProvider (optional)
                    'libprotobuf-mutator', # For fuzzing with Protobufs (optional)
                  ],
                },
                'sources': [
                  'your_fuzzer.cc',
                ],
              },
            ],
          }],
        ],
      }
      

      See the midis GYP file for a complete example.

    • If your package is not built with a GYP file, then you will need to update your Makefile (or whatever build system you use), in such a way that your fuzzer binary gets built when a special flag or argument is passed to the normal build command.

      If testing this by hand (without the ebuild file changes in step 2 below), you will need to manually pass the compiler flags -fsanitize=address -fsanitize=fuzzer-no-link and the linker flags -fsanitize=address -fsanitize=fuzzer to your build. You will not need to pass these flags manually once you have updated the ebuild file.

  2. Update your package's ebuild file:

    1. Update the ebuild file to build the new binary when the fuzzer USE flag is being used. For platform packages built with GYP files, you should skip this step and go directly to the next step for installing the fuzzer binary.

      1. Find the src_compile() function in your ebuild file. If there isn't one, add one:

        src_compile() {
        }
        
      2. Add the call to actually build your fuzzer.

        Find the line in your src_compile function that actually builds your package (the command will probably look like emake or make or cmake). This is the command that is meant by original build command below. Copy the original build command and add whatever flags or arguments you need in order to make it build just your fuzzer binary (see step 1 above). Replace the original build command in the src_compile function with a conditional statement similar to the one below, so that when USE="fuzzer" is used to build the package, it will build your fuzzer binary, otherwise it will build the package normally.

        if use fuzzer ; then
             <modified build command>
        else
             <original build command>
        fi
        
    2. Install your binary in /usr/libexec/fuzzers/

      In your ebuild file, find the src_install() function. Add a statement to install your fuzzer:

      platform_fuzzer_install "${S}"/OWNERS "${OUT}"/<your_fuzzer>
      
  3. Build and test your new fuzzer locally.

    To build your new fuzzer, once you have updated the ebuild file, it should be sufficient to build it with USE="asan fuzzer".

    Note: Fuzzing using undefined behavior sanitizer (ubsan) is also supported. To use ubsan, simply replace asan with ubsan in the commands below.

    Note 2: If your package depends on chromeos-chrome, the build_packages command below can take a very long time. In such a case, it would be advisable to adjust the package dependencies to not depend on chromeos-chrome with ‘USE=fuzzer’.

    # Run build_packages to build the package and its dependencies.
    $ USE="asan fuzzer" ./build_packages --board=${BOARD} --skip_chroot_upgrade <your_package>
    # If you make more changes to your fuzzer or build, you can rebuild the package with:
    $ USE="asan fuzzer" emerge-${BOARD} <your_package>
    

    For more details on cros-workon packages, see the Developer Guide.

    You should verify that your fuzzer was built and that it was installed in /build/${BOARD}/usr/libexec/fuzzers/ (make sure the owners file was installed there as well). To run your fuzzer locally, run this command to prepare the environment and get a shell ready for fuzzing:

    $ cros_fuzz --board=${BOARD} shell
    

    Then run your fuzzer:

    (in board) # /usr/libexec/fuzzers/<your_fuzzer>
    
    Note: The fuzzer will run forever (or until it finds a bug), so you will want to halt it manually (using Ctrl-C) after a couple of minutes.

    You can read more about the cros_fuzz script in the section Using cros_fuzz. You should also verify that your package still builds correctly without USE="fuzzer".

    Once you are happy with your new fuzzer, commit your changes.

  4. Add the package dependency to the chromium-os-fuzzers ebuild. Inside your chroot:

    Edit ~/trunk/src/third_party/chromiumos-overlay/virtual/chromium-os-fuzzers/chromium-os-fuzzers-1.ebuild. In that file, find the RDEPEND list and add your package/fuzzer (you can look at the other packages there, to see how it‘s done). Don’t forget to uprev the ebuild symlink. Commit the changes and upload them for review.

  5. Optional: Verify that the amd64-generic-fuzzer builder is happy with your changes.

    Submit a tryjob outside of the chroot:

    $ cros tryjob -g 'CL1 CL2' amd64-generic-fuzzer-tryjob
    

    Replace CL1 and CL2 with the actual CL numbers.

    You should verify that your package is picked up by the builder by looking at the BuildPackages stage logs.

    The builder builds the full system with AddressSanitizer and libFuzzer instrumentation. If you do not want a particular library pulled in by your changes to be instrumented, you can add a call to filter_sanitizers in the library's ebuild file.

Build a fuzzer for any other package

  1. Update the build system for your package to build your *_fuzzer binary.

    The exact instructions here are going to vary widely, depending on your package and the build system in your package (Make, Ninja, SCons, etc.). In general, you will need to be able to invoke your normal build command, passing a special flag or argument or environment variable, so that it will build your fuzzer binary and only your fuzzer binary. This will involve updating GYP files or Makefiles or whatever other build files your package uses.

    If testing this by hand (without the ebuild file changes in step 3 below), you will need to manually pass the compiler flags -fsanitize=address -fsanitize=fuzzer-no-link and the linker flags -fsanitize=address -fsanitize=fuzzer to your build. You will not need to pass these flags manually once you have updated the ebuild file.

  2. Update your package's ebuild file:

    1. Add fuzzer to the IUSE flags list.

      In all probability your package ebuild already contains an IUSE definition. Look for a line starting IUSE="...", and add fuzzer to the list. If your file does not already contain such a line, add one near the top:

      IUSE="fuzzer"
      

      See the puffin ebuild for a good example.

    2. Update the ebuild file to build the new binary when the fuzzer USE flag is being used:

      1. Find the inherit line in your ebuild (near the top of the file). Make sure that cros-fuzzer and cros-sanitizers are in the inherit list. If your file does not have a line that starts with inherit, add one near the top (after the EAPI line and before the KEYWORDS line):

        inherit cros-fuzzer cros-sanitizers
        
      2. Find the src_configure() function in your ebuild file. If there isn't one, add one:

        src_configure() {
        }
        
      3. Add calls sanitizers-setup-env, near the top of src_configure, to set the appropriate compiler/linker flags:

        src_configure() {
            sanitizers-setup-env
            ...
        }
        
      4. Find the line in your src_compile function that actually builds your package (the command will probably look like emake or make or cmake). This is the command that is meant by original build command below. Copy the original build command and add whatever flags or arguments you need in order to make it build just your fuzzer binary (see step 1 above). Replace the original build command in the src_compile function with a conditional statement similar to the one below, so that when USE="fuzzer" is used to build the package, it will build your fuzzer binary, otherwise it will build the package normally.

        if use fuzzer ; then
            <modified build command>
        else
            <original build command>
        fi
        
    3. Install your binary in /usr/libexec/fuzzers/

      In your ebuild file, find the src_install() function. Add a fuzzer_install statement to install your fuzzer:

      fuzzer_install "${S}/OWNERS" <your_fuzzer>
      

      (The owners part above is so that ClusterFuzz knows to whom to assign the bugs generated by this fuzzer.)

  3. Build and test your new fuzzer locally.

    To build your new fuzzer, once you have updated the ebuild file, it should be sufficient to build it with USE="asan fuzzer":

    Note: Fuzzing using undefined behavior sanitizer (ubsan) is also supported. To use ubsan, simply replace asan with ubsan in the commands below.

    # Run build_packages to build the package and its dependencies.
    $ USE="asan fuzzer" ./build_packages --board=${BOARD} --skip_chroot_upgrade <your_package>
    # If you make more changes to your fuzzer or build, you can rebuild the package by:
    $ USE="asan fuzzer" emerge-${BOARD} <your_package>
    

    You should verify that your fuzzer was built and that it was installed in /build/${BOARD}/usr/libexec/fuzzers/ (make sure the owners file was installed there as well). To run your fuzzer locally, run this command to prepare the environment and get a shell ready for fuzzing:

    $ cros_fuzz --board=${BOARD} shell
    

    Then run your fuzzer:

    (in board) # /usr/libexec/fuzzers/<your_fuzzer>
    
    Note: The fuzzer will run forever (or until it finds a bug), so you will want to halt it manually (using Ctrl-C) after a couple of minutes.

    You can read more about the cros_fuzz script in the section Using cros_fuzz. You should also verify that your package still builds correctly without USE="fuzzer".

    Once you are happy with your new fuzzer, commit your changes.

  4. Add the package dependency to the chromium-os-fuzzers ebuild. Inside your chroot:

    Edit ~/trunk/src/third_party/chromiumos-overlay/virtual/chromium-os-fuzzers/chromium-os-fuzzers-1.ebuild. In that file, find the RDEPEND list and add your package/fuzzer (you can look at the other packages there, to see how it‘s done). Don’t forget to uprev the ebuild symlink. Commit the changes and upload them for review.

  5. Optional: Verify that the amd64-generic-fuzzer builder is happy with your changes.

    Submit a tryjob outside of the chroot as:

    $ cros tryjob -g 'CL1 CL2' amd64-generic-fuzzer-tryjob
    

    Replace CL1 and CL2 with the actual CL numbers.

    You should verify that your package is picked up by the builder by looking at the BuildPackages stage logs.

    The builder builds the full system with AddressSanitizer and libFuzzer instrumentation. If you do not want a particular library pulled in by your changes to be instrumented, you can add a call to filter_sanitizers in the library's ebuild file.

Write a fuzzer

Now that you can build and execute your fuzzer, you can start working on getting it to actually test things. If the function you want to test takes a chunk of bytes and a length, then you‘re done: that’s what the fuzzing scaffolding will provide in the LLVMFuzzerTestOneInput function. Just pass that to your function:

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
  // Call your function here.
  return 0;
}

Some things to keep in mind:

  • The fuzz target will be executed many times with different inputs in the same process.
  • It must tolerate any kind of input (empty, huge, malformed, etc). You can write your fuzz target to simply return 0 on certain types of malformed input. See the GURL fuzzer for such an example.
  • It must not exit() on any input.
  • It may use threads but ideally all threads should be joined at the end of the function.
  • It must be as deterministic as possible. Non-determinism (e.g. random decisions not based on the input bytes) will make fuzzing inefficient.
  • It must be fast. Avoid >= cubic complexity, logging, high memory consumption.
  • Ideally, it should not modify any global state (although that's not strict).

In particular, if you‘re fuzzing code that’s using the logging primitives from <base/logging.h>, you should disable logging:

#include "base/logging.h"

class Environment {
 public:
  Environment() {
    logging::SetMinLogLevel(logging::LOG_FATAL);  // <- DISABLE LOGGING.
  }
};

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
  static Environment env;
  // Fuzzing code.
  return 0;
}

What if my code doesn't simply consume a chunk of bytes?

Odds are that the code that you want to test doesn't just take a chunk of bytes and a length. You might want to fuzz an API where functions take integers, strings, or protocol buffers. There might be useful state to set up before hitting parsing code, or you might want to test a state machine where some calls need to happen in a certain order to avoid erroring out of functions early.

Don't despair! This is actually very common. Most fuzzers end up extracting some structure out of the random data received in order to better exercise the code under test. Below are two different tools for handling this case. One for non-protos and another for protocol buffers.

FuzzedDataProvider

FuzzedDataProvider is a class that can genating many things such as integers or strings from the fuzz target input data.

Consider the interface of the permission_broker firewall implementation:

bool AddAcceptRules(ProtocolEnum protocol,
                    uint16_t port,
                    const std::string& interface);
bool DeleteAcceptRules(ProtocolEnum protocol,
                       uint16_t port,
                       const std::string& interface);

It's not really straightforward to feed a chunk of bytes to this API. To address this, we can use code provided by libchrome: FuzzedDataProvider.

FuzzedDataProvider will consume fuzzing input and allow extracting structure out of it:

class FuzzedDataProvider {
public:
  ...
  std::string ConsumeBytes(size_t num_bytes);

  std::string ConsumeRemainingBytes();

  std::string ConsumeRandomLengthString(size_t max_length);

  uint32_t ConsumeUint32InRange(uint32_t min, uint32_t max);
  int ConsumeInt32InRange(int min, int max);

  bool ConsumeBool();

  uint8_t ConsumeUint8();

  uint16_t ConsumeUint16();

  ...
};

Using this API, we can obtain integers and strings to pass to the above API, and also use booleans to decide how to call the API:

firewall_fuzzer

// Copyright 2018 The Chromium OS Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#include <set>

#include "base/logging.h"
#include "base/test/fuzzed_data_provider.h"

#include "permission_broker/firewall.h"

...

class Environment {
 public:
  Environment() {
    logging::SetMinLogLevel(logging::LOG_FATAL);
  }
};

extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
  static Environment env;

  permission_broker::FakeFirewall fake_firewall;  // <- Mocks out system calls.
  base::FuzzedDataProvider data_provider(data, size);

  std::set<uint16_t> tcp_ports;
  std::set<uint16_t> udp_ports;

  // How many ports should we try?
  uint8_t num_ports = data_provider.ConsumeUint8();
  for (size_t i = 0; i < num_ports; i++) {
    bool is_tcp = data_provider.ConsumeBool();
    uint16_t port = data_provider.ConsumeUint16();

    if (!is_tcp && port == 0) {
      // Did we run out of data? Consume another bool to check.
      if (!data_provider.ConsumeBool())
        break;
    }

    bool do_add = true;

    if ((is_tcp && tcp_ports.count(port) == 0) ||
        (!is_tcp && udp_ports.count(port) == 0)) {
      // Port does not exist.
      // With small probability, hit the error case: delete a port that doesn't
      // exist.
      do_add = data_provider.ConsumeUint8() < 0xFF;
    } else {
      // Port exists.
      // With small probability, hit the error case: add a port that already
      // exists.
      do_add = data_provider.ConsumeUint8() == 0xFF;
    }

    if (do_add) {
      fake_firewall.AddAcceptRules(is_tcp ? permission_broker::kProtocolTcp
                                          : permission_broker::kProtocolUdp,
                                   port, "iface");
    } else {
      fake_firewall.DeleteAcceptRules(is_tcp ? permission_broker::kProtocolTcp
                                             : permission_broker::kProtocolUdp,
                                      port, "iface");
    }
  }
  return 0;
}

The fuzzer is using its input to decide what to do and which functions to call. As long as the FuzzedDataProvider object is initialized with the same input, its functions will return the same values and the results will be deterministic.

Note that in this particular case we could have chosen to also pass a fuzzed string for the interface argument. However, the check for that argument happens very early in the code, so fuzzing that argument ends up being counterproductive because it prevents the rest of the code from being reached. When writing your fuzzer, take this into account. If random input is unlikely to make it past an initial check consider using a boolean value to decide whether or not to test that argument, or forgo testing that argument altogether.

Running the fuzzer locally will continuously print a measure of coverage that can be used (in relative terms) to understand whether skipping an argument allows the fuzzer to reach more code:

#3268 NEW cov: 218

The cov value will increase (unsurprisingly) with increased coverage. In the current example, avoiding random strings for the interface argument significantly increased coverage because both API functions were no longer erroring out early.

Libprotobuf-mutator

For cases where your code accepts a protocol buffer as input, there is a library libprotobuf-mutator that generates mutated protocol buffers for fuzz targets. First identify the function(s) you want to call in your fuzz target. If there is only one input proto, writing a fuzz target is really easy and you can skip the next paragraph.

Whenever you need more than one input proto or want to invoke a function multiple times, you can make a new proto message that includes all the input protos. Repeated fields can be used in cases where more than one action needs to be performed multiple times to get good code coverage from a fuzz target.

// Copyright 2018 The Chromium OS Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

#include <libprotobuf-mutator/src/libfuzzer/libfuzzer_macro.h>

#include "my_code.h"
#include "my_proto.pb.h"

DEFINE_PROTO_FUZZER(const my_package::MyProto& input) {
  my_code::MyFunction(input);
}

Building this fuzz target requires linking against libprotobuf-mutator. You can add 'libprotobuf-mutator' under the deps section in the GYP/GN build file for platform projects (GYP Example).

Lastly, for both platform and non-platform projects fuzzer? ( dev-libs/libprotobuf-mutator ) should be added to the dependency list in the ebuild. This makes sure the dev-libs/libprotobuf-mutator package is installed on the target and the headers are available at compile time.

For additional tips you can take a look at this chromium document for writing a grammar-based-fuzzer with libprotobuf-mutator.

Getting help with modifying ebuild files

Some ebuild files are more complex or confusing than others. There are several links in the References section of this document that can help you with understanding/editing your ebuild file. If you are still having difficulties editing your ebuild file and need more help, please file a bug in the Chromium issue tracker, assign it to the Tools>ChromeOS-Toolchain component, and send an email to chromeos-fuzzing@google.com. We will try to help you figure this out.

Using ClusterFuzz

As already mentioned, ClusterFuzz will pick up any fuzzer written using the above steps, run the fuzzer, and file bugs for any crashes found. ClusterFuzz runs fuzzers as soon as the fuzzer builder completes a build and uploads it to the Google Cloud Storage bucket: gs://chromeos-fuzzing-artifacts/.

ClusterFuzz has many features such as statistics reporting that you may find useful. Below are links to some of the more important ones:

  • Fuzzer statistics - Statistics from fuzzer runs, updated daily. Ignore the columns edge_cov, func_cov, and cov_report as these are not supported for Chrome OS. Graphs of stats can viewed by changing the “Group by” drop down to “Time” and specifying the fuzzer you are interested in, rather than “libFuzzer”.
  • Crash statistics - Statistics on recent crashes.
  • Fuzzer logs - Cloud storage bucket containing logs output by your fuzzer each time ClusterFuzz runs it. This is usually a good place to debug issues with your fuzzer. The URI for this bucket is: gs://chromeos-libfuzzer-logs/.
  • Fuzzer corpus - Cloud storage bucket containing test cases produced by the fuzzer that libFuzzer has deemed “interesting” (meaning it causes unique program behavior). The URI for this bucket is: gs://chromeos-corpus/libfuzzer/.

Note that you may need to authenticate to access the corpus and logs buckets using gsutil. See the documentation on Configuring Authentication for how to do this.

Using cros_fuzz

cros_fuzz is a script we have provided to use within the Chrome OS SDK chroot. Its purpose is to make fuzzer development easier by automating important tasks. These include:

cros_fuzz is used like so:

$ cros_fuzz --board=${BOARD} <command> <command arguments>

You can get detailed help information on each command by using the command argument: --help. We explain the more important commands in each subsection.

Tip: Set your .default_board to avoid the need to specify --board every time you run cros_fuzz

Tip: cros_fuzz will print every shell command it runs if you set the log-level to debug (“--log-level debug”).

Preparing the environment for fuzzing

This section describes the shell, cleanup, setup and commands, in that order. To run a fuzzer, you should not just chroot into the board where it was built and run it since libFuzzer needs certain special files that are present on most systems but not on the board. You also should also not run the fuzzer outside of the board because that environment will contain devices and other things that will not be available on ClusterFuzz. Instead you should run the shell command from cros_fuzz. shell will prepare the board for fuzzing and then chroot into the board giving you a shell. It is simple to use:

$ cros_fuzz --board=${BOARD} shell
(board chroot) # /usr/libexec/fuzzers/<your_fuzzer>

The changes made by the shell command to the board can be mostly undone using the cleanup command.

Internally, the shell command does the same thing as the setup command and then gives you a shell with the ASAN_OPTIONS env variable set to log_path=stderr, which is needed to view stack traces when the fuzzer finds a bug. You can use the setup command to run your fuzzer using this sequence of these bash commands:

$ cros_fuzz --board=${BOARD} setup
$ sudo chroot /build/${BOARD}
(board chroot) # ASAN_OPTIONS="log_path=stderr" /usr/libexec/fuzzers/<your_fuzzer>

Getting a coverage report for your fuzzer

This section describes the coverage command. Coverage reports are a great way to tell what code your fuzzer is actually testing. cros_fuzz provides an easy way to get a coverage report using the coverage command.

Before we explain how to use coverage, there are two things you should understand about the coverage report process.

  • First is the importance of a corpus in generating a coverage report. As explained in the What is fuzz testing? section, a testcase is added to the corpus if it increases the fuzzer‘s coverage of the target code. Thus, if everything works correctly, one can use a fuzzer’s corpus from fuzzing to quickly find the code covered by the fuzzer.

  • The second thing to understand is how we generate the report. The report is generated by clang's source based coverage. This is not the same instrumentation libFuzzer uses to get coverage data, but we use it because the reports it generates are much better than the ones libFuzzer generates. Because we need a new kind of instrumentation to generate the report we must do a new kind of build.

The coverage command on the most simple level: runs a fuzzer, collects coverage info from the run, and then generates an HTML coverage report for you to view. It can also automate many things you would want to do in this process including doing a build with the source based coverage instrumentation and using a fuzzer's corpus from disk or from ClusterFuzz (if it is already run on ClusterFuzz).

These are a list of the options the coverage command accepts, and an explanation of each of them.

  • --fuzzer <your_fuzzer>: The name of the fuzzer to generate a coverage report for. This is the only mandatory option.

  • --package <your_package>: The name of the fuzzer's package. Not mandatory, but you probably should use this unless you know what you are doing. If this option is used, a coverage build of the package will be done. A coverage build uses the USE flags fuzzer asan coverage. Unless the current build of the fuzzer on the board is a coverage build, the coverage command will fail.

  • --corpus <path_to_corpus>: The path to a corpus we should use when generating the coverage report. If this option is used cros_fuzz will copy the corpus into the board (if needed) and the fuzzer will run every testcase in the corpus once and will not actually do any fuzzing. This option is mutually exclusive with --download.

  • --download: Whether to download the corpus from ClusterFuzz. This will fail if the fuzzer has not yet been run on ClusterFuzz. It will also fail if you are not logged into gsutil using your @google.com account (it will instruct you on how to do this if that happens). If the download succeeds, then pretty much the same thing happens if the --corpus command were used, the fuzzer will run every testcase in the corpus but not try to find any new testcases (fuzzing). This option is mutually exclusive with --corpus.

  • --fuzz-args <libfuzzer_options>: libFuzzer options to pass to the fuzzer. Each separated by a space. Note that using this option will probably confuse the script's argument parser unless you use it like this: --fuzz-args="-max_total_time=60"

Once this command has completed, it will print a file:// URI of an HTML coverage report that you can view in your browser. If neither the --corpus or --download options is used, there needs to be some kind of arbitrary limit imposed on fuzzing so it doesn‘t continue forever. If you don’t specify one, the script will specify a timeout of 30 seconds. You can specify a limit by passing one of the -max_total_time or -runs libFuzzer options in --fuzz-args.

If you want access to some of the data used by this command, it is probably stored in the directory: /build/${BOARD}/tmp/fuzz/ but we make no guarantees about this.

Reproducing crashes from ClusterFuzz

This section explains how to reproduce bugs found by ClusterFuzz. No knowledge of fuzzing is assumed and it summarizes info from elsewhere in this document.

To reproduce crashes you should use the reproduce command from cros_fuzz, here's a guide on how to use it to reproduce crashes reported by ClusterFuzz.

  1. Download the reproducer testcase from the link on the bug report, and copy it to the board.

    $ cp ~/Downloads/<testcase_name> /path/to/chromiumos-checkout/chroot/build/${BOARD}/tmp/
    
  2. Identify the package, build type, and the name of the fuzzer. The name of the fuzzer will be on the “Fuzzer:” line. It will begin with “libFuzzer_chromeos_” but this prefix is added by ClusterFuzz and isn't actually part of the binary name. You can determine the build type from the “Sanitizer” line in the bug report. It will either be “asan” or “ubsan”.

  3. Run the reproduce command of cros_fuzz from within the Chrome OS SDK chroot like so:

    (chromeos_sdk) $
    cros_fuzz --board=${BOARD} reproduce --fuzzer <your_fuzzer> --testcase /path/to/testcase/<testcase_name> \
              --package <package_of_fuzzer> --build-type <build_type>
    

You should see a stack trace after running the reproduce command.

For more advanced uses, we will explain the details for the reproduce command. Below is an explanation of what the options to reproduce mean, note that no option is mandatory unless explicitly specified.

  • --testcase: The path to the testcase to run the fuzzer on. Mandatory. Note that the path must be in the chroot, but does not need to be on the board, it will be copied if necessary.

  • --fuzzer: The fuzzer to run the testcase on. Mandatory.

  • --package: The package of the fuzzer to build. If this option is provided then one must also provide the --build_type option.

  • --build_type: The type of build we want to do. This can either be asan, ubsan, or coverage. Note that coverage is not actually used on ClusterFuzz. This option may only be used if the --package option is used.

Note that once the build needed is in a root it doesn‘t need to be done again, so it isn’t necessary to use --package unless another build clobbered the one we need if we have already done the build we need.

To summarize: the reproduce command does a build of the package (if needed), copies the testcase into the board, prepares the board for running the fuzzer, and then runs the fuzzer on the testcase.

Improving fuzzer effectiveness

Below are some optional things you can do to improve the effectiveness of your fuzzer. Note that these instructions primarily explain how to make these improvements for ClusterFuzz to use. Using them locally is explained at the end of each section.

Adding a seed corpus

A seed corpus is a set of interesting test cases (input files) that are meaningful for your fuzz target and provide a good starting point for libFuzzer to mutate. For example, if you are fuzzing a PNG parser, a good seed corpus would be a small number of small PNG files from the parser's test suite. A seed corpus is helpful in this case since it is much easier for libFuzzer to produce interesting PNGs if it starts from valid ones than if it starts from nothing (where it would need to learn to produce PNG-like inputs over many thousands of executions).

A small file that has the same coverage as a large file is better since it is more likely that the fuzzer will mutate important bytes rather than unimportant ones (eg: comments in a C file). Your seed corpus should contain no redundant (ie: causing no unique program behavior) test cases. You can minimize the size of your seed corpus (but not individual testcases) by running these commands:

$ mkdir MINIMIZED_SEED_CORPUS
$ ./<your_fuzzer> -merge MINIMIZED_SEED_CORPUS <path_to_your_fuzzers_corpus>

Once you have decided what files you will include in your seed corpus: archive them in a zip file called <your_fuzzer>_seed_corpus.zip. Then modify the ebuild to install it to the correct location, by passing --seed_corpus "${S}"/policy/testdata/<your_fuzzer>_seed_corpus.zip to platform_fuzzer_install like so:

platform_fuzzer_install "${S}"/OWNERS "${OUT}"/<your_fuzzer> \
    --seed_corpus "${S}"/path/to/<your_fuzzer>_seed_corpus.zip

The ebuild for preg_parser_fuzzer can be used as an example for installing a seed corpus.

DO NOT put an entire corpus you got from fuzzing locally in a seed corpus. There are three reasons for this:

  • A seed corpus gets unpacked by ClusterFuzz each time your fuzzer is run. So unlike testcases in the target‘s regular corpus, seed corpus files don’t get removed during minimization when they are found to no longer be useful (such as when the fuzzer has found another test case that covers a superset of its coverage). This reason is also why it is important to minimize your seed corpus.
  • These test cases are probably not very helpful to ClusterFuzz. These test cases are by definition findable by a fuzzer and ClusterFuzz will likely have an easier time finding them than you.
  • Real inputs probably lead to better coverage than inputs generated by fuzzing from scratch.

Instead of putting your local corpus in the seed corpus, you can upload it directly to the target's corpus (after the fuzzer has already run for the first time, which will probably happen within a day of commiting it) by running this command:

$ gsutil -m cp <path_to_your_fuzzers_corpus>/* gs://chromeos-corpus/libfuzzer/chromeos_<your_fuzzer_name>

Files uploaded to this directory will get pruned when no longer useful. Your Chrome OS SDK chroot comes with a copy of gsutil, or you can install it on your host by following the instructions to set up the gsutil tool. Note that gsutil must be connected to your @google.com account to have access to the corpus.

To use a corpus in local fuzzing, pass the directory to your fuzzer, like so:

$ ./<your_fuzzer> <corpus_directory>

Adding a dictionary

A dictionary is a file containing tokens that are useful for fuzzing a particular format. For example if we are fuzzing a C compiler, useful tokens would be things like void, int, if, break etc. Dictionaries are passed to libFuzzer using the -dict=$DICTIONARY_FILE argument. In a dictionary, each token should be quoted and appear on a line by itself. Here is an example from the libFuzzer documentation:

# Lines starting with '#' and empty lines are ignored.

# Adds "blah" (w/o quotes) to the dictionary.
"blah"
# Use \\ for backslash and \" for quotes.
"\"ac\\dc\""
# Use \xAB for hex values
"\xF7\xF8"

Once you have decided the content of your dictionary, add it to a file called <your_fuzzer>.dict and then edit your ebuild to install the dictionary, like so:

platform_fuzzer_install "${S}"/OWNERS "${OUT}"/<your_fuzzer> \
    --dict "${S}"/path/to/<your_fuzzer>.dict

The ebuild for preg_parser_fuzzer can be used as an example for installing a dictionary.

There are many dictionaries in the Chromium code base, you may be able to reuse one if your fuzzer's format is also fuzzed in Chrome.

To use a dictionary in local fuzzing, use the -dict= option, like so:

$ ./<your_fuzzer> -dict=/path/to/your/dictionary

Adding an options file

You can set options that will be passed to libFuzzer. You can read about libFuzzer options in the libFuzzer docs. The options below will probably be most useful to you.

  • -ascii_only which instructs libFuzzer to only feed ascii characters to your target.
  • -max_len which instructs libFuzzer not to feed inputs larger than a certain size to your target.

Note that your fuzzer can not depend on options being passed to it, because ClusterFuzz sometimes intentionally omits options. This means that if your target cannot accept inputs beyond a certain length, you need to handle this in the target function (LLVMFuzzerTestOneInput), rather than depending on max_len, which is an optimization. Your options file should use this format:

[libfuzzer]
ascii_only = 1
max_len = 1024

Your options file should be named: <your_fuzzer>.options You can install the options file by editing your ebuild to use --options like so:

platform_fuzzer_install "${S}"/OWNERS "${OUT}"/<your_fuzzer> \
    --options "${S}"/path/to/<your_fuzzer>.options

Note that you do not need to change an options file when adding a dictionary. ClusterFuzz automatically passes the dictionary, if named correctly, to libFuzzer.

FAQ

Will my fuzzer get past checks or conditional statements?

It depends on the check, but the answer is probably yes. Most checks (such as string comparisons against a magic string) are easy for the fuzzer to get past. Other less common operations, such as verification of a checksum, hash, or signature are quite hard for the fuzzer to get past. You should first verify (for example, by adding print statements) that the fuzzer is not passing a certain check before worrying about it. If the checks are not among the difficult ones listed above or similar to them, adding a dictionary or a seed corpus can help. It is only recommend to try the options below if these suggestions does not work and you are confident that the fuzzer is having trouble passing some check.

  • The first option is to #ifdef the check so that it isn‘t done in fuzzing builds. This generally works in cases where it is easy to craft an input by hand that causes the targeted code, with the check, to behave the same as a fuzzer-generated input without the check. A good example of this are the checksums on each chunk section in a PNG. libFuzzer isn’t smart enough to write the correct checksum each time it mutates the chunk, but correcting the chunk by hand is trivial and a chunk with the correct causes the same behavior (except for the check itself) in the program with or without the check.

  • The second option is to process the input from libFuzzer in LLVMFuzzerTestOneInput to make it acceptable to your targeted code. For example, if your target code accepts input with the following format:

    [md5sum(Body)][Body]
    

    You can write LLVMFuzzerTestOneInput so that it passes the hash check:

    extern "C" int LLVMFuzzerTestOneInput(uint8t_t* data, size_t size) {
        std::string processed_data = md5sum(data, size) + std::string(data);
        TargetedFunction(processed_data.data(), processed_data.size());
        return 0;
    }
    
  • The options above may not be enough for a complicated format. There are other tools such as libprotobuf-mutator that allow you to specify a format for libFuzzer to mutate, which you then convert into raw bytes. It is not yet supported in Chrome OS (follow issue 853017 for updates).

Getting help/Asking questions

You can send an email to chromeos-fuzzing@google.com if you get stuck, or to ask questions.

See also

References

Setting up Fuzzing for Chrome OS

Continuous in-process fuzzing for Chrome OS targets

libFuzzer - a library for coverage-guided fuzz testing.

“Fault injection through unexpected input data (aka Fuzz Testing)”

Getting Started with libFuzzer in Chromium

Gentoo Cheat Sheet

Gentoo Ebuild Variables Guide

Gentoo Ebuild Install Functions Reference

Gentoo Ebuild Writing Developers Manual

Useful go links

go/ClusterFuzz

go/libfuzzer-Chrome

go/fuzzing

go/whyfuzz