Reproducing libFuzzer and AFL crashes

libFuzzer and AFL crashes can be reproduced easily with the ClusterFuzz Reproduce Tool. However, if you are unable to use the tool (e.g. unsupported platform, some other tool issue), you can still reproduce the crash manually using the steps below:

Requirements: For Windows, you must convert the forward slashes (/) to backslashes (\) in the commands below and use set command instead of export to set the environment variable (step 4). Note that these commands are intended to be used with cmd.exe, not PowerShell. Also, you may find these tips on how to debug an ASAN instrumented binary helpful.

Crashes reported as Reproducible

The majority of the bugs reported by ClusterFuzz have Reproducible label. That means there is a testcase that can be used to reliably reproduce the crash.

  1. Download the testcase from ClusterFuzz. If you are CCed on an issue filed by ClusterFuzz, a link to it is next to “Reproducer testcase” in the bug description.

    For the rest of this walkthrough, we call the path of this file: $TESTCASE_PATH and the fuzz target you want to reproduce a crash on: $FUZZER_NAME (provided as “Fuzz target binary” in the bug description).

  2. Generate gn build configuration:

gn args out/fuzz

This will open up an editor. Copy the gn configuration parameters from the values provided in GN Config section in the ClusterFuzz testcase report.

  1. Build the fuzzer:
autoninja -C out/fuzz $FUZZER_NAME
  1. Set the *SAN_OPTIONS environment variable as provided in the Crash Stacktrace section in the testcase report. Here is an example value of ASAN_OPTIONS that is similar to its value on ClusterFuzz:
export ASAN_OPTIONS=redzone=256:print_summary=1:handle_sigill=1:allocator_release_to_os_interval_ms=500:print_suppressions=0:strict_memcmp=1:allow_user_segv_handler=0:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:detect_stack_use_after_return=0:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:allocator_may_return_null=1:handle_abort=1:check_malloc_usable_size=0:detect_container_overflow=0:quarantine_size_mb=256:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=1
  1. Run the fuzz target:
out/fuzz/$FUZZER_NAME -runs=100 $TESTCASE_PATH

If you see an un-symbolized stacktrace, please see the instructions here.

File a bug if you run into any issues.

Crashes reported as Unreproducible

ClusterFuzz generally does not report issues that it cannot reliably reproduce, unless the following condition is met. If a certain crash is occurring often enough, such a crash might be reported with Unreproducible label and an explicit clarification that there is no convenient way to reproduce it. There are two ways to work with such crashes.

  1. Try a speculative fix based on the stacktrace. Once the fix is landed, wait a couple days and then check Crash Statistics section on the ClusterFuzz testcase report page. If the fix works out, you will see that the crash is not happening anymore. If the crash does not occur again for a little while, ClusterFuzz will automatically close the issue as Verified.

  2. (libFuzzer only) Try to reproduce the whole fuzzing session. This workflow is very similar to the one described above for the Reproducible crashes. The only differences are:

  • On step 1, instead of downloading a single testcase, you need to download corpus backup. This can be done using the following command:
gsutil cp gs://clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME/latest.zip .
  • Alternatively, you can navigate to the following URL in your browser and download the latest.zip file:
https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME
  • Create an empty directory and unpack the corpus into it.
  • Follow steps 2-4 in the Reproducible section above.
  • On step 5, use the following command:
out/fuzz/$FUZZER_NAME -timeout=25 -rss_limit_mb=2048 -print_final_stats=1 $CORPUS_DIRECTORY_FROM_THE_PREVIOUS_STEP
  • Wait and hope that the fuzzer will crash.

Waiting for a crash to occur may take some time (up to 1hr), but if it happens, you will be able to test the fix locally and/or somehow debug the issue.