Reproducing Crashes from ClusterFuzz

Introduction

This guide provides step-by-step instructions for reproducing crashes found by ClusterFuzz. Reproducing a crash locally is the first step toward debugging and fixing it.

Requirements: For Windows, you must convert the forward slashes (/) to backslashes (\) in the commands below and use set command instead of export to set the environment variable (step 4). Note that these commands are intended to be used with cmd.exe, not PowerShell. Also, you may find these tips on how to debug an ASAN instrumented binary helpful.

Which Workflow Should I Use?

The first step is to identify how the crash was found. Look at the ClusterFuzz bug report:

If the report specifies a “Fuzz Target” and mentions a fuzzing engine (like libFuzzer, AFL, Centipede, etc.), you should follow the steps under Reproducing a Crash from a Fuzzing Engine.
Otherwise, the crash was likely caused by a blackbox fuzzer that produced a file to crash a larger application (like content_shell). You should follow the steps under Reproducing a Crash from a Blackbox Fuzzer.

Reproducing a Crash from a Fuzzing Engine

The majority of the bugs reported by ClusterFuzz have the Reproducible label. This workflow applies to those cases.

Download the testcase from ClusterFuzz. If you are CCed on an issue filed by ClusterFuzz, a link to it is next to “Reproducer testcase” in the bug description.
For the rest of this section, we call the path of this file: $TESTCASE_PATH and the fuzz target you want to reproduce a crash on: $FUZZER_NAME (provided as “Fuzz Target” in the bug description).
Generate gn build configuration:
```
gn args out/fuzz
```
This will open up an editor. Copy the gn configuration parameters from the values provided in GN Config section in the ClusterFuzz testcase report.
Build the fuzzer:
```
autoninja -C out/fuzz $FUZZER_NAME
```

Set the *SAN_OPTIONS environment variable as provided in the Crash Stacktrace section in the testcase report. Here is an example value of ASAN_OPTIONS that is similar to its value on ClusterFuzz:

export ASAN_OPTIONS=redzone=256:print_summary=1:handle_sigill=1:allocator_release_to_os_interval_ms=500:print_suppressions=0:strict_memcmp=1:allow_user_segv_handler=0:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:detect_stack_use_after_return=0:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:allocator_may_return_null=1:handle_abort=1:check_malloc_usable_size=0:detect_container_overflow=0:quarantine_size_mb=256:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=0

Note: The sanitizer options may reference suppressions files. You can find those at go/crfuzz-clusterfuzz-suppressions (google-internal).

Run the fuzz target with the downloaded testcase. The -runs=100 flag helps ensure reproduction even if the crash has minor flakiness.
```
out/fuzz/$FUZZER_NAME -runs=100 $TESTCASE_PATH
```

File a bug if you run into any issues.

Reproducing a Crash from a Blackbox Fuzzer

Get info from the report.
- Download the Reproducer testcase. We'll call its path $TESTCASE_PATH.
- Identify the target application from the Job Type (e.g., linux_asan_content_shell_drt means the target is content_shell).
Configure and build the application:
Generate the build configuration using the arguments from the “GN Config” section.
```
gn args out/fuzz
```
Build the target application binary (e.g., content_shell).
```
autoninja -C out/fuzz content_shell
```
Set environment variables:
Export all variables listed under [Environment] in the stack trace of the report.
```
export ASAN_OPTIONS=...
# Also export any other variables listed.
```
Construct and run the command:
In the stack trace, find the line beginning with Command line:. Construct your local command by replacing the executable path (e.g., /proc/self/exe or /mnt/.../content_shell) with the path to your built binary (e.g., out/fuzz/content_shell), copying all the flags, and appending the $TESTCASE_PATH at the end.
```
# Example command based on a content_shell crash
out/fuzz/content_shell \
--run-web-tests \
--disable-in-process-stack-traces \
--autoplay-policy=no-user-gesture-required \
--lang=en-US \
$TESTCASE_PATH
```

Crashes reported as Unreproducible

ClusterFuzz generally does not report issues that it cannot reliably reproduce, unless the following condition is met. If a certain crash is occurring often enough, such a crash might be reported with Unreproducible label and an explicit clarification that there is no convenient way to reproduce it. There are two ways to work with such crashes.

Option 1: Attempt a Speculative Fix

Try a speculative fix based on the stacktrace. Once the fix is landed, wait a couple days and then check Crash Statistics section on the ClusterFuzz testcase report page. If the fix works out, you will see that the crash is not happening anymore. If the crash does not occur again for a little while, ClusterFuzz will automatically close the issue as Verified.

Option 2: Replay the Fuzzing Session (libFuzzer Only)

Try to reproduce the whole fuzzing session. This workflow is very similar to the one described above for reproducing a crash from a fuzzing engine. The only differences are:

Instead of downloading a single testcase, you need to download corpus backup. This can be done using the following command:
```
gsutil cp gs://clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME/latest.zip .
```

Alternatively, you can navigate to the following URL in your browser and download the latest.zip file:

https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME

Create an empty directory and unpack the corpus into it.
Follow steps 2-4 in the reproducing a crash from a fuzzing engine section above.

On step 5, use the following command:

out/fuzz/$FUZZER_NAME -timeout=25 -rss_limit_mb=2048 -print_final_stats=1 $CORPUS_DIRECTORY_FROM_THE_PREVIOUS_STEP

Wait and hope that the fuzzer will crash.

Waiting for a crash to occur may take some time (up to 1hr), but if it happens, you will be able to test the fix locally and/or somehow debug the issue.

Symbolizing stack traces

Stack traces from ASAN builds are not symbolized by default. However, you can symbolize them by piping the output into:

src/tools/valgrind/asan/asan_symbolize.py

Minimizing a crash input (optional)

ClusterFuzz does crash input minimization automatically, and a typical crash report has two testcases available for downloading:

An original testcase that has triggered the crash;
A minimized testcase that is smaller than the original but triggers the same crash.

If you would like to further minimize a testcase, run the fuzz target with the two additional arguments:

-minimize_crash=1
-exact_artifact_path=<output_filename_for_minimized_testcase>

The full command would be:

out/fuzz/$FUZZER_NAME -minimize_crash=1 -exact_artifact_path=<minimized_testcase_path> $TESTCASE_PATH

This might be useful for large testcases that make it hard to identify a root cause of a crash. You can leave the minimization running locally for a while (e.g. overnight) for better results.