libFuzzer and AFL crashes can be reproduced easily with the ClusterFuzz Reproduce Tool. However, if you are unable to use the tool (e.g. unsupported platform, some other tool issue), you can still reproduce the crash manually using the steps below:
set
command instead of export
to set the environment variable (step 4). Note that these commands are intended to be used with cmd.exe, not PowerShell.The majority of the bugs reported by ClusterFuzz have Reproducible label. That means there is a testcase that can be used to reliably reproduce the crash.
Download the testcase from ClusterFuzz. If you are CCed on an issue filed by ClusterFuzz, a link to it is next to “Reproducer testcase” in the bug description.
For the rest of this walkthrough, we call the path of this file: $TESTCASE_PATH
and the fuzz target you want to reproduce a crash on: $FUZZER_NAME
(provided as “Fuzz target binary” in the bug description).
Generate gn build configuration:
gn args out/fuzz
This will open up an editor. Copy the gn configuration parameters from the values provided in GN Config
section in the ClusterFuzz testcase report.
autoninja -C out/fuzz $FUZZER_NAME
*SAN_OPTIONS
environment variable as provided in the Crash Stacktrace
section in the testcase report. Here is an example value of ASAN_OPTIONS
that is similar to its value on ClusterFuzz:export ASAN_OPTIONS=redzone=256:print_summary=1:handle_sigill=1:allocator_release_to_os_interval_ms=500:print_suppressions=0:strict_memcmp=1:allow_user_segv_handler=0:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:detect_stack_use_after_return=0:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:allocator_may_return_null=1:handle_abort=1:check_malloc_usable_size=0:detect_container_overflow=0:quarantine_size_mb=256:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=1
out/fuzz/$FUZZER_NAME -runs=100 $TESTCASE_PATH
If you see an un-symbolized stacktrace, please see the instructions here.
File a bug if you run into any issues.
ClusterFuzz generally does not report issues that it cannot reliably reproduce, unless the following condition is met. If a certain crash is occurring often enough, such a crash might be reported with Unreproducible label and an explicit clarification that there is no convenient way to reproduce it. There are two ways to work with such crashes.
Try a speculative fix based on the stacktrace. Once the fix is landed, wait a couple days and then check Crash Statistics section on the ClusterFuzz testcase report page. If the fix works out, you will see that the crash is not happening anymore. If the crash does not occur again for a little while, ClusterFuzz will automatically close the issue as Verified.
(libFuzzer only) Try to reproduce the whole fuzzing session. This workflow is very similar to the one described above for the Reproducible crashes. The only differences are:
gsutil cp gs://clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME/latest.zip .
latest.zip
file:https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME
out/fuzz/$FUZZER_NAME -timeout=25 -rss_limit_mb=2048 -print_final_stats=1 $CORPUS_DIRECTORY_FROM_THE_PREVIOUS_STEP
Waiting for a crash to occur may take some time (up to 1hr), but if it happens, you will be able to test the fix locally and/or somehow debug the issue.