| # Reproducing libFuzzer and AFL crashes |
| |
| *** note |
| **Requirements:** For Windows, you must convert the forward slashes (/) to |
| backslashes (\\) in the commands below and use `set` command instead of `export` |
| to set the environment variable (step 4). Note that these commands are intended |
| to be used with cmd.exe, not PowerShell. Also, you may find [these tips] on how |
| to debug an ASAN instrumented binary helpful. |
| *** |
| |
| [TOC] |
| |
| ## Crashes reported as Reproducible |
| |
| The majority of the bugs reported by ClusterFuzz have **Reproducible** label. |
| That means there is a testcase that can be used to reliably reproduce the crash. |
| |
| 1. Download the testcase from ClusterFuzz. If you are CCed on an issue filed by |
| ClusterFuzz, a link to it is next to "Reproducer testcase" in the bug |
| description. |
| |
| For the rest of this walkthrough, we call the path of this |
| file: `$TESTCASE_PATH` and the fuzz target you want to reproduce a |
| crash on: `$FUZZER_NAME` (provided as "Fuzz Target" in the bug |
| description). |
| |
| 2. Generate gn build configuration: |
| |
| ``` |
| gn args out/fuzz |
| ``` |
| |
| This will open up an editor. Copy the gn configuration parameters from the |
| values provided in `GN Config` section in the ClusterFuzz testcase report. |
| |
| |
| 3. Build the fuzzer: |
| |
| ``` |
| autoninja -C out/fuzz $FUZZER_NAME |
| ``` |
| |
| 4. Set the `*SAN_OPTIONS` environment variable as provided in the |
| `Crash Stacktrace` section in the testcase report. |
| Here is an example value of `ASAN_OPTIONS` that is similar to its value on |
| ClusterFuzz: |
| |
| ``` |
| export ASAN_OPTIONS=redzone=256:print_summary=1:handle_sigill=1:allocator_release_to_os_interval_ms=500:print_suppressions=0:strict_memcmp=1:allow_user_segv_handler=0:use_sigaltstack=1:handle_sigfpe=1:handle_sigbus=1:detect_stack_use_after_return=0:alloc_dealloc_mismatch=0:detect_leaks=0:print_scariness=1:allocator_may_return_null=1:handle_abort=1:check_malloc_usable_size=0:detect_container_overflow=0:quarantine_size_mb=256:detect_odr_violation=0:symbolize=1:handle_segv=1:fast_unwind_on_fatal=0 |
| ``` |
| |
| 5. Run the fuzz target: |
| |
| ``` |
| out/fuzz/$FUZZER_NAME -runs=100 $TESTCASE_PATH |
| ``` |
| |
| [File a bug] if you run into any issues. |
| |
| ## Symbolizing stack traces |
| |
| Stack traces from ASAN builds are not symbolized by default. However, you |
| can symbolize them by piping the output into: |
| |
| ``` |
| src/tools/valgrind/asan/asan_symbolize.py |
| ``` |
| |
| ## Crashes reported as Unreproducible |
| |
| ClusterFuzz generally does not report issues that it cannot reliably reproduce, |
| unless the following condition is met. If a certain crash is occurring often |
| enough, such a crash might be reported with **Unreproducible** label and an |
| explicit clarification that there is no convenient way to reproduce it. There |
| are two ways to work with such crashes. |
| |
| 1. Try a speculative fix based on the stacktrace. Once the fix is landed, wait a |
| couple days and then check Crash Statistics section on the ClusterFuzz |
| testcase report page. If the fix works out, you will see that the crash is |
| not happening anymore. If the crash does not occur again for a little while, |
| ClusterFuzz will automatically close the issue as Verified. |
| |
| 2. (libFuzzer only) Try to reproduce the whole fuzzing session. This workflow is |
| very similar to the one described above for the **Reproducible** crashes. The |
| only differences are: |
| |
| * On step 1, instead of downloading a single testcase, you need to download |
| corpus backup. This can be done using the following command: |
| ``` |
| gsutil cp gs://clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME/latest.zip . |
| ``` |
| |
| * Alternatively, you can navigate to the following URL in your browser and |
| download the `latest.zip` file: |
| ``` |
| https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-backup/corpus/libfuzzer/$FUZZER_NAME |
| ``` |
| |
| * Create an empty directory and unpack the corpus into it. |
| * Follow steps 2-4 in the **Reproducible** section above. |
| * On step 5, use the following command: |
| |
| ``` |
| out/fuzz/$FUZZER_NAME -timeout=25 -rss_limit_mb=2048 -print_final_stats=1 $CORPUS_DIRECTORY_FROM_THE_PREVIOUS_STEP |
| ``` |
| |
| * Wait and hope that the fuzzer will crash. |
| |
| Waiting for a crash to occur may take some time (up to 1hr), but if it happens, |
| you will be able to test the fix locally and/or somehow debug the issue. |
| |
| ## Minimizing a crash input (optional) |
| |
| ClusterFuzz does crash input minimization automatically, and a typical crash |
| report has two testcases available for downloading: |
| |
| * An original testcase that has triggered the crash; |
| * A minimized testcase that is smaller than the original but triggers the same |
| crash. |
| |
| If you would like to further minimize a testcase, run the fuzz target with the |
| two additional arguments: |
| |
| * `-minimize_crash=1` |
| * `-exact_artifact_path=<output_filename_for_minimized_testcase>` |
| |
| The full command would be: |
| |
| ``` |
| out/fuzz/$FUZZER_NAME -minimize_crash=1 -exact_artifact_path=<minimized_testcase_path> $TESTCASE_PATH |
| ``` |
| |
| This might be useful for large testcases that make it hard to identify a root |
| cause of a crash. You can leave the minimization running locally for a while |
| (e.g. overnight) for better results. |
| |
| |
| [File a bug]: https://bugs.chromium.org/p/chromium/issues/entry?components=Tools%3EStability%3ElibFuzzer&comment=What%20problem%20are%20you%20seeing |
| [these tips]: https://github.com/google/sanitizers/wiki/AddressSanitizerWindowsPort#debugging |