Testing in Chromium

Chrome OS Debugging Instructions

Chrome on Chrome OS is tested using a handful of frameworks, each of which you‘ll find running on Chrome’s CQ and waterfalls. If you're investigating failures in these tests, below are some tips for debugging and identifying the cause.

This doc outlines tests running in true Chrome OS environments (ie: on virtual machines or real devices). linux-chromeos tests, on the other hand, can be debugged like any other linux test.

Tast

Tast is Chrome OS‘s integration testing framework. Since Chrome itself is instrumental to the Chrome OS system, it’s equally important that we run some of these integration tests on Chrome's waterfalls. If you find one of these tests failing (likely in the chrome_all_tast_tests step), you can:

  • Inspect the failed test's log snippet: There should be a log link for each failed test with failure information. eg: For this failed build, opening the ui.WindowControl log link contains stack traces and error messages.

  • View browser & system logs: A common cause of failure on Chrome‘s builders are browser crashes. When this happens, each test’s log snippets will simply contain warnings like “Chrome probably crashed”. To debug these crashes, navigate to the test‘s Isolated output, listed in the build under the test step’s shard #0 isolated out link. There you‘ll find expanded logs for every test. For example, the tests/ui.WindowControl/messages log has more info than its earlier snippet. Additionally, you can find system logs under the system_logs/ prefix. To find a system log for a particular test, match the timestamps printed in the test’s log with the timestamps present in the system log filename. For instance, the previous ui.WindowControl failure matches the system_logs/chrome/chrome_20201029-195153 browser log, which contains the culprit Chrome crash and backtrace.

  • Symbolizing a browser crash dump: See below.

Disabling a test

There a couple ways to disable a test on Chrome's builders:

  • With a full CrOS checkout: If you have a full CrOS checkout, you can add the informational attribute to the test‘s definition (see Tast attributes for more info). This can take time (ie: many hours) to land and propagate onto Chrome’s builders. So if you need the test disabled ASAP, consult the next option.
  • With only a Chromium checkout: You can also add the test to the list of disabled tests for the step's GN target. For example, to disable a test in the chrome_all_tast_tests step, add it to this list.

In both cases, please make sure a bug is filed for the test, and route it to the appropriate owners.

Symbolizing a crash dump

If a test fails due to a browser crash, there should be a Minidump crash report present in the test‘s isolated out under the prefix crashes/chrome.... These reports aren’t very useful by themselves, but with a few commands you can symbolize the report locally to get insight into what conditions caused Chrome to crash.

To do so, first download both the task‘s input isolate (this provides the symbols and the symbolizing tools) as well as the task’s output isolate (this provides the crash reports). See the commands listed under the Reproducing the task locally section on the task page. For example, to download them for this task, cd into a tmp directory and run:

$CHROME_DIR/tools/luci-go/isolated download -I https://chrome-isolated.appspot.com --namespace default-gzip -isolated 64919fee8b02d826df2401544a9dc0f7dfa2172d -output-dir input
python $CHROME_DIR/tools/swarming_client/swarming.py collect -S chrome-swarming.appspot.com 506a01dd12c8a610 --task-output-dir output

Once both isolates have been fetched you must then generate the breakpad symbols by pointing the generate_breakpad_symbols.py script to the input's build dir:

python input/components/crash/content/tools/generate_breakpad_symbols.py --symbols-dir symbols --build-dir input/out/Release/ --binary input/out/Release/chrome

That will generate the symbols in the symbols/ dir. Then to symbolize a Chrome crash report present in the task's output (such as chrome.20201211.041043.31022.5747.dmp):

./input/out/Release/minidump_stackwalk output/0/crashes/chrome.20201211.041043.31022.5747.dmp symbols/

Running a test locally

To run a Tast test the same way it‘s ran on Chrome’s builders:

  • Decide which Chrome OS device type or VM to test on.

  • Build Chrome via the Simple Chrome workflow for that board.

  • Deploy your Chrome to the device via the deploy_chrome.py tool.

  • Finally, run the Tast test on the device via the cros_run_test tool under //third_party/chromite/bin/. eg: cros_run_test --device $DEVICE_IP --tast ui.ChromeLogin. See here for more info on cros_run_test.

Telemetry

TODO: Add instructions for debugging telemetry failures.

GTest

TODO: Add instructions for debugging GTest failures.

Rerunning these tests locally

TODO: Add instructions for rerunning these tests locally.