Chromium runs over 500,000 tests for each CL. There are many layers of UI for parsing and interpreting these test results. This doc provides a brief guide for navigating these UI layers.
Tests are segmented by build and test configurations. The segments are usually referred to as builds. In the example below, each green and red rectangle refers to a build.
The name of each build usually contains enough information to get a rough idea of the configuration. Some examples:
Green boxes refer to builds that passed. Red boxes refer to builds that failed. Some failed builds get automatically retried by the CQ. In this example, linux_chromium_rel_ng and mac_chromium_rel_ng were automatically retried [hence the two Xs], but win7_chromium_rel_ng was not. The X on the left is the first build, and the X on the right is the second build.
Each of these boxes is a link that provides more information about the build failure.
Selecting any of the build results from the previous section will navigate to the build results UI. Each build is implemented by a recipe -- effectively a Python script. Each recipe is divided into steps. Each step represents a well-defined action, such as updating the repository to point to tip of tree, or compiling the necessary build artifacts.
Under the Steps and Logfiles heading is a list of numbered steps. Each step has a color (red, green or purple) which indicates whether the step failed, succeeded, or encountered an unexpected condition. Failing steps are also grouped into the Results section at the very top for convenience.
Most builds follow a similar pattern. The key steps are listed here.
If all test suites pass, then the build is marked as a success and no further steps are run. If at least one test suite has failures, then the failing tests are rerun with the patch deapplied. This allows the recipe to determine if the test failure is due to the CL or due to a problem with tip of tree.
Important safety notice. When test suites are run with the patch applied, each test is run up to N times -- any success will mark the test as a success. When test suites are run without the patch, each failing test is run exactly N times. Any failure will mark the test as a failure.
If there are tests that failed with the patch applied, but not with the patch deapplied, then that implies that it's likely that the CL broke a test. Just to confirm, the first suite of steps is run again, this time with the suffix (retry with patch).