Life of a Chrome OS commit

This document provides a brief overview of how Chrome OS changes get committed and tested.

For details on Chrome changes, see Life of a Chrome commit on Chrome OS.

Create a Chrome OS change

Make and upload changes

See the developer guide for how to create a branch and make changes.

Once a change is completed and tested locally, upload it to chromium-review:

repo upload --current-branch .

Have your change reviewed

Use chromium-review to review your change and prepare it for the commit queue.

See the gerrit workflow for details.

The Chrome OS commit pipeline

As of January 2020, Chrome OS is using the parallel CQ. This documentation has been updated to reflect how that behaves.

A gross simplification of the Chrome OS Continous Integration (CI): Simplified diagram of Chrome OS CQ pipeline

The CQ collects all changes in the patch series, along with any changes linked via Cq-Depend in the commit messages, and applies them to tip of tree (ToT). The result is then built and tested by the cq-orchestrator.

The cq-orchestrator starts jobs to compile and unit test a CL across a variety of boards (see chromeos-cq builders). All of the jobs run in parallel and the cq-orchestator will collect all the results. If there are failures, the cq-orchestrator is a good place to start debugging what went wrong.

Once all the jobs are done, the cq-orchestrator kicks off cros_test_platform which starts and monitors all the hardware tests. Tests run by cros_test_platform require hardware resources and they will request this from the lab infrastructure that is currently in place (Skylab as of Jan 2020). If the hardware resource needed by the test isn't available, the test will wait for it to become available or timeout after several hours (exact time depends on the current state of builds and the health of the lab; ask in go/cros-oncall if you need a more accurate number). Gerrit displays all the jobs started by cq-orchestrator but not the jobs started by cros_test_platform so you will need to click through that chip to monitor the tests.

To get additional help debugging issues with the CQ you can ask for help at go/cros-oncall.

Dry-run

Once a change is marked CQ+1, the CQ will trigger a dry-run. The dry-run will trigger the cq-orchestrator and it will only run the compile + unit test tasks. The hardware tests will not be run.

The Commit-Queue

When a change is marked verified, approved and marked Commit-Queue ready, it will be sent to the commit queue.

It will follow the procedure described above, compiling the code, running unit tests and running hardware tests.

Post-submit builders

Post-submit builders are triggered roughly every 7-10 hours. They will run on the same subset of boards that CQ builders run. Chrome OS sheriffs monitor these builders to catch any regressions that made it through CQ.

The Release (Canary) Builders

The ToT (canary) release builders are triggered automatically according to a schedule, currently 3 times per day. The release builders will run on all boards (not just a subset) and so may catch issues that may be missed by the post-submit builders.

When you view goldeneye, the builds and test results you see are from the release builders.

Simple Chrome

Simple Chrome is intended to provide developers with a reasonably stable Chrome OS environment for Chrome development.

It uses the results of the Chrome LKGM builder to identify a recent stable canary build (generally the most recent).