This document describes how the Chromium Commit Queue (CQ) is structured and managed. This is specific for the Chromium CQ. Questions about other CQs should be directed to infra-dev@chromium.org.
The Chromium CQ exists to test developer changes before they land into chromium/src. It runs all the test suites which a given CL affects, and ensures that they all pass.
The Chromium CQ supports a variety of options that can change what it checks.
These options are supported via git footers. They must appear in the last paragraph of your commit message to be used. See
git help footersor git_footers.py for more information.
Binary-Size: <rationale>
This should be used when you are landing a change that will intentionally increase the size of the Chrome binaries on Android (since we try not to accidentally do so). The rationale should explain why this is okay to do.
Commit: false
You can mark a CL with this if you are working on experimental code and do not want to risk accidentally submitting it via the CQ. The CQ will immediately stop processing the change if it contains this option.
Cq-Include-Trybots: <trybots>
This flag allows you to specify some additional bots to run for this CL, in addition to the default bots. The format for the list of trybots is “bucket:trybot1,trybot2;bucket2:trybot3”.
Disable-Retries: true
The CQ will normally try to retry failed test shards (up to a point) to work around any intermittent infra failures. If this footer is set, it won't try to retry failed shards no matter what happens.
Ignore-Freeze: true
Whenever there is an active prod freeze (usually around Christmas), it can be bypassed by setting this footer.
Include-Ci-Only-Tests: true
Some builder configurations may run some tests only post-submit (on CI), and not pre-submit by default (in the CQ), for one reason or another (for example, if the tests are too slow, or too expensive). In order to still be able to explicitly reproduce what the CI builder is doing, you can specify this footer to run those tests pre-submit anyway.
No-Presubmit: true
If you want to skip the presubmit check, you can add this line, and the commit queue won‘t run the presubmit for your change. This should only be used when there’s a bug in the PRESUBMIT scripts. Please check that there‘s a bug filed against the bad script, and if there isn’t, file one.
No-Tree-Checks: true
Add this line if you want to skip the tree status checks. This means the CQ will commit a CL even if the tree is closed. Obviously this is strongly discouraged, since the tree is usually closed for a reason. However, in rare cases this is acceptable, primarily to fix build breakages (i.e., your CL will help in reopening the tree).
No-Try: true
This should only be used for reverts to green the tree, since it skips try bots and might therefore break the tree. You shouldn't use this otherwise.
CQ runs the jobs specified in commit-queue.cfg. See cq-builders.md for an auto generated file with links to information about the builders on the CQ.
Some of these jobs are experimental. This means they are executed on a percentage of CQ builds, and the outcome of the build doesn't affect if the CL can land or not. See the schema linked at the top of the file for more information on what the fields in the config do.
The CQ has the following structure:
Please follow these general guidelines:
bot_update) are failing, or you have reason to believe the CQ itself is broken, or you can‘t really tell what’s wrong, please file a trooper bug.In both cases, when filing bugs, please include links to the build and/or CL (including relevant patchset information) in question.
There are several requirements for a builder to be added to the Commit Queue.
Please email estaab@chromium.org, who will approve new build configurations.
Several builders are included in the CQ only for changes that affect specific directories. These used to be configured via Cq-Include-Trybots footers injected at CL upload time. They are now configured via the location_regexp attribute of the tryjob parameter to the try builder's definition e.g.
try_.some_builder_function(
name = "my-specific-try-builder",
tryjob = try_.job(
location_regexp = [
".+/{+]/path/to/my/specific/directory/.+"
],
),
)
The CQ can sometimes be flaky. Flakiness is when a test on the CQ fails, but should have passed (commonly known as a false negative). There are a few common causes of flaky tests on the CQ:
The CQ mitigates flakiness by retrying failed tests. The core tradeoff in retry policy is that adding retries increases the probability that a flaky test will land on tip of tree sublinearly, but mitigates the impact of the flaky test on unrelated CLs exponentially.
For example, imagine a CL that adds a test that fails with 50% probability. Even with no retries, the test will land with 50% probability. Subsequently, 50% of all unrelated CQ attempts would flakily fail. This effect is cumulative across different flaky tests. Since the CQ has roughly ~20,000 unique flaky tests, without retries, pretty much no CL would ever pass the CQ.
Have other questions? Run into any issues with the CQ? Email infra-dev@chromium.org, or file a trooper bug.