Recipes are the build scripts in chrome infra. More info is here.
A common reason is that your CL may cause downstream recipe expectations to change. Examine the output of the tryjob ; there should be steps like
build tests. The red steps correspond to the repo where, if your change was rolled, expectations would be changed.
If your change needs a manual patch, because you are doing backwards incompatible changes which require non machine creatable patches downstream, Use a flag.
If your change is dangerous, please Use a flag. “Dangerous” means the author, infra people, or reviewers are scared about this breaking bots. This is ultimately up to your discretion, but please be thoughtful.
Otherwise, just use the Autoroller.
When downstream repos depend on your recipe, the autorollers, controlled via these config options, automatically loads your change and makes CLs on the downstream repos. The autorollers also understands the cross-repo dependency graph, and propgates changes from upstream to downstream sequentially.
Rollout CLs have 2 types:
trivial, when the CL run the tests without any errors.
nontrivial, when the CL changes the expectations JSON files, but otherwise runs the tests successfully.
See config options for details.
The review processes are different as follows:
trivial, the autoroller submits the CL without owner approvals.
nontrivial, a repo owner needs to approve and submit the CL.
This means that you guard your change behind a flag, and disable it by default. It is your responsibility to then roll the change out manually (e.g. go to users of the feature and issue CLs to enable the new functionality). Once everything uses the new functionality:
Write the new code you want to roll out. Example CL
Add a new
Single value to the
recipe_module’s config which controls the behavior
pragma: no coverfor the if statement which triggers the new behavior
Commit the change; it should pass all tryjobs and roll safely to all dependencies with no effects/expectation changes.
Flip users of your recipe module over to the new behavior
Once everything is flipped over, remove the flag and the old behavior.
After you revert an upstream recipe change, the autoroller also makes CLs to rollout the revert to the downstream repos.
trivial, the rollback is propagated without owner approvals.
nontrivial, the rollback requires owner approvals.
These processes may not be efficient to rollback production quicikly because:
For fast rollback, we have some options.
You can bypass intermediate repos, and apply the rollback to the affected repos directly.
recipes.cfgto include the revert change.
You can revert the rollout CLs on the affected repos directly. But you need to stop the autoroller, otherwise it will reland the problematic CL.
If you bypass the tryjob, and you break other bots, your change may be reverted by a sheriff/trooper, and you may receive a Post Mortem with your name plastered all over it by the relevant owners of the code, and/or other interested parties. These recipe changes can end up breaking the entire chrome infra bot fleet (~4000 machines), so please be careful. The tryjob is there for a reason.
Recipe-Tryjob-Bypass-Reason: your reason here to the bottom of your CL description (example here), and the tryjob will turn green.