Recipe Developer Workflow

What are recipes

Recipes are the build scripts in chrome infra. More info is here.

Why did CQ reject my change

A common reason is that your CL may cause downstream recipe expectations to change. Examine the output of the tryjob ; there should be steps like recipe_engine tests, build tests. The red steps correspond to the repo where, if your change was rolled, expectations would be changed.

What should I do

If your change needs a manual patch, because you are doing backwards incompatible changes which require non machine creatable patches downstream, Use a flag.

If your change is dangerous, please Use a flag. “Dangerous” means the author, infra people, or reviewers are scared about this breaking bots. This is ultimately up to your discretion, but please be thoughtful.

Otherwise, just use the Autoroller.

Autoroller

For cross-repo dependencies, please check cross repo.

When downstream repos depend on your recipe, the autorollers, controlled via these config options, automatically loads your change and makes CLs on the downstream repos. The autorollers also understands the cross-repo dependency graph, and propgates changes from upstream to downstream sequentially.

Rollout CLs have 2 types:

  • trivial, when the CL run the tests without any errors.
  • nontrivial, when the CL changes the expectations JSON files, but otherwise runs the tests successfully.

See config options for details.

The review processes are different as follows:

  • When the rollout CL is trivial, the autoroller submits the CL without owner approvals.
  • When the rollout CL is nontrivial, a repo owner needs to approve and submit the CL.

If there is a large number of recipe changes, you should probably Use a flag. If your change has expectation changes and you still want the autoroller to land it, see I just want to commit a change.

Use a flag

This means that you guard your change behind a flag, and disable it by default. It is your responsibility to then roll the change out manually (e.g. go to users of the feature and issue CLs to enable the new functionality). Once everything uses the new functionality:

  • change the default flag value to on-by-default
  • roll that out
  • remove the old code and flag.

Workflow:

  1. Write the new code you want to roll out. Example CL

  2. Add a new Single value to the recipe_module’s config which controls the behavior

    • Guard the new behavior behind the configuration value
    • It is acceptable to put a pragma: no cover for the if statement which triggers the new behavior

  3. Commit the change; it should pass all tryjobs and roll safely to all dependencies with no effects/expectation changes.

  4. Flip users of your recipe module over to the new behavior

    • It is recommended to do this one at a time, starting at the leaves of the dependency graph for your recipe module. This makes it much easier to revert your change if it breaks something, since you don’t need the roller to pick anything up to have the revert take effect.

  5. Once everything is flipped over, remove the flag and the old behavior.

    • This step is critical to ensuring that the recipes ecosystem doesn’t accumulate a bunch of half-working logic.
    • The infra team reserves the right to nag you about flagged behavior which is in the intermediate state for long periods of time.

Rollback

For cross-repo dependencies, please check cross repo.

After you revert an upstream recipe change, the autoroller also makes CLs to rollout the revert to the downstream repos.

  • When the rollout CL is trivial, the rollback is propagated without owner approvals.
  • When the rollout CL is nontrivial, the rollback requires owner approvals.

These processes may not be efficient to rollback production quicikly because:

  • The rollout CLs may take a long time in CQ.
  • The rollout CLs may fail at the downstream CQ.
  • The rollout CLs may wait for long time to get owner approvals.

For fast rollback, we have some options.

Bypass intermediate repos

You can bypass intermediate repos, and apply the rollback to the affected repos directly.

  1. Revert your recipe in the upstream repo.
  2. Update the revision in the downstream recipes.cfg to include the revert change.
  3. Wait for the autoroller to catchup other recipe dependencies.

Disable autoroller

You can revert the rollout CLs on the affected repos directly. But you need to stop the autoroller, otherwise it will reland the problematic CL.

  1. Add disable_reason to the downstream recipes.cfg.
  2. Revert the rollout CL on the affected repos.
  3. Revert your recipe in the upstream repo.
  4. Wait for the autoroller of other repos to propagte the revert.
  5. Remove the disable_reason from recipes.cfg.

I just want to commit my change

If you bypass the tryjob, and you break other bots, your change may be reverted by a sheriff/trooper, and you may receive a Post Mortem with your name plastered all over it by the relevant owners of the code, and/or other interested parties. These recipe changes can end up breaking the entire chrome infra bot fleet (~4000 machines), so please be careful. The tryjob is there for a reason.

Add Recipe-Tryjob-Bypass-Reason: your reason here to the bottom of your CL description (example here), and the tryjob will turn green.

I want to talk to someone

Email infra-dev@chromium.org (or chrome-infra@google.com, if it’s an internal question). If you really want to talk to a human, you can email martiniss@