Chromium OS infrastructure recipes (

Clone this repo:


  1. e38a12b Add method to analyze swarming TaskResults. by Andrew Lamb · 3 minutes ago master
  2. 06cbafe cros_history: sort the list of passed builds by Dhanya Ganesh · 5 minutes ago
  3. 873ebcd Roll recipe dependencies (trivial). by recipe-roller · 36 minutes ago
  4. 0d1ddb4 failures: Highlight that listed builds failed by Evan Hernandez · 41 minutes ago
  5. ed51aa8 orchestrator: Nest test collection under step by Evan Hernandez · 43 minutes ago

Chrome OS Recipes

This is a quick-start guide for Chrome OS Recipes. For more in-depth information about the Recipes framework, see the Recipes User Guide.

The recipes and recipe modules in this repo have documentation auto-generated by the ‘test train’ process:

Getting Started

The script in this repo will bootstrap itself. You can verify that it is working by running tests:

./ test run

After making a change, you will usually need to “re-train” expected output in order for tests to pass:

./ test train

This will also regenerate the documentation. That file and the expectation data in *.expected directories should be checked in with your changes.

Support Tools

This repo contains some non-recipes support tooling in the support/ directory. These tools are delivered to CI hosts via CIPD. This process is automated by the support/ script. See support/README.txt for more details.

Testing With led

Expectation tests only mock the external services a recipe depends on. This means the expectation tests cannot catch bugs caused by calling external services incorrectly. For example, if a recipe made the call

ls --bad_flag

it would not be caught by ./ test train. A more realistic example is the recipe makes incorrect assumptions about ACLs, paths, responses from the service, etc.

One way to get more confidence the recipe code will work is by running it with the actual builder definitions and swarming environment, using the led command-line tool. This will call the external services with the same user, timeouts, swarming caches, etc. as the submitted recipes.

As stated above, led means the code will actually run commands on swarming. This means the code being tested can impact production resources, e.g. clobber a Google Storage file, break Skylab DUTs, etc. Do NOT blindly test code with led, and use the staging environment where possible, which may limit the blast radius of bugs.

To use led, first call get-build or get-builder to get a job definition. For example, to test a change to the Annealing builder, we could start by running

led get-builder staging:staging-Annealing > builder_def.json

which will produce a JSON object containing properties, swarming information, etc. Sometimes, builders require properties that are not set in the lucicfg definition. For example, build_target requires the gitiles_commit property. In this case, we could get a definition from a previous build. For example, say the last build of staging-amd64-generic-postsubmit has id 12345. To get a job definition with the same properties (including gitiles_commit), run the command

led get-build 12345 > builder_def.json

Next, the led edit-recipe-bundle command patches your local recipes edits into the job definition, and led launch launches the job on swarming. The job definition is piped from each command to the next, so in practice an example command would be

led get-build 12345 | led edit-recipe-bundle | led launch

The output from led launch should give a link to the swarming task, e.g.

[I 2019-05-06 15:31:43] Launched swarming task:
[I 2019-05-06 15:31:43] LUCI UI:

Recipe Tester Presubmit

Recipe changes run through a presubmit job that verifies them on actual ChromeOS CI builders (essentially automating the led testing process).

The builders to be run are controlled with the builders property on the recipes_tester recipe. Builders will not be run if the recipe change does not affect them (as determined by ./ analyze, which checks if the recipe or any of its dependencies has been modified).

If you are confident that your change does not need testing, you can include the Recipes-Tester-Skip-Builder footer in your CL. That this should be done sparingly. If the presubmit is flaky, the root cause should be fixed. Please put a justification in the CL description for why the builder is skipped. The footer syntax is a bit picky; an example commit message would look like

A test commit

- Note: Skipping testers because this change must be submitted now.

Recipes-Tester-Skip-Builder: Builder1
Recipes-Tester-Skip-Builder: Builder2
Change-Id: 1234

Use depot_tools/git-footers to verify the footer syntax. For example, the above commit returns

Recipes-Tester-Skip-Builder: Builder2
Recipes-Tester-Skip-Builder: Builder1
Change-Id: 1234

Note that adding a space between footers is not allowed. For example, the commit message

A test commit

- Note: Skipping testers because this change must be submitted now.

Recipes-Tester-Skip-Builder: Builder1
Recipes-Tester-Skip-Builder: Builder2

Change-Id: 1234

will return

Change-Id: 1234

and thus the builders will not be skipped. Also note that each builder to skip must have its own line; Recipes-Tester-Skip-Builder: Builder1, Builder2 will not work.

Chrome OS Recipes Code Style

  • The Python code in this repo should largely conform to Chromium Python style except that Recipes code has a convention of 2 space indents. Practically, we use YAPF for automatic code formatting; whatever style is generated by YAPF should be used.

  • The code in this repo uses docstring-based type annotations as implemented by the Sphinx Napoleon Plugin. This format is supported by PyCharm / IntelliJ (Settings > Tools > Python Integrated Tools > Docstring format > Google).

  • The Recipes framework does not require the RecipesApi subclass in recipe modules to have any particular name, but this repo has some tooling that expects it to be named NameApi derived from the recipe_modules/<name>/.