ChromiumOS infrastructure recipes (

Clone this repo:
  1. dba84a3 Roll recipe dependencies (trivial). by recipe-roller · 7 hours ago main
  2. 59e2f7d Roll recipe dependencies (trivial). by recipe-roller · 10 hours ago
  3. dc7f58a Roll recipe dependencies (trivial). by recipe-roller · 11 hours ago
  4. 9549e96 Roll recipe dependencies (trivial). by recipe-roller · 12 hours ago
  5. 3edd9b9 Roll recipe dependencies (trivial). by recipe-roller · 15 hours ago

ChromeOS Recipes

This is a quick-start guide for ChromeOS Recipes. For more in-depth information about the Recipes framework, see the Recipes User Guide in the Chromium repo. This document only covers the ChromeOS recipes built on top of the Chrome-provided framework.

The recipes and recipe modules in this repo have documentation auto-generated by the ‘test train’ process:

Getting Started

The script in this repo will bootstrap itself. You can verify that it is working by running tests:

./ test run

After making a change, you will usually need to “re-train” expected output in order for tests to pass:

./ test train

This will also regenerate the documentation. That file and the expectation data in *.expected directories should be checked in with your changes.

Navigating the code

The top-level code for the main set of builders can be found in the infra/recipes/recipes directory, in the build_*.py files. For example, the top-level code for the postsubmit builders is in the RunSteps function in The tests for each builder are in the same file, in the GenTests function.

The builders in infra/recipes/recipes will generally call into support functions in infra/recipes/recipe_modules. In particular, calls to the RPCs defined in chromite/api generally happen from recipe_modules.

Support Tools

This repo contains some non-recipes support tooling in the support/ directory. These tools are delivered to CI hosts via CIPD. This process is automated by the support/ script. See support/README.txt for more details.

Testing Locally

Recipes can be run locally:

./ run <recipe>

Testing With led

Expectation tests only mock the external services a recipe depends on. This means the expectation tests cannot catch bugs caused by calling external services incorrectly. For example, if a recipe made the call

ls --bad_flag

it would not be caught by ./ test train. A more realistic example is the recipe makes incorrect assumptions about ACLs, paths, responses from the service, etc.

One way to get more confidence the recipe code will work is by running it with the actual builder definitions and swarming environment, using the led command-line tool. This will call the external services with the same user, timeouts, swarming caches, etc. as the submitted recipes.

As stated above, led means the code will actually run commands on swarming. This means the code being tested can impact production resources, e.g. clobber a Google Storage file, break Skylab DUTs, etc. Do NOT blindly test code with led, and use the staging environment where possible, which may limit the blast radius of bugs.

To use led, first call get-build or get-builder to get a job definition. For example, to test a change to the Annealing builder, we could start by running

led get-builder staging:staging-Annealing > builder_def.json

which will produce a JSON object containing properties, swarming information, etc. Sometimes, builders require properties that are not set in the lucicfg definition. For example, build_target requires the gitiles_commit property. In this case, we could get a definition from a previous build. For example, say the last build of staging-amd64-generic-postsubmit has id 12345. To get a job definition with the same properties (including gitiles_commit), run the command

led get-build 12345 > builder_def.json

Next, the led edit-recipe-bundle command patches your local recipes edits into the job definition, and led launch launches the job on swarming. The job definition is piped from each command to the next, so in practice an example command would be

led get-build 12345 | led edit-recipe-bundle | led launch

The output from led launch should give a link to the swarming task, e.g.

[I 2019-05-06 15:31:43] Launched swarming task:
[I 2019-05-06 15:31:43] LUCI UI:

Test Recipes Presubmit

Recipe changes run through a presubmit job that verifies them on actual ChromeOS CI builders (essentially automating the led testing process).

The builders to be run are controlled with the builders property on the test_recipes recipe. Builders will not be run if the recipe change does not affect them (as determined by ./ analyze, which checks if the recipe or any of its dependencies has been modified).

If you are confident that your change does not need testing, you can include the Test-Recipes-Skip-Builder footer in your CL. That this should be done sparingly. If the presubmit is flaky, the root cause should be fixed. Please put a justification in the CL description for why the builder is skipped. The footer syntax is a bit picky; an example commit message would look like

A test commit

- Note: Skipping testers because this change must be submitted now.

Test-Recipes-Skip-Builder: Builder1
Test-Recipes-Skip-Builder: Builder2
Change-Id: 1234

Use depot_tools/git-footers to verify the footer syntax. For example, the above commit returns

Test-Recipes-Skip-Builder: Builder2
Test-Recipes-Skip-Builder: Builder1
Change-Id: 1234

Note that adding a space between footers is not allowed. For example, the commit message

A test commit

- Note: Skipping testers because this change must be submitted now.

Test-Recipes-Skip-Builder: Builder1
Test-Recipes-Skip-Builder: Builder2

Change-Id: 1234

will return

Change-Id: 1234

and thus the builders will not be skipped. Also note that each builder to skip must have its own line; Test-Recipes-Skip-Builder: Builder1, Builder2 will not work.

ChromeOS Recipes Code Style

  • The Python code in this repo should largely conform to Chromium Python style except that Recipes code has a convention of 2 space indents. Practically, we use YAPF for automatic code formatting; whatever style is generated by YAPF should be used.

  • The code in this repo uses docstring-based type annotations as implemented by the Sphinx Napoleon Plugin. This format is supported by PyCharm / IntelliJ (Settings > Tools > Python Integrated Tools > Docstring format > Google).

  • The Recipes framework does not require the RecipesApi subclass in recipe modules to have any particular name, but this repo has some tooling that expects it to be named NameApi derived from the recipe_modules/<name>/.


See go/chromeos-ci-releases (internal link) for information about the recipes deployment process.