Refactor recipe engine step execution.

This CL is unfortunately massive, but because the logic was previously
tangled across the whole codebase it was difficult to make this refactor

1. The StepRunner API is vastly simplified.

StepRunner API has been improved to have a small collection of well-defined
methods, replacing the previous API which required implementations to return
active 'OpenStep' implementations and implement context handlers.

None of these new methods are allowed to raise an exception, meaning that
the engine can operate purely in terms of their stated API. If an
implementation is naughty and raises an exception anyway, it will be logged
to the UI (as a RECIPE CRASH step) and then the engine will abort.

2. The StreamEngine is now solely owned by the Engine.

The StreamEngine is now owned entirely by the RecipeEngine. The
StepRunner is purely concerned with running steps, not with
manipulating or reporting anything to the UI.

3. All recipe running logic is now implemented in the Engine.

The engine now handles all calculation of step exceptions. Previously
the raising of exceptions was handled by various parties in various
ways around the engine and StepRunner, leading to bugs like

4. Unexpected errors are now reported and handled much better.

All steps now have a '$debug' log which records the engine's internal
thoughts about how it's processing the step. Now if anything goes wrong
while attempting to run the step, it will be recorded here. This includes
placeholders raising exceptions, bugs in the engine, failure to locate
the executable to run, failure to CD into the target directory of the
step, etc.

5. Recipe engine logs are no longer mixed into stdout of steps.

Previously information about the step would be dumped into the step's
stdout log (e.g. the command we're running, the working directory,
environment tweaks, the final return code of the step, etc.). Now all
of this data is reported to an 'execution details' log on every step.

6. Timeouts are now handled orthogonally to step status.

Previously timeouts would raise a StepTimeout and (I think? It's
hard to tell) would turn the step red regardless of if it was an
infra step or not. Now timeouts are recorded alongside the failure,
and `infra_step` correctly controls the default color and exception.

7. Parent steps are now explicitly handled vs. non-parent steps

Previously 'parent steps' were handled by the engine as-if they were
normal steps without a command. However their semantics are very
different since they 'remain open' even while other steps run (i.e.
the steps inside of them). The engine now explicitly handles parent
steps separately from leaf steps.

This also means that parent nesting steps now have much clearer
defined behavior in terms of what color they'll turn under different

8. Test runner is much improved.

Simulation tests now exercise a much greater portion of the recipe
engine logic. Test expectations now include (normalized) stack traces
when the recipe has exception(s), and (non-dropped) test expectations
are always written to disk, both of which make debugging failing
recipe simulations much easier.

9. Placeholder data on StepData will always be populated with None
as a default, even if one of the placeholders crashes.

This can help avoid secondary errors where one placeholder crashes
and the second placeholder's collection step isn't able to run.
Previously placeholder data was populated in-order for placeholders
and if a placeholder didn't run there was nothing added to StepData
for that value (leading to AttributeError's in unrelated areas of
the recipe).

Bug: 931473, 910369
Recipe-Manual-Change: build
Recipe-Manual-Change: release_scripts
Recipe-Nontrivial-Roll: build_limited_scripts_slave
Recipe-Nontrivial-Roll: chromiumos
Recipe-Nontrivial-Roll: depot_tools
Recipe-Nontrivial-Roll: infra
Recipe-Nontrivial-Roll: skia
Change-Id: Ie85704e30ff20eb7cc43c6a814a6b4f6ad2eae65
Reviewed-by: Andrii Shyshkalov <>
Commit-Queue: Robbie Iannucci <>
60 files changed
tree: d5ef36207d717159c963745d3a69cf72695285a2
  1. .gitattributes
  2. .gitignore
  3. .vpython
  11. codereview.settings
  12. doc/
  13. infra/
  14. misc/
  15. recipe_engine/
  16. recipe_modules/
  17. recipe_proto/
  19. recipes/
  20. unittests/


Recipes are a domain-specific language (embedded in python) for specifying sequences of subprocess calls in a cross-platform and testable way.



    This file!

  • doc/

    Documentation for the recipe engine (including this file!). See the design doc for more detailed design information about the recipe engine.

  • infra/

    Chrome infra config files.


    The main entry point to the recipe engine. It has many subcommands and flags; run -h to see them. Include this in your repository to start using recipes.

  • recipes/

    Recipes in the recipe engine. These are either example recipes, or recipes which are used to test the engine (see to see these run)

  • recipe_modules/

    Built in recipe modules. These are very useful when writing recipes; take a look in there, and look at each of their examples subfolders to get an idea how to use them in a recipe.

  • recipe_engine/

    The core functionality of the recipe engine. Noteworthy files include:

    • -- The main entrypoint for the recipe engine.
    • recipes_cfg.proto -- The protobuf file which defines the format of a recipes.cfg file.
    • third_party/ -- third_party code which is vendored into the recipe engine.
    • -- The api exposed to a recipe module.
    • unittests -- Unittests for the engine.

    There are also several files which correspond to a subcommand of; run, and autoroll are some examples.

  • unittests/

    Somewhat poorly named, these are higher level integration tests.


  • Sign the Google CLA.
  • Make sure your and are configured in git config.

Run the following to setup the code review tool and create your first review:

git clone $HOME/src/depot_tools
export PATH="$PATH:$HOME/src/depot_tools"
git checkout -b work origin/master

# hack hack

git commit -a -m "This is awesome"
# This will ask for your Google Account credentials.
git cl upload -s -r
# Wait for approval over email.
# Click "Submit to CQ" button or ask reviewer to do it for you.
# Wait for the change to be tested and landed automatically.

Use git cl help and git cl help <cmd> for more details.