Revert "Refactor recipe engine step execution."

This reverts commit 440a45cae6b329192f60313e41714bd78e886c8d.

Reason for revert: Resolving cmd0 fails for commands provided with relative paths (e.g. './foo'): https://skia-review.googlesource.com/c/buildbot/+/213520

Original change's description:
> Refactor recipe engine step execution.
> 
> This CL is unfortunately massive, but because the logic was previously
> tangled across the whole codebase it was difficult to make this refactor
> smaller.
> 
> 1. The StepRunner API is vastly simplified.
> 
> StepRunner API has been improved to have a small collection of well-defined
> methods, replacing the previous API which required implementations to return
> active 'OpenStep' implementations and implement context handlers.
> 
> None of these new methods are allowed to raise an exception, meaning that
> the engine can operate purely in terms of their stated API. If an
> implementation is naughty and raises an exception anyway, it will be logged
> to the UI (as a RECIPE CRASH step) and then the engine will abort.
> 
> 
> 2. The StreamEngine is now solely owned by the Engine.
> 
> The StreamEngine is now owned entirely by the RecipeEngine. The
> StepRunner is purely concerned with running steps, not with
> manipulating or reporting anything to the UI.
> 
> 
> 3. All recipe running logic is now implemented in the Engine.
> 
> The engine now handles all calculation of step exceptions. Previously
> the raising of exceptions was handled by various parties in various
> ways around the engine and StepRunner, leading to bugs like crbug.com/931473.
> 
> 
> 4. Unexpected errors are now reported and handled much better.
> 
> All steps now have a '$debug' log which records the engine's internal
> thoughts about how it's processing the step. Now if anything goes wrong
> while attempting to run the step, it will be recorded here. This includes
> placeholders raising exceptions, bugs in the engine, failure to locate
> the executable to run, failure to CD into the target directory of the
> step, etc.
> 
> 
> 5. Recipe engine logs are no longer mixed into stdout of steps.
> 
> Previously information about the step would be dumped into the step's
> stdout log (e.g. the command we're running, the working directory,
> environment tweaks, the final return code of the step, etc.). Now all
> of this data is reported to an 'execution details' log on every step.
> 
> 
> 6. Timeouts are now handled orthogonally to step status.
> 
> Previously timeouts would raise a StepTimeout and (I think? It's
> hard to tell) would turn the step red regardless of if it was an
> infra step or not. Now timeouts are recorded alongside the failure,
> and `infra_step` correctly controls the default color and exception.
> 
> 
> 7. Parent steps are now explicitly handled vs. non-parent steps
> 
> Previously 'parent steps' were handled by the engine as-if they were
> normal steps without a command. However their semantics are very
> different since they 'remain open' even while other steps run (i.e.
> the steps inside of them). The engine now explicitly handles parent
> steps separately from leaf steps.
> 
> This also means that parent nesting steps now have much clearer
> defined behavior in terms of what color they'll turn under different
> circumstances.
> 
> 
> 8. Test runner is much improved.
> 
> Simulation tests now exercise a much greater portion of the recipe
> engine logic. Test expectations now include (normalized) stack traces
> when the recipe has exception(s), and (non-dropped) test expectations
> are always written to disk, both of which make debugging failing
> recipe simulations much easier.
> 
> 
> 9. Placeholder data on StepData will always be populated with None
> as a default, even if one of the placeholders crashes.
> 
> This can help avoid secondary errors where one placeholder crashes
> and the second placeholder's collection step isn't able to run.
> Previously placeholder data was populated in-order for placeholders
> and if a placeholder didn't run there was nothing added to StepData
> for that value (leading to AttributeError's in unrelated areas of
> the recipe).
> 
> 
> Bug: 931473, 910369
> Recipe-Manual-Change: build
> Recipe-Manual-Change: release_scripts
> Recipe-Nontrivial-Roll: build_limited_scripts_slave
> Recipe-Nontrivial-Roll: chromiumos
> Recipe-Nontrivial-Roll: depot_tools
> Recipe-Nontrivial-Roll: infra
> Recipe-Nontrivial-Roll: skia
> Change-Id: Ie85704e30ff20eb7cc43c6a814a6b4f6ad2eae65
> Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/1534344
> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
> Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

TBR=iannucci@chromium.org,nodir@chromium.org,tandrii@chromium.org,martiniss@chromium.org

Change-Id: Ic6b7f2c2ecb2d9277f25b7ec14c56c1a9d805b68
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 931473, 910369
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/1610702
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
60 files changed
tree: 6aa9a88acfcefaa397b3e6126b8b5cdb919224dd
  1. .gitattributes
  2. .gitignore
  3. .vpython
  4. AUTHORS
  5. CONTRIBUTORS
  6. LICENSE
  7. OWNERS
  8. PRESUBMIT.py
  9. README.md
  10. README.recipes.md
  11. codereview.settings
  12. doc/
  13. infra/
  14. misc/
  15. recipe_engine/
  16. recipe_modules/
  17. recipe_proto/
  18. recipes.py
  19. recipes/
  20. unittests/
README.md

Recipes

Recipes are a domain-specific language (embedded in python) for specifying sequences of subprocess calls in a cross-platform and testable way.

Files

  • README.md

    This file!

  • doc/

    Documentation for the recipe engine (including this file!). See the design doc for more detailed design information about the recipe engine.

  • infra/

    Chrome infra config files.

  • recipes.py

    The main entry point to the recipe engine. It has many subcommands and flags; run recipes.py -h to see them. Include this in your repository to start using recipes.

  • recipes/

    Recipes in the recipe engine. These are either example recipes, or recipes which are used to test the engine (see run_test.py to see these run)

  • recipe_modules/

    Built in recipe modules. These are very useful when writing recipes; take a look in there, and look at each of their examples subfolders to get an idea how to use them in a recipe.

  • recipe_engine/

    The core functionality of the recipe engine. Noteworthy files include:

    • main.py -- The main entrypoint for the recipe engine.
    • recipes_cfg.proto -- The protobuf file which defines the format of a recipes.cfg file.
    • third_party/ -- third_party code which is vendored into the recipe engine.
    • recipe_api.py -- The api exposed to a recipe module.
    • unittests -- Unittests for the engine.

    There are also several files which correspond to a subcommand of recipes.py; run, and autoroll are some examples.

  • unittests/

    Somewhat poorly named, these are higher level integration tests.

Contributing

  • Sign the Google CLA.
  • Make sure your user.email and user.name are configured in git config.

Run the following to setup the code review tool and create your first review:

git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git $HOME/src/depot_tools
export PATH="$PATH:$HOME/src/depot_tools"
git checkout -b work origin/master

# hack hack

git commit -a -m "This is awesome"
# This will ask for your Google Account credentials.
git cl upload -s -r joe@example.com
# Wait for approval over email.
# Click "Submit to CQ" button or ask reviewer to do it for you.
# Wait for the change to be tested and landed automatically.

Use git cl help and git cl help <cmd> for more details.