Rewrite candidate_algorithm.

The current candidate algorithm does not consider all possible valid
configurations. As an example, if there is a diamond dependency e.g. X
-> A -> C and X -> B -> C, when a change lands in C and is rolled into A
and B, the algorithm will not produce a candidate that uses C's revision
value unless there is a subsequent change in either A or B. The reason
for this is that despite the other downstreams being considered when
scoring a repo to advance, only dependencies of the repo to advance are
updated. Other repos that are downstream of those dependencies are not
adjusted to make the config consistent. So none of configs will be
accepted and the commit lists will be advanced to try again. At that
point, if either A or B has another commit that uses the same change
from C, a consistent config can be created.

This change modifies the algorithm to instead produce all of the
consistent configs that can be created with one of the top-level repos
being kept at its current revision; inconsistent configs will never be
created to be scored/evaluated. The portion of code that creates the
algorithm depends only on the contents of the commit lists (not the
positions within the commit list), the initial commit to pin and the set
of repos that need to be pinned. Since the set of repos that need to be
pinned changes rarely, the results of these calculations can be cached
so that when there are multiple top level deps, work to create the
configs for a pin that wasn't moved can be re-used. Scoring is kept
separate since it depends on the currently applied config, so it must be
re-evaluated once a new config has been accepted.

As part of this change, the tracking of current positions and advancing
was moved into a _Cursor class nested in CommitList. The cursors are
only used to mark where to create candidate commits from; the current
config is tracked separately. This removes the need to repeatedly copy
the commit lists and removes the possibility for callers to depend on
the position within the commit lists that the algorithm ends at.

Bug: 1309920
Change-Id: Ida6653ea30d3dca42b42e5a3b13aeb486138dc27
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/3561638
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Commit-Queue: Garrett Beaty <gbeaty@google.com>
4 files changed
tree: 75eb8b3bdd93063b9cb9a20878d14388c3e0e400
  1. doc/
  2. infra/
  3. misc/
  4. recipe_engine/
  5. recipe_modules/
  6. recipe_proto/
  7. recipes/
  8. unittests/
  9. .gitattributes
  10. .gitignore
  11. .style.yapf
  12. .vpython
  13. .vpython3
  14. .vpython3_dev
  15. AUTHORS
  16. codereview.settings
  17. CONTRIBUTORS
  18. LICENSE
  19. OWNERS
  20. PRESUBMIT.py
  21. README.md
  22. README.recipes.md
  23. recipe.warnings
  24. recipes.py
README.md

Recipes

Recipes are a domain-specific language (embedded in Python) for specifying sequences of subprocess calls in a cross-platform and testable way.

They allow writing build flows which integrate with the rest of LUCI.

Documentation for the recipe engine (including this file!). Take a look at the user guide for some hints on how to get started. See the implementation details doc for more detailed implementation information about the recipe engine.

Contributing

  • Sign the Google CLA.
  • Make sure your user.email and user.name are configured in git config.

Run the following to setup the code review tool and create your first review:

# Get `depot_tools` in $PATH if you don't have it
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git $HOME/src/depot_tools
export PATH="$PATH:$HOME/src/depot_tools"

# Check out the recipe engine repo
git clone https://chromium.googlesource.com/infra/luci/recipes-py $HOME/src/recipes-py

# make your change
cd $HOME/src/recipes-py
git new-branch cool_feature
# hack hack
git commit -a -m "This is awesome"

# This will ask for your Google Account credentials.
git cl upload -s -r joe@example.com
# Wait for approval over email.
# Click "Submit to CQ" button or ask reviewer to do it for you.
# Wait for the change to be tested and landed automatically.

Use git cl help and git cl help <cmd> for more details.