| commit | 103411aea8b9331f783aea49814962031b254c54 | [log] [tgz] |
|---|---|---|
| author | Yiwei Zhang <yiwzhang@google.com> | Mon Oct 26 20:10:09 2020 |
| committer | LUCI CQ <infra-scoped@luci-project-accounts.iam.gserviceaccount.com> | Mon Oct 26 20:10:09 2020 |
| tree | 01e68399f46d69aa21c4322df9f472bb0d53c17f | |
| parent | c210466b5c2bcb4d94191efa30d5d1a6f7ebbcc8 [diff] |
[Deadline] Add deadline and SIGTERM support to recipe_engine. This makes the recipe engine react to, and manipulate, LUCI_CONTEXT['deadline']. The engine reserves .5s for itself from both the soft_deadline and grace_period. On SIGTERM it activates a pressure-release valve of sorts (GLOBAL_SHUTDOWN) which will immediately move all currently running subprocesses to the SIGTERM-and-wait state. After `grace_period` (as tracked by the `context` module) seconds, the processes move to SIGKILL. Due to historical reasons, the recipe engine currently must allow grandchild processes to leak past the return of the launched process. The most notable instance of this is the goma compiler daemon which intentionally generates a leaked process. Because of this, processes which successfully exit have their pgid saved to a global set of 'unkilled' process groups, which will be killed `grace_period` seconds after termination (or on the return from Engine.run_steps, whichever is sooner). (In some future world where all builders use luciexe mode, the futures module could be used to create a background green-thread which would allow the goma compiler proxy to run as a normal subprocess with normal logging output. At this point we could move the 'killpg' operation unconditionally into the subprocess runner and get rid of the 'unkilled' set.) Note also that recipe_engine does only primitive encapsulation for Windows process groups and so cannot reliably emulate SIGKILL. The engine could fix this in the future by doing the same Job Object manipulation done by swarming's subprocess42, however. R=tikuta, vadimsh, yiwzhang Bug: 1127089 Change-Id: I22b7a48857fee5fc4ce220a3420ed390e2fbcf66 Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/2438981 Reviewed-by: Yiwei Zhang <yiwzhang@google.com> Commit-Queue: Yiwei Zhang <yiwzhang@google.com>
Recipes are a domain-specific language (embedded in python) for specifying sequences of subprocess calls in a cross-platform and testable way.
They allow writing build flows which integrate with the rest of LUCI.
Documentation for the recipe engine (including this file!). Take a look at the user guide for some hints on how to get started. See the implementation details doc for more detailed implementation information about the recipe engine.
user.email and user.name are configured in git config.Run the following to setup the code review tool and create your first review:
# Get `depot_tools` in $PATH if you don't have it git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git $HOME/src/depot_tools export PATH="$PATH:$HOME/src/depot_tools" # Check out the recipe engine repo git clone https://chromium.googlesource.com/infra/luci/recipes-py $HOME/src/recipes-py # make your change cd $HOME/src/recipes-py git new-branch cool_feature # hack hack git commit -a -m "This is awesome" # This will ask for your Google Account credentials. git cl upload -s -r joe@example.com # Wait for approval over email. # Click "Submit to CQ" button or ask reviewer to do it for you. # Wait for the change to be tested and landed automatically.
Use git cl help and git cl help <cmd> for more details.