Package documentation for recipe_engine

buildbucket — API for interacting with the buildbucket service.
context — The context module provides APIs for manipulating a few pieces of ‘ambient’ data that affect how steps are run.
file — File manipulation (read/write/delete/glob) methods.
generator_script — A simplistic method for running steps generated by an external script.
json — Methods for producing and consuming JSON.
path — All functions related to manipulating paths in recipes.
platform — Mockable system platform identity functions.
properties — Provides access to the recipes input properties.
python — Provides methods for running python scripts correctly.
raw_io — Provides objects for reading and writing raw data to and from steps.
runtime
scheduler — API for interacting with the LUCI Scheduler service.
service_account — API for getting OAuth2 access tokens for LUCI tasks or private keys.
source_manifest
step — Step is the primary API for running steps (external programs, scripts, etc.
tempfile — Simplistic temporary directory manager (deprecated).
time — Allows mockable access to the current time.
url — Methods for interacting with HTTP(s) URLs.

buildbucket:examples/full — This file is a recipe demonstrating the buildbucket recipe module.
buildbucket:tests/build_id
buildbucket:tests/get
buildbucket:tests/properties
buildbucket:tests/put
context:examples/full
context:tests/cwd
context:tests/env
context:tests/increment_nest_level
context:tests/infra_step
engine_tests/bad_subprocess — Tests that daemons that hang on to STDOUT can't cause the engine to hang.
engine_tests/depend_on/bad_properties
engine_tests/depend_on/bottom
engine_tests/depend_on/dont_need_properties
engine_tests/depend_on/dont_need_properties_helper
engine_tests/depend_on/need_return_schema
engine_tests/depend_on/need_return_schema_helper
engine_tests/depend_on/no_return
engine_tests/depend_on/top
engine_tests/expect_exception — Tests that step_data can accept multiple specs at once.
engine_tests/functools_partial — Engine shouldn't explode when step_test_data gets functools.
engine_tests/missing_start_dir — Tests that deleting the current working directory doesn't immediately fail.
engine_tests/module_injection_site — This test serves to demonstrate that the ModuleInjectionSite object on recipe modules (i.
engine_tests/multi_test_data — Tests that step_data can accept multiple specs at once.
engine_tests/nonexistent_command
engine_tests/recipe_paths — Tests that recipes have access to names, resources and their package.
engine_tests/sort_properties — Tests that step presentation properties can be ordered.
engine_tests/step_stack_exhaustion — Tests that placeholders can't wreck the world by exhausting the step stack.
engine_tests/undeclared_method
engine_tests/unicode
engine_tests/whitelist_steps — Tests that step_data can accept multiple specs at once.
file:examples/copy
file:examples/copytree
file:examples/error
file:examples/glob
file:examples/listdir
file:examples/raw_copy
file:examples/symlink
file:examples/truncate
generator_script:examples/full
json:examples/full
json:tests/add_json_log
path:examples/full
platform:examples/full
properties:examples/full
python:examples/full — Launches the repo bundler.
raw_io:examples/full
runtime:tests/full
scheduler:examples/emit_triggers — This file is a recipe demonstrating emitting triggers to LUCI Scheduler.
service_account:examples/full
source_manifest:examples/simple
step:examples/full
step:tests/active_result
step:tests/defer
step:tests/inject_paths
step:tests/nested
step:tests/stdio
step:tests/step_call_args
step:tests/subannotations
step:tests/timeout
step:tests/trigger
tempfile:examples/full
time:examples/full
url:examples/full
url:tests/join
url:tests/validate_url

Recipe Modules

recipe_modules / buildbucket

DEPS: json, platform, properties, raw_io, runtime, step

API for interacting with the buildbucket service.

Depends on ‘buildbucket’ binary available in PATH: https://godoc.org/go.chromium.org/luci/buildbucket/client/cmd/buildbucket

class BuildbucketApi(RecipeApi):

A module for interacting with buildbucket.

@property
— def build_id(self):

Returns int64 identifier of the current build.

It is unique per buildbucket instance. In practice, it means globally unique.

May return None if it is not a buildbucket build.

— def cancel_build(self, build_id, **kwargs):

— def get_build(self, build_id, **kwargs):

@property
— def properties(self):

Returns (dict-like or None): The BuildBucket properties, if present.

— def put(self, builds, **kwargs):

Puts a batch of builds.

Args: builds (list): A list of dicts, where keys are: ‘bucket’: (required) name of the bucket for the request. ‘parameters’ (dict): (required) arbitrary json-able parameters that a build system would be able to interpret. ‘tags’: (optional) a dict(str->str) of tags for the build. These will be added to those generated by this method and override them if appropriate. If you need to remove a tag set by default, set its value to None (for example, tags={‘buildset’: None} will ensure build is triggered without ‘buildset’ tag).

Returns: A step that as its .stdout property contains the response object as returned by buildbucket.

— def set_buildbucket_host(self, host):

Changes the buildbucket backend hostname used by this module.

Args: host (str): buildbucket server host (e.g. ‘cr-buildbucket.appspot.com’).

@property
— def tags_for_child_build(self):

A dict of tags (key -> value) derived from current (parent) build for a child build.

— def use_service_account_key(self, key_path):

Tells this module to start using given service account key for auth.

Otherwise the module is using the default account (when running on LUCI or locally), or no auth at all (when running on Buildbot).

Exists mostly to support Buildbot environment. Recipe for LUCI environment should not use this.

Args: key_path (str): a path to JSON file with service account credentials.

recipe_modules / context

DEPS: path

The context module provides APIs for manipulating a few pieces of ‘ambient’ data that affect how steps are run.

The pieces of information which can be modified are:

cwd - The current working directory.
env - The environment variables.
infra_step - Whether or not failures should be treated as infrastructure failures vs. normal failures.
name_prefix - A prefix for all step names.
nest_level - An indicator for the UI of how deeply to nest steps.

The values here are all scoped using Python‘s with statement; there’s no mechanism to make an open-ended adjustment to these values (i.e. there's no way to change the cwd permanently for a recipe, except by surrounding the entire recipe with a with statement). This is done to avoid the surprises that typically arise with things like os.environ or os.chdir in a normal python program.

Example:

with api.context(cwd=api.path['start_dir'].join('subdir')):
  # this step is run inside of the subdir directory.
  api.step("cat subdir/foo", ['cat', './foo'])

class ContextApi(RecipeApi):

@contextmanager
— def __call__(self, cwd=None, env_prefixes=None, env_suffixes=None, env=None, increment_nest_level=None, infra_steps=None, name_prefix=None):

Allows adjustment of multiple context values in a single call.

Args:

cwd (Path) - the current working directory to use for all steps. To ‘reset’ to the original cwd at the time recipes started, pass api.path['start_dir'].
env_prefixes (dict) - Environmental variable prefix augmentations. See below for more info.
env_suffixes (dict) - Environmental variable suffix augmentations. See below for more info.
env (dict) - Environmental variable overrides. See below for more info.
increment_nest_level (True) - increment the nest level by 1 in this context. Typically you won't directly interact with this, but should use api.step.nest instead.
infra_steps (bool) - if steps in this context should be considered infrastructure steps. On failure, these will raise InfraFailure exceptions instead of StepFailure exceptions.
name_prefix (str) - A string to prepend to the names of all steps in this context. These compose with ‘.’ characters if multiple name prefix contexts occur. See below for more info.

Name prefixes:

Multiple invocations concatenate values with ‘.’.

Example:

with api.context(name_prefix='hello'):
  # has name 'hello.something'
  api.step('something', ['echo', 'something'])

  with api.context(name_prefix='world'):
    # has name 'hello.world.other'
    api.step('other', ['echo', 'other'])

Environmental Variable Overrides:

Env is a mapping of environment variable name to the value you want that environment variable to have. The value is one of:

None, indicating that the environment variable should be removed from the environment when the step runs.
A string value. Note that string values will be %-formatted with the current value of the environment at the time the step runs. This means that you can have a value like: “/path/to/my/stuff:%(PATH)s” Which, at the time the step executes, will inject the current value of $PATH.

“env_prefix” and “env_suffix” are a list of Path or strings that get prefixed (or suffixed) to their respective environment variables, delimited with the system's path separator. This can be used to add entries to environment variables such as “PATH” and “PYTHONPATH”. If prefixes are specified and a value is also defined in “env”, the value will be installed as the last path component if it is not empty.

TODO(iannucci): combine nest_level and name_prefix

Look at the examples in “examples/” for examples of context module usage.

@property
— def cwd(self):

Returns the current working directory that steps will run in.

Returns (Path|None) - The current working directory. A value of None is equivalent to api.path[‘start_dir’], though only occurs if no cwd has been set (e.g. in the outermost context of RunSteps).

@property
— def env(self):

Returns modifications to the environment.

By default this is empty; There‘s no facility to observe the program’s startup environment. If you want to pass data to the recipe, it should be done with properties.

Returns (dict) - The env-key -> value mapping of current environment modifications.

@property
— def env_prefixes(self):

Returns Path prefix modifications to the environment.

This will return a mapping of environment key to Path tuple for Path prefixes registered with the environment.

Returns (dict) - The env-key -> value(Path) mapping of current environment prefix modifications.

@property
— def env_suffixes(self):

Returns Path suffix modifications to the environment.

This will return a mapping of environment key to Path tuple for Path suffixes registered with the environment.

Returns (dict) - The env-key -> value(Path) mapping of current environment suffix modifications.

@property
— def infra_step(self):

Returns the current value of the infra_step setting.

Returns (bool) - True iff steps are currently considered infra steps.

@property
— def name_prefix(self):

Gets the current step name prefix.

Returns (str) - The string prefix that every step will have prepended to it.

@property
— def nest_level(self):

Returns the current ‘nesting’ level.

Note: This api is low-level, and you should always prefer to use api.step.nest. This api is included for completeness and documentation purposes.

Returns (int) - The current nesting level.

recipe_modules / file

DEPS: json, path, python, raw_io, step

File manipulation (read/write/delete/glob) methods.

class FileApi(RecipeApi):

— def copy(self, name, source, dest):

Copies a file (including mode bits) from source to destination on the local filesystem.

Behaves identically to shutil.copy.

Args:

name (str) - The name of the step.
source (Path|Placeholder) - The path to the file you want to copy.
dest (Path|Placeholder) - The path to the destination file name. If this path exists and is a directory, the basename of source will be appended to derive a path to a destination file.

Raises file.Error

— def copytree(self, name, source, dest, symlinks=False):

Recursively copies a directory tree.

Behaves identically to shutil.copytree. dest must not exist.

Args:

name (str) - The name of the step.
source (Path) - The path of the directory to copy.
dest (Path) - The place where you want the recursive copy to show up. This must not already exist.
symlinks (bool) - Preserve symlinks. No effect on Windows.

Raises file.Error

— def ensure_directory(self, name, dest, mode=511):

Ensures that dest exists and is a directory.

Args:

name (str) - The name of the step.
dest (Path) - The directory to ensure.
mode (int) - The mode to use if the directory doesn't exist. This method does not ensure the mode if the directory already exists (if you need that behaviour, file a bug).

Raises file.Error if the path exists but is not a directory.

— def filesizes(self, name, files, test_data=None):

Returns list of filesizes for the given files.

Args:

name (str) - The name of the step.
files (list[Path]) - Paths to files.

Returns list[int], size of each file in bytes.

— def glob_paths(self, name, source, pattern, test_data=()):

Performs glob expansion on pattern.

glob rules for pattern follow the same syntax as for the python glob stdlib module.

Args:

name (str) - The name of the step.
source (Path) - The directory whose contents should be globbed.
pattern (str) - The glob pattern to apply under source.
test_data (iterable[str]) - Some default data for this step to return when running under simulation. This should be the list of file items found in this directory.

Returns list[Path] - All paths found.

Raises file.Error.

— def listdir(self, name, source, test_data=()):

List all files inside a directory.

Args:

name (str) - The name of the step.
source (Path) - The directory to list.
test_data (iterable[str]) - Some default data for this step to return when running under simulation. This should be the list of file items found in this directory.

Returns list[Path]

Raises file.Error.

— def move(self, name, source, dest):

Moves a file or directory.

Behaves identically to shutil.move.

Args:

name (str) - The name of the step.
source (Path) - The path of the item to move.
dest (Path) - The new name of the item.

Raises file.Error

— def read_raw(self, name, source, test_data=''):

Reads a file as raw data.

Args:

name (str) - The name of the step.
source (Path) - The path of the file to read.
test_data (str) - Some default data for this step to return when running under simulation.

Returns (str) - The unencoded (binary) contents of the file.

Raises file.Error

— def read_text(self, name, source, test_data=''):

Reads a file as UTF-8 encoded text.

Args:

name (str) - The name of the step.
source (Path) - The path of the file to read.
test_data (str) - Some default data for this step to return when running under simulation.

Returns (str) - The content of the file.

Raises file.Error

— def remove(self, name, source):

Remove a file.

Does not raise Error if the file doesn't exist.

Args:

name (str) - The name of the step.
source (Path) - The file to remove.

Raises file.Error.

— def rmcontents(self, name, source):

Similar to rmtree, but removes only contents not the directory.

This is useful e.g. when removing contents of current working directory. Deleting current working directory makes all further getcwd calls fail until chdir is called. chdir would be tricky in recipes, so we provide a call that doesn't delete the directory itself.

Args:

name (str) - The name of the step.
source (Path) - The directory whose contents should be removed.

Raises file.Error.

— def rmglob(self, name, source, pattern):

Removes all entries in source matching the glob pattern.

Args:

name (str) - The name of the step.
source (Path) - The directory whose contents should be filtered and removed.
pattern (str) - The glob pattern to apply under source. Anything matching this pattern will be removed.

Raises file.Error.

— def rmtree(self, name, source):

Recursively removes a directory.

This uses a native python on Linux/Mac, and uses rd on Windows to avoid issues w.r.t. path lengths and read-only attributes. If the directory is gone already, this returns without error.

Args:

name (str) - The name of the step.
source (Path) - The directory to remove.

Raises file.Error.

— def symlink(self, name, source, link):

Creates a symlink from link to source on the local filesystem.

Behaves identically to os.symlink.

Args:

name (str) - The name of the step.
source (Path|Placeholder) - The path to link to.
link (Path|Placeholder) - The link to create.

Raises file.Error

— def truncate(self, name, path, size_mb=100):

Creates an empty file with path and size_mb on the local filesystem.

Args:

name (str) - The name of the step.
path (Path|str) - The absolute path to create.
size_mb (int) - The size of the file in megabytes. Defaults to 100

Raises file.Error

— def write_raw(self, name, dest, data):

Write the given data to dest.

Args:

name (str) - The name of the step.
dest (Path) - The path of the file to write.
data (str) - The data to write.

Raises file.Error.

— def write_text(self, name, dest, text_data):

Write the given UTF-8 encoded text_data to dest.

Args:

name (str) - The name of the step.
dest (Path) - The path of the file to write.
text_data (str) - The UTF-8 encoded data to write.

Raises file.Error.

recipe_modules / generator_script

DEPS: context, json, path, python, step

A simplistic method for running steps generated by an external script.

This module was created before there was a way to put recipes directly into another repo. It is not recommended to use this, and it will be removed in the near future.

class GeneratorScriptApi(RecipeApi):

— def __call__(self, path_to_script, *args):

Run a script and generate the steps emitted by that script.

The script will be invoked with --output-json /path/to/file.json. The script is expected to exit 0 and write steps into that file. Once the script outputs all of the steps to that file, the recipe will read the steps from that file and execute them in order. Any *args specified will be additionally passed to the script.

The step data is formatted as a list of JSON objects. Each object corresponds to one step, and contains the following keys:

name: the name of this step.
cmd: a list of strings that indicate the command to run (e.g. argv)
env: a {key:value} dictionary of the environment variables to override. every value is formatted with the current environment with the python % operator, so a value of “%(PATH)s:/some/other/path” would resolve to the current PATH value, concatenated with “:/some/other/path”
cwd: an absolute path to the current working directory for this script.
always_run: a bool which indicates that this step should run, even if some previous step failed.
outputs_presentation_json: a bool which indicates that this step will emit a presentation json file. If this is True, the cmd will be extended with a --presentation-json /path/to/file.json. This file will be used to update the step's presentation on the build status page. The file will be expected to contain a single json object, with any of the following keys:
- logs: {logname: [lines]} specifies one or more auxillary logs.
- links: {link_name: link_content} to add extra links to the step.
- step_summary_text: A string to set as the step summary.
- step_text: A string to set as the step text.
- properties: {prop: value} build_properties to add to the build status page. Note that these are write-only: The only way to read them is via the status page. There is intentionally no mechanism to read them back from inside of the recipes.
allow_subannotations: allow this step to emit legacy buildbot subannotations. If you don‘t know what this is, you shouldn’t use it. If you know what it is, you also shouldn't use it.

recipe_modules / json

DEPS: properties, python, raw_io

Methods for producing and consuming JSON.

class JsonApi(RecipeApi):

@returns_placeholder
— def input(self, data):

A placeholder which will expand to a file path containing .

— def is_serializable(self, obj):

Returns True if the object is JSON-serializable.

@staticmethod
— def loads(data, **kwargs):

Works like json.loads, but strips out unicode objects (replacing them with utf8-encoded str objects).

@returns_placeholder
— def output(self, add_json_log=True, name=None, leak_to=None):

A placeholder which will expand to ‘/tmp/file’.

If leak_to is provided, it must be a Path object. This path will be used in place of a random temporary file, and the file will not be deleted at the end of the step.

Args:

add_json_log (True|False|‘on_failure’) - Log a copy of the output json to a step link named name. If this is ‘on_failure’, only create this log when the step has a non-SUCCESS status.

— def read(self, name, path, add_json_log=True, output_name=None, **kwargs):

Returns a step that reads a JSON file.

recipe_modules / path

DEPS: platform, properties

All functions related to manipulating paths in recipes.

Recipes handle paths a bit differently than python does. All path manipulation in recipes revolves around Path objects. These objects store a base path (always absolute), plus a list of components to join with it. New paths can be derived by calling the .join method with additional components.

In this way, all paths in Recipes are absolute, and are constructed from a small collection of anchor points. The built-in anchor points are:

api.path['start_dir'] - This is the directory that the recipe started in. it‘s similar to cwd, except that it’s constant.
api.path['cache'] - This directory is provided by whatever's running the recipe. Files and directories created under here /may/ be evicted in between runs of the recipe (i.e. to relieve disk pressure).
api.path['cleanup'] - This directory is provided by whatever's running the recipe. Files and directories created under here /are guaranteed/ to be evicted in between runs of the recipe. Additionally, this directory is guaranteed to be empty when the recipe starts.
api.path['tmp_base'] - This directory is the system-configured temp dir. This is a weaker form of ‘cleanup’, and its use should be avoided. This may be removed in the future (or converted to an alias of ‘cleanup’).
api.path['checkout'] - This directory is set by various ‘checkout’ modules in recipes. It was originally intended to make recipes easier to read and make code somewhat generic or homogenous, but this was a mistake. New code should avoid ‘checkout’, and instead just explicitly pass paths around. This path may be removed in the future.

There are other anchor points which can be defined (e.g. by the depot_tools/infra_paths module). Refer to those modules for additional documentation.

class PathApi(RecipeApi):

— def __getitem__(self, name):

Gets the base path named name. See module docstring for more information.

— def abs_to_path(self, abs_string_path):

Converts an absolute path string string_path to a real Path object, using the most appropriate known base path.

abs_string_path MUST be an absolute path
abs_string_path MUST be rooted in one of the configured base paths known to the path module.

This method will find the longest match in all the following:

module resource paths
recipe resource paths
package repo paths
dynamic_paths
base_paths

Example:

# assume [START_DIR] == "/basis/dir/for/recipe"
api.path.abs_to_path("/basis/dir/for/recipe/some/other/dir") ->
  Path("[START_DIR]/some/other/dir")

Raises an ValueError if the preconditions are not met, otherwise returns the Path object.

— def assert_absolute(self, path):

Raises AssertionError if the given path is not an absolute path.

Args:

path (Path|str) - The path to check.

— def get(self, name, default=None):

Gets the base path named name. See module docstring for more information.

— def get_config_defaults(self):

Internal recipe implementation function.

— def initialize(self):

Internal recipe implementation function.

— def mkdtemp(self, prefix):

Makes a new temporary directory, returns Path to it.

— def mock_add_paths(self, path):

For testing purposes, mark that |path| exists.

— def mock_copy_paths(self, source, dest):

For testing purposes, copy |source| to |dest|.

— def mock_remove_paths(self, path, filt=(lambda p: True)):

For testing purposes, assert that |path| doesn't exist.

Args:

path (str|Path) - The path to remove.
filt (func[str] bool) - Called for every candidate path. Return True to remove this path.

recipe_modules / platform

Mockable system platform identity functions.

class PlatformApi(RecipeApi):

Provides host-platform-detection properties.

Mocks:

name (str): A value equivalent to something that might be returned by sys.platform.
bits (int): Either 32 or 64.

@property
— def arch(self):

Returns the current CPU architecture.

TODO: This is currently always hard-coded to ‘intel’... Apparently no one has actually needed this function?

@property
— def bits(self):

Returns the bitness of the userland for the current system (either 32 or 64 bit).

TODO: If anyone needs to query for the kernel bitness, another accessor should be added.

@property
— def cpu_count(self):

The number of CPU cores, according to multiprocessing.cpu_count().

@property
— def is_linux(self):

Returns True iff the recipe is running on Linux.

@property
— def is_mac(self):

Returns True iff the recipe is running on OS X.

@property
— def is_win(self):

Returns True iff the recipe is running on Windows.

@property
— def name(self):

Returns the current platform name which will be in:

win
mac
linux

@staticmethod
— def normalize_platform_name(plat):

One of python's sys.platform values -> ‘win’, ‘linux’ or ‘mac’.

recipe_modules / properties

Provides access to the recipes input properties.

Every recipe is run with a JSON object called “properties”. These contain all inputs to the recipe. Some common examples would be properties like “revision”, which the build scheduler sets to tell a recipe to build/test a certain revision.

The properties that affect a particular recipe are defined by the recipe itself, and this module provides access to them.

Recipe properties are read-only; the values obtained via this API reflect the values provided to the recipe engine at the beginning of execution. There is intentionally no API to write property values (lest they become a kind of random-access global variable).

class PropertiesApi(RecipeApiPlain, collections.Mapping):

PropertiesApi implements all the standard Mapping functions, so you can use it like a read-only dict.

— def legacy(self):

DEPRECATED: Returns a set of properties, possibly used by legacy scripts.

This excludes any recipe module-specific properties (i.e. those beginning with $).

Instead of passing all of the properties as a blob, please consider passing specific arguments to scripts that need them. Doing this makes it much easier to debug and diagnose which scripts use which properties.

— def thaw(self):

Returns a read-write copy of all of the properties.

recipe_modules / python

DEPS: context, raw_io, step

Provides methods for running python scripts correctly.

This includes support for vpython, and knows how to specify parameters correctly for bots (e.g. ensuring that python is working on Windows, passing the unbuffered flag, etc.)

class PythonApi(RecipeApi):

— def __call__(self, name, script, args=None, unbuffered=True, venv=None, **kwargs):

Return a step to run a python script with arguments.

TODO: We should just use a single “args” list. Having “script” separate but required/first leads to weird things like:

(... script='-m', args=['module'])

Args:

name (str): The name of the step.
script (str or Path): The Path of the script to run, or the first command-line argument to pass to Python.
args (list or None): If not None, additional arguments to pass to the Python command.
unbuffered (bool): If True, run Python in unbuffered mode.
venv (None or False or True or Path): If True, run the script through “vpython”. This will, by default, probe the target script for a configured VirtualEnv and, failing that, use an empty VirtualEnv. If a Path, this is a path to an explicit “vpython” VirtualEnv spec file to use. If False or None (default), the script will be run through the standard Python interpreter.
kwargs: Additional keyword arguments to forward to “step”.

Returns (types.StepData) - The StepData object as returned by api.step.

— def failing_step(self, name, text, as_log=None):

Runs a succeeding step (exits 1).

— def inline(self, name, program, add_python_log=True, **kwargs):

Run an inline python program as a step.

Program is output to a temp file and run when this step executes.

Args:

name (str) - The name of the step
program (str) - The literal python program text. This will be dumped to a file and run like python /path/to/file.py
add_python_log (bool) - Whether to add a ‘python.inline’ link on this step on the build page. If true, the link will point to a log with a copy of program.

Returns (types.StepData) - The StepData object as returned by api.step.

— def result_step(self, name, text, retcode, as_log=None):

Runs a no-op step that exits with a specified return code.

— def succeeding_step(self, name, text, as_log=None):

Runs a succeeding step (exits 0).

recipe_modules / raw_io

Provides objects for reading and writing raw data to and from steps.

class RawIOApi(RecipeApi):

@returns_placeholder
@staticmethod
— def input(data, suffix=''):

Returns a Placeholder for use as a step argument.

This placeholder can be used to pass data to steps. The recipe engine will dump the ‘data’ into a file, and pass the filename to the command line argument.

data MUST be of type ‘str’ (not basestring, not unicode).

If ‘suffix’ is not '', it will be used when the engine calls tempfile.mkstemp.

See examples/full.py for usage example.

@returns_placeholder
@staticmethod
— def input_text(data, suffix=''):

Returns a Placeholder for use as a step argument.

data MUST be of type ‘str’ (not basestring, not unicode). The str is expected to have valid utf-8 data in it.

Similar to input(), but ensures that ‘data’ is valid utf-8 text. Any non-utf-8 characters will be replaced with �.

@returns_placeholder
@staticmethod
— def output(suffix='', leak_to=None, name=None, add_output_log=False):

Returns a Placeholder for use as a step argument, or for std{out,err}.

If ‘leak_to’ is None, the placeholder is backed by a temporary file with a suffix ‘suffix’. The file is deleted when the step finishes.

If ‘leak_to’ is not None, then it should be a Path and placeholder redirects IO to a file at that path. Once step finishes, the file is NOT deleted (i.e. it's ‘leaking’). ‘suffix’ is ignored in that case.

Args:

add_output_log (True|False|‘on_failure’) - Log a copy of the output to a step link named name. If this is ‘on_failure’, only create this log when the step has a non-SUCCESS status.

@returns_placeholder
@staticmethod
— def output_dir(suffix='', leak_to=None, name=None):

Returns a directory Placeholder for use as a step argument.

If ‘leak_to’ is None, the placeholder is backed by a temporary dir with a suffix ‘suffix’. The dir is deleted when the step finishes.

If ‘leak_to’ is not None, then it should be a Path and placeholder redirects IO to a dir at that path. Once step finishes, the dir is NOT deleted (i.e. it's ‘leaking’). ‘suffix’ is ignored in that case.

@returns_placeholder
@staticmethod
— def output_text(suffix='', leak_to=None, name=None):

Returns a Placeholder for use as a step argument, or for std{out,err}.

Similar to output(), but uses an OutputTextPlaceholder, which expects utf-8 encoded text. Similar to input(), but tries to decode the resulting data as utf-8 text, replacing any decoding errors with �.

recipe_modules / runtime

DEPS: path, properties

class RuntimeApi(RecipeApi):

This module assists in experimenting with production recipes.

For example, when migrating builders from Buildbot to pure LUCI stack.

@property
— def is_experimental(self):

True if this recipe is currently running in experimental mode.

Typical usage is to modify steps which produce external side-effects so that non-production runs of the recipe do not affect production data.

Examples:

Uploading to an alternate google storage file name when in non-prod mode
Appending a ‘non-production’ tag to external RPCs

@property
— def is_luci(self):

True if this recipe is currently running on LUCI stack.

Should be used only during migration from Buildbot to LUCI stack.

recipe_modules / scheduler

DEPS: buildbucket, json, platform, properties, raw_io, runtime, step, time

API for interacting with the LUCI Scheduler service.

Depends on ‘prpc’ binary available in $PATH: https://godoc.org/go.chromium.org/luci/grpc/cmd/prpc Documentation for scheduler API is in https://chromium.googlesource.com/infra/luci/luci-go/+/master/scheduler/api/scheduler/v1/scheduler.proto RPCExplorer available at https://luci-scheduler.appspot.com/rpcexplorer/services/scheduler.Scheduler

class SchedulerApi(RecipeApi):

A module for interacting with LUCI Scheduler service.

— def emit_trigger(self, trigger, project, jobs, step_name=None):

Emits trigger to one or more jobs of a given project.

Args: trigger (Trigger): defines payload to trigger jobs with. project (str): name of the project in LUCI Config service, which is used by LUCI Scheduler instance. See https://luci-config.appspot.com/. jobs (iterable of str): job names per LUCI Scheduler config for the given project. These typically are the same as builder names.

— def emit_triggers(self, trigger_project_jobs, timestamp_usec=None, step_name=None):

Emits a batch of triggers spanning one or more projects.

Up to date documentation is at https://chromium.googlesource.com/infra/luci/luci-go/+/master/scheduler/api/scheduler/v1/scheduler.proto

Args: trigger_project_jobs (iterable of tuples(trigger, project, jobs)): each tuple corresponds to parameters of emit_trigger API above. timestamp_usec (int): unix timestamp in microseconds. Useful for idempotency of calls if your recipe is doing its own retries. https://chromium.googlesource.com/infra/luci/luci-go/+/master/scheduler/api/scheduler/v1/triggers.proto

— def set_host(self, host):

Changes the backend hostname used by this module.

Args: host (str): server host (e.g. ‘luci-scheduler.appspot.com’).

recipe_modules / service_account

DEPS: path, platform, raw_io, step

API for getting OAuth2 access tokens for LUCI tasks or private keys.

This is a thin wrapper over the authutil go executable, which itself calls https://godoc.org/go.chromium.org/luci/client/authcli

Depends on authutil to be in PATH.

class ServiceAccountApi(RecipeApi):

— def default(self):

Returns an account associated with the task.

On LUCI, this is default account exposed through LUCI_CONTEXT[“local_auth”] protocol. When running locally this is an account the user logged in via “authutil login ...” command prior to running the recipe.

— def from_credentials_json(self, key_path):

Returns a service account based on a JSON credentials file.

This is the file generated by Cloud Console when creating a service account key. It contains the private key inside.

Args: key_path: (str|Path) object pointing to a service account JSON key.

recipe_modules / source_manifest

class SourceManfiestApi(RecipeApi):

— def set_json_manifest(self, name, data):

Uploads a source manifest with the given name.

NOTE: Due to current implementation restrictions, this method may only be called after some step has been run from the recipe. Calling this before running any steps is invalid and will fail. We hope to lift this restriction sometime after we don't need to support buildbot any more.

TODO(iannucci): remove this restriction.

Args:

name (str) - the name of the manifest. These names must be valid LogDog stream names, and must be unique within a recipe run. e.g.
- “main_checkout”
- “bisect/deadbeef”
data (dict) - the JSONPB representation of the source_manifest.proto Manifest message.

recipe_modules / step

DEPS: context, path

Step is the primary API for running steps (external programs, scripts, etc.).

class StepApi(RecipeApiPlain):

@property
— def InfraFailure(self):

InfraFailure is a subclass of StepFailure, and will translate to a purple build.

This exception is raised from steps which are marked as infra_steps when they fail.

@property
— def StepFailure(self):

This is the base Exception class for all step failures.

It can be manually raised from recipe code to cause the build to turn red.

Usage:

raise api.StepFailure("some reason")
except api.StepFailure:

@property
— def StepTimeout(self):

StepTimeout is a subclass of StepFailure and is raised when a step times out.

@property
— def StepWarning(self):

StepWarning is a subclass of StepFailure, and will translate to a yellow build.

@recipe_api.composite_step
— def __call__(self, name, cmd, ok_ret=None, infra_step=False, wrapper=(), timeout=None, allow_subannotations=None, trigger_specs=None, stdout=None, stderr=None, stdin=None, step_test_data=None):

Returns a step dictionary which is compatible with annotator.py.

Args:

name (string): The name of this step.
cmd (list of strings): in the style of subprocess.Popen or None to create a no-op fake step.
ok_ret (tuple or set of ints, str): allowed return codes. Any unexpected return codes will cause an exception to be thrown. If you pass in the value ‘any’ or ‘all’, the engine will allow any return code to be returned. Defaults to {0}
infra_step: Whether or not this is an infrastructure step. Infrastructure steps will place the step in an EXCEPTION state and raise InfraFailure.
wrapper: If supplied, a command to prepend to the executed step as a command wrapper.
timeout: If supplied, the recipe engine will kill the step after the specified number of seconds.
allow_subannotations (bool): if True, lets the step emit its own annotations. NOTE: Enabling this can cause some buggy behavior. Please strongly consider using step_result.presentation instead. If you have questions, please contact infra-dev@chromium.org.
trigger_specs: a list of trigger specifications
stdout: Placeholder to put step stdout into. If used, stdout won‘t appear in annotator’s stdout (and |allow_subannotations| is ignored).
stderr: Placeholder to put step stderr into. If used, stderr won‘t appear in annotator’s stderr.
stdin: Placeholder to read step stdin from.
step_test_data (func -> recipe_test_api.StepTestData): A factory which returns a StepTestData object that will be used as the default test data for this step. The recipe author can override/augment this object in the GenTests function.

Returns a types.StepData for the running step.

@property
— def active_result(self):

The currently active (open) result from the last step that was run. This is a types.StepData object.

Allows you to do things like:

try:
  api.step('run test', [..., api.json.output()])
finally:
  result = api.step.active_result
  if result.json.output:
    new_step_text = result.json.output['step_text']
    api.step.active_result.presentation.step_text = new_step_text

This will update the step_text of the test, even if the test fails. Without this api, the above code would look like:

try:
  result = api.step('run test', [..., api.json.output()])
except api.StepFailure as f:
  result = f.result
  raise
finally:
  if result.json.output:
    new_step_text = result.json.output['step_text']
    api.step.active_result.presentation.step_text = new_step_text

@property
— def defer_results(self):

See recipe_api.py for docs.

@contextlib.contextmanager
— def nest(self, name):

Nest allows you to nest steps hierarchically on the build UI.

Calling

with api.step.nest(<name>):
  ...

will generate a dummy step with the provided name. All other steps run within this with statement will be hidden from the UI by default under this dummy step in a collapsible hierarchy. Nested blocks can also nest within each other.

The nesting is implemented by adjusting the ‘name’ and ‘nest_level’ fields of the context (see the context() method above).

recipe_modules / tempfile

DEPS: file, path

Simplistic temporary directory manager (deprecated).

class TempfileApi(RecipeApi):

@contextlib.contextmanager
— def temp_dir(self, prefix):

This makes a temporary directory which lives for the scope of the with statement.

Example:

with api.tempfile.temp_dir("some_prefix") as path:
  # use path
# path is deleted here.

recipe_modules / time

Allows mockable access to the current time.

class TimeApi(RecipeApi):

— def sleep(self, secs):

Suspend execution of |secs| (float) seconds. Does nothing during testing.

— def time(self):

Return current timestamp as a float number of seconds since epoch.

— def utcnow(self):

Return current UTC time as a datetime.datetime.

recipe_modules / url

DEPS: context, json, path, python, raw_io

Methods for interacting with HTTP(s) URLs.

class UrlApi(RecipeApi):

— def get_file(self, url, path, step_name=None, headers=None, transient_retry=True, strip_prefix=None, timeout=None):

GET data at given URL and writes it to file.

Args:

url: URL to request.
path (Path): the Path where the content will be written.
step_name: optional step name, ‘fetch ’ by default.
headers: a {header_name: value} dictionary for HTTP headers.
transient_retry (bool or int): Determines how transient HTTP errorts (>500) will be retried. If True (default), errors will be retried up to 10 times. If False, no transient retries will occur. If an integer is supplied, this is the number of transient retries to perform. All retries have exponential backoff applied.
strip_prefix (str or None): If not None, this prefix must be present at the beginning of the response, and will be stripped from the resulting content (e.g., GERRIT_JSON_PREFIX).
timeout: Timeout (see step.call).

Returns (UrlApi.Response) - Response with “path” as its “output” value.

Raises:

HTTPError, InfraHTTPError: if the request failed.
ValueError: If the request was invalid.

— def get_json(self, url, step_name=None, headers=None, transient_retry=True, strip_prefix=None, log=False, timeout=None, default_test_data=None):

GET data at given URL and writes it to file.

Args:

url: URL to request.
step_name: optional step name, ‘fetch ’ by default.
headers: a {header_name: value} dictionary for HTTP headers.
transient_retry (bool or int): Determines how transient HTTP errorts (>500) will be retried. If True (default), errors will be retried up to 10 times. If False, no transient retries will occur. If an integer is supplied, this is the number of transient retries to perform. All retries have exponential backoff applied.
strip_prefix (str or None): If not None, this prefix must be present at the beginning of the response, and will be stripped from the resulting content (e.g., GERRIT_JSON_PREFIX).
log (bool): If True, emit the JSON content as a log.
timeout: Timeout (see step.call).
default_test_data (jsonish): If provided, use this as the unmarshalled JSON result when testing if no overriding data is available.

Returns (UrlApi.Response) - Response with the JSON as its “output” value.

Raises:

HTTPError, InfraHTTPError: if the request failed.
ValueError: If the request was invalid.

— def get_text(self, url, step_name=None, headers=None, transient_retry=True, timeout=None, default_test_data=None):

GET data at given URL and writes it to file.

Args:

url: URL to request.
step_name: optional step name, ‘fetch ’ by default.
headers: a {header_name: value} dictionary for HTTP headers.
transient_retry (bool or int): Determines how transient HTTP errorts (>500) will be retried. If True (default), errors will be retried up to 10 times. If False, no transient retries will occur. If an integer is supplied, this is the number of transient retries to perform. All retries have exponential backoff applied.
timeout: Timeout (see step.call).
default_test_data (str): If provided, use this as the text output when testing if no overriding data is available.

Returns (UrlApi.Response) - Response with the content as its output value.

Raises:

HTTPError, InfraHTTPError: if the request failed.
ValueError: If the request was invalid.

— def join(self, *parts):

Constructs a URL path from composite parts.

Args:

parts (str...): Strings to concastenate. Any leading or trailing slashes will be stripped from intermediate strings to ensure that they join together. Trailing slashes will not be stripped from the last part.

— def validate_url(self, v):

Validates that “v” is a valid URL.

A valid URL has a scheme and netloc, and must begin with HTTP or HTTPS.

Args:

v (str): The URL to validate.

Returns (bool) - True if the URL is considered secure, False if not.

Raises:

ValueError: if “v” is not valid.

Recipes

recipes / buildbucket:examples/full

DEPS: buildbucket, platform, properties, raw_io, step

DEPS: python

recipes / engine_tests/functools_partial

DEPS: step

Engine shouldn't explode when step_test_data gets functools.partial.

This is a regression test for a bug caused by this revision: http://src.chromium.org/viewvc/chrome?revision=298072&view=revision

When this recipe is run (by run_test.py), the _print_step code is exercised.

— def RunSteps(api):

recipes / engine_tests/missing_start_dir

DEPS: path, step

Tests that deleting the current working directory doesn't immediately fail.

— def RunSteps(api):

recipes / engine_tests/module_injection_site

DEPS: path, step

This test serves to demonstrate that the ModuleInjectionSite object on recipe modules (i.e. the .m) also contains a reference to the module which owns it.

This was implemented to aid in refactoring some recipes (crbug.com/782142).

— def RunSteps(api):

recipes / engine_tests/multi_test_data

DEPS: raw_io, step

Tests that step_data can accept multiple specs at once.

— def RunSteps(api):

DEPS: properties, python, step

— def RunSteps(api, from_recipe, attribute, module):

recipes / engine_tests/unicode

DEPS: properties, step

— def RunSteps(api):

recipes / engine_tests/whitelist_steps

recipes / python:examples/full

DEPS: path, python, raw_io, step

Launches the repo bundler.

— def RunSteps(api):

recipes / raw_io:examples/full

DEPS: path, properties, python, raw_io, step

— def RunSteps(api):

recipes / runtime:tests/full

DEPS: runtime, step

— def RunSteps(api):

recipes / scheduler:examples/emit_triggers

DEPS: json, properties, runtime, scheduler, time

Table of Contents

Recipe Modules

recipe_modules / buildbucket

class BuildbucketApi(RecipeApi):

recipe_modules / context

class ContextApi(RecipeApi):

recipe_modules / file

class FileApi(RecipeApi):

recipe_modules / generator_script

class GeneratorScriptApi(RecipeApi):

recipe_modules / json

class JsonApi(RecipeApi):

recipe_modules / path

class PathApi(RecipeApi):

recipe_modules / platform

class PlatformApi(RecipeApi):

recipe_modules / properties

class PropertiesApi(RecipeApiPlain, collections.Mapping):

recipe_modules / python

class PythonApi(RecipeApi):

recipe_modules / raw_io

class RawIOApi(RecipeApi):

recipe_modules / runtime

class RuntimeApi(RecipeApi):

recipe_modules / scheduler

class SchedulerApi(RecipeApi):

recipe_modules / service_account

class ServiceAccountApi(RecipeApi):

recipe_modules / source_manifest

class SourceManfiestApi(RecipeApi):

TODO(iannucci): remove this restriction.

recipe_modules / step

class StepApi(RecipeApiPlain):

recipe_modules / tempfile

class TempfileApi(RecipeApi):

recipe_modules / time

class TimeApi(RecipeApi):

recipe_modules / url

class UrlApi(RecipeApi):

Recipes

recipes / buildbucket:examples/full

recipes / buildbucket:tests/build_id

recipes / buildbucket:tests/get

recipes / buildbucket:tests/properties

recipes / buildbucket:tests/put

recipes / context:examples/full

recipes / context:tests/cwd

recipes / context:tests/env

recipes / context:tests/increment_nest_level

recipes / context:tests/infra_step

recipes / engine_tests/bad_subprocess

recipes / engine_tests/depend_on/bad_properties

recipes / engine_tests/depend_on/bottom

recipes / engine_tests/depend_on/dont_need_properties

recipes / engine_tests/depend_on/dont_need_properties_helper

recipes / engine_tests/depend_on/need_return_schema

recipes / engine_tests/depend_on/need_return_schema_helper

recipes / engine_tests/depend_on/no_return

recipes / engine_tests/depend_on/top

recipes / engine_tests/expect_exception

recipes / engine_tests/functools_partial

recipes / engine_tests/missing_start_dir

recipes / engine_tests/module_injection_site

recipes / engine_tests/multi_test_data

recipes / engine_tests/nonexistent_command

recipes / engine_tests/recipe_paths

recipes / engine_tests/sort_properties

recipes / engine_tests/step_stack_exhaustion

recipes / engine_tests/undeclared_method

recipes / engine_tests/unicode

recipes / engine_tests/whitelist_steps

recipes / file:examples/copy

recipes / file:examples/copytree

recipes / file:examples/error

recipes / file:examples/glob

recipes / file:examples/listdir

recipes / file:examples/raw_copy

recipes / file:examples/symlink

recipes / file:examples/truncate