The context module provides APIs for manipulating a few pieces of ‘ambient’ data that affect how steps are run.
The pieces of information which can be modified are:
The values here are all scoped using Python‘s with
statement; there’s no mechanism to make an open-ended adjustment to these values (i.e. there's no way to change the cwd permanently for a recipe, except by surrounding the entire recipe with a with statement). This is done to avoid the surprises that typically arise with things like os.environ or os.chdir in a normal python program.
Example:
with api.context(cwd=api.path['start_dir'].join('subdir')): # this step is run inside of the subdir directory. api.step("cat subdir/foo", ['cat', './foo'])
@contextmanager
— def __call__(self, cwd=None, env_prefixes=None, env=None, increment_nest_level=None, infra_steps=None, name_prefix=None):
Allows adjustment of multiple context values in a single call.
Args:
api.path['start_dir']
.Name prefixes:
Multiple invocations concatenate values with ‘.’.
Example:
with api.context(name_prefix='hello'): # has name 'hello.something' api.step('something', ['echo', 'something']) with api.context(name_prefix='world'): # has name 'hello.world.other' api.step('other', ['echo', 'other'])
Environmental Variable Overrides:
Env is a mapping of environment variable name to the value you want that environment variable to have. The value is one of:
“env_prefix” is a list of Path or strings that get prefixed to their respective environment variables, delimited with the system's path separator. This can be used to add entries to environment variables such as “PATH” and “PYTHONPATH”. If prefixes are specified and a value is also defined in “env”, it will be installed as the last path component if it is not empty.
TODO(iannucci): combine nest_level and name_prefix
Look at the examples in “examples/” for examples of context module usage.
@property
— def cwd(self):
Returns the current working directory that steps will run in.
Returns (Path|None) - The current working directory. A value of None is equivalent to api.path[‘start_dir’], though only occurs if no cwd has been set (e.g. in the outermost context of RunSteps).
@property
— def env(self):
Returns modifications to the environment.
By default this is empty; There‘s no facility to observe the program’s startup environment. If you want to pass data to the recipe, it should be done with properties.
Returns (dict) - The env-key -> value mapping of current environment modifications.
@property
— def env_prefixes(self):
Returns Path prefix modifications to the environment.
This will return a mapping of environment key to Path tuple for Path prefixes registered with the environment.
Returns (dict) - The env-key -> value(Path) mapping of current environment prefix modifications.
@property
— def infra_step(self):
Returns the current value of the infra_step setting.
Returns (bool) - True iff steps are currently considered infra steps.
@property
— def name_prefix(self):
Gets the current step name prefix.
Returns (str) - The string prefix that every step will have prepended to it.
@property
— def nest_level(self):
Returns the current ‘nesting’ level.
Note: This api is low-level, and you should always prefer to use api.step.nest
. This api is included for completeness and documentation purposes.
Returns (int) - The current nesting level.
DEPS: json, path, python, raw_io, step
File manipulation (read/write/delete/glob) methods.
— def copy(self, name, source, dest):
Copies a file (including mode bits) from source to destination on the local filesystem.
Behaves identically to shutil.copy.
Args:
source
will be appended to derive a path to a destination file.Raises file.Error
— def copytree(self, name, source, dest, symlinks=False):
Recursively copies a directory tree.
Behaves identically to shutil.copytree. dest
must not exist.
Args:
Raises file.Error
— def ensure_directory(self, name, dest, mode=511):
Ensures that dest
exists and is a directory.
Args:
Raises file.Error if the path exists but is not a directory.
— def filesizes(self, name, files, test_data=None):
Returns list of filesizes for the given files.
Args:
Returns list[int], size of each file in bytes.
— def glob_paths(self, name, source, pattern, test_data=()):
Performs glob expansion on pattern
.
glob rules for pattern
follow the same syntax as for the python glob
stdlib module.
Args:
source
.Returns list[Path] - All paths found.
Raises file.Error.
— def listdir(self, name, source, test_data=()):
List all files inside a directory.
Args:
Returns list[Path]
Raises file.Error.
— def move(self, name, source, dest):
Moves a file or directory.
Behaves identically to shutil.move.
Args:
Raises file.Error
— def read_text(self, name, source, test_data=''):
Reads a file as UTF-8 encoded text.
Args:
Returns (str) - The content of the file.
Raises file.Error
— def remove(self, name, source):
Remove a file.
Does not raise Error if the file doesn't exist.
Args:
Raises file.Error.
— def rmcontents(self, name, source):
Similar to rmtree, but removes only contents not the directory.
This is useful e.g. when removing contents of current working directory. Deleting current working directory makes all further getcwd calls fail until chdir is called. chdir would be tricky in recipes, so we provide a call that doesn't delete the directory itself.
Args:
Raises file.Error.
— def rmglob(self, name, source, pattern):
Removes all entries in source
matching the glob pattern
.
Args:
source
. Anything matching this pattern will be removed.Raises file.Error.
— def rmtree(self, name, source):
Recursively removes a directory.
This uses a native python on Linux/Mac, and uses rd
on Windows to avoid issues w.r.t. path lengths and read-only attributes. If the directory is gone already, this returns without error.
Args:
Raises file.Error.
— def write_text(self, name, dest, text_data):
Write the given UTF-8 encoded text_data
to dest
.
Args:
Raises file.Error.
DEPS: context, json, path, python, step
A simplistic method for running steps generated by an external script.
This module was created before there was a way to put recipes directly into another repo. It is not recommended to use this, and it will be removed in the near future.
— def __call__(self, path_to_script, *args):
Run a script and generate the steps emitted by that script.
The script will be invoked with --output-json /path/to/file.json. The script is expected to exit 0 and write steps into that file. Once the script outputs all of the steps to that file, the recipe will read the steps from that file and execute them in order. Any *args specified will be additionally passed to the script.
The step data is formatted as a list of JSON objects. Each object corresponds to one step, and contains the following keys:
--presentation-json /path/to/file.json
. This file will be used to update the step's presentation on the build status page. The file will be expected to contain a single json object, with any of the following keys:DEPS: properties, python, raw_io
Methods for producing and consuming JSON.
@returns_placeholder
— def input(self, data):
A placeholder which will expand to a file path containing .
— def is_serializable(self, obj):
Returns True if the object is JSON-serializable.
@staticmethod
— def loads(data, **kwargs):
Works like json.loads
, but strips out unicode objects (replacing them with utf8-encoded str objects).
@returns_placeholder
— def output(self, add_json_log=True, name=None, leak_to=None):
A placeholder which will expand to ‘/tmp/file’.
If leak_to is provided, it must be a Path object. This path will be used in place of a random temporary file, and the file will not be deleted at the end of the step.
Args:
name
. If this is ‘on_failure’, only create this log when the step has a non-SUCCESS status.— def read(self, name, path, add_json_log=True, output_name=None, **kwargs):
Returns a step that reads a JSON file.
All functions related to manipulating paths in recipes.
Recipes handle paths a bit differently than python does. All path manipulation in recipes revolves around Path objects. These objects store a base path (always absolute), plus a list of components to join with it. New paths can be derived by calling the .join method with additional components.
In this way, all paths in Recipes are absolute, and are constructed from a small collection of anchor points. The built-in anchor points are:
api.path['start_dir']
- This is the directory that the recipe started in. it‘s similar to cwd
, except that it’s constant.api.path['cache']
- This directory is provided by whatever's running the recipe. Files and directories created under here /may/ be evicted in between runs of the recipe (i.e. to relieve disk pressure).api.path['cleanup']
- This directory is provided by whatever's running the recipe. Files and directories created under here /are guaranteed/ to be evicted in between runs of the recipe. Additionally, this directory is guaranteed to be empty when the recipe starts.api.path['tmp_base']
- This directory is the system-configured temp dir. This is a weaker form of ‘cleanup’, and its use should be avoided. This may be removed in the future (or converted to an alias of ‘cleanup’).api.path['checkout']
- This directory is set by various ‘checkout’ modules in recipes. It was originally intended to make recipes easier to read and make code somewhat generic or homogenous, but this was a mistake. New code should avoid ‘checkout’, and instead just explicitly pass paths around. This path may be removed in the future.There are other anchor points which can be defined (e.g. by the depot_tools/infra_paths
module). Refer to those modules for additional documentation.
— def __getitem__(self, name):
Gets the base path named name
. See module docstring for more information.
— def abs_to_path(self, abs_string_path):
Converts an absolute path string string_path
to a real Path object, using the most appropriate known base path.
This method will find the longest match in all the following:
Example:
# assume [START_DIR] == "/basis/dir/for/recipe" api.path.abs_to_path("/basis/dir/for/recipe/some/other/dir") -> Path("[START_DIR]/some/other/dir")
Raises an ValueError if the preconditions are not met, otherwise returns the Path object.
— def assert_absolute(self, path):
Raises AssertionError if the given path is not an absolute path.
Args:
— def get(self, name, default=None):
Gets the base path named name
. See module docstring for more information.
— def get_config_defaults(self):
Internal recipe implementation function.
— def initialize(self):
Internal recipe implementation function.
— def mkdtemp(self, prefix):
Makes a new temporary directory, returns Path to it.
— def mock_add_paths(self, path):
For testing purposes, mark that |path| exists.
— def mock_copy_paths(self, source, dest):
For testing purposes, copy |source| to |dest|.
— def mock_remove_paths(self, path, filt=(lambda p: True)):
For testing purposes, assert that |path| doesn't exist.
Args:
Mockable system platform identity functions.
Provides host-platform-detection properties.
Mocks:
@property
— def arch(self):
Returns the current CPU architecture.
TODO: This is currently always hard-coded to ‘intel’... Apparently no one has actually needed this function?
@property
— def bits(self):
Returns the bitness of the userland for the current system (either 32 or 64 bit).
TODO: If anyone needs to query for the kernel bitness, another accessor should be added.
@property
— def cpu_count(self):
The number of CPU cores, according to multiprocessing.cpu_count().
@property
— def is_linux(self):
Returns True iff the recipe is running on Linux.
@property
— def is_mac(self):
Returns True iff the recipe is running on OS X.
@property
— def is_win(self):
Returns True iff the recipe is running on Windows.
@property
— def name(self):
Returns the current platform name which will be in:
@staticmethod
— def normalize_platform_name(plat):
One of python's sys.platform values -> ‘win’, ‘linux’ or ‘mac’.
Provides access to the recipes input properties.
Every recipe is run with a JSON object called “properties”. These contain all inputs to the recipe. Some common examples would be properties like “revision”, which the build scheduler sets to tell a recipe to build/test a certain revision.
The properties that affect a particular recipe are defined by the recipe itself, and this module provides access to them.
Recipe properties are read-only; the values obtained via this API reflect the values provided to the recipe engine at the beginning of execution. There is intentionally no API to write property values (lest they become a kind of random-access global variable).
PropertiesApi implements all the standard Mapping functions, so you can use it like a read-only dict.
— def legacy(self):
DEPRECATED: Returns a set of properties, possibly used by legacy scripts.
This excludes any recipe module-specific properties (i.e. those beginning with $
).
Instead of passing all of the properties as a blob, please consider passing specific arguments to scripts that need them. Doing this makes it much easier to debug and diagnose which scripts use which properties.
— def thaw(self):
Returns a read-write copy of all of the properties.
Provides methods for running python scripts correctly.
This includes support for vpython
, and knows how to specify parameters correctly for bots (e.g. ensuring that python is working on Windows, passing the unbuffered flag, etc.)
— def __call__(self, name, script, args=None, unbuffered=True, venv=None, **kwargs):
Return a step to run a python script with arguments.
TODO: We should just use a single “args” list. Having “script” separate but required/first leads to weird things like:
(... script='-m', args=['module'])
Args:
Returns (types.StepData
) - The StepData object as returned by api.step.
— def failing_step(self, name, text, as_log=None):
Runs a succeeding step (exits 1).
— def inline(self, name, program, add_python_log=True, **kwargs):
Run an inline python program as a step.
Program is output to a temp file and run when this step executes.
Args:
python /path/to/file.py
program
.Returns (types.StepData
) - The StepData object as returned by api.step.
— def result_step(self, name, text, retcode, as_log=None):
Runs a no-op step that exits with a specified return code.
— def succeeding_step(self, name, text, as_log=None):
Runs a succeeding step (exits 0).
Provides objects for reading and writing raw data to and from steps.
@returns_placeholder
@staticmethod
— def input(data, suffix=''):
Returns a Placeholder for use as a step argument.
This placeholder can be used to pass data to steps. The recipe engine will dump the ‘data’ into a file, and pass the filename to the command line argument.
data MUST be of type ‘str’ (not basestring, not unicode).
If ‘suffix’ is not '', it will be used when the engine calls tempfile.mkstemp.
See examples/full.py for usage example.
@returns_placeholder
@staticmethod
— def input_text(data, suffix=''):
Returns a Placeholder for use as a step argument.
data MUST be of type ‘str’ (not basestring, not unicode). The str is expected to have valid utf-8 data in it.
Similar to input(), but ensures that ‘data’ is valid utf-8 text. Any non-utf-8 characters will be replaced with �.
@returns_placeholder
@staticmethod
— def output(suffix='', leak_to=None, name=None):
Returns a Placeholder for use as a step argument, or for std{out,err}.
If ‘leak_to’ is None, the placeholder is backed by a temporary file with a suffix ‘suffix’. The file is deleted when the step finishes.
If ‘leak_to’ is not None, then it should be a Path and placeholder redirects IO to a file at that path. Once step finishes, the file is NOT deleted (i.e. it's ‘leaking’). ‘suffix’ is ignored in that case.
@returns_placeholder
@staticmethod
— def output_dir(suffix='', leak_to=None, name=None):
Returns a directory Placeholder for use as a step argument.
If ‘leak_to’ is None, the placeholder is backed by a temporary dir with a suffix ‘suffix’. The dir is deleted when the step finishes.
If ‘leak_to’ is not None, then it should be a Path and placeholder redirects IO to a dir at that path. Once step finishes, the dir is NOT deleted (i.e. it's ‘leaking’). ‘suffix’ is ignored in that case.
@returns_placeholder
@staticmethod
— def output_text(suffix='', leak_to=None, name=None):
Returns a Placeholder for use as a step argument, or for std{out,err}.
Similar to output(), but uses an OutputTextPlaceholder, which expects utf-8 encoded text. Similar to input(), but tries to decode the resulting data as utf-8 text, replacing any decoding errors with �.
— def set_json_manifest(self, name, data):
Uploads a source manifest with the given name.
NOTE: Due to current implementation restrictions, this method may only be called after some step has been run from the recipe. Calling this before running any steps is invalid and will fail. We hope to lift this restriction sometime after we don't need to support buildbot any more.
Args:
Step is the primary API for running steps (external programs, scripts, etc.).
@property
— def InfraFailure(self):
InfraFailure is a subclass of StepFailure, and will translate to a purple build.
This exception is raised from steps which are marked as infra_step
s when they fail.
@property
— def StepFailure(self):
This is the base Exception class for all step failures.
It can be manually raised from recipe code to cause the build to turn red.
Usage:
raise api.StepFailure("some reason")
except api.StepFailure:
@property
— def StepTimeout(self):
StepTimeout is a subclass of StepFailure and is raised when a step times out.
@property
— def StepWarning(self):
StepWarning is a subclass of StepFailure, and will translate to a yellow build.
@recipe_api.composite_step
— def __call__(self, name, cmd, ok_ret=None, infra_step=False, wrapper=(), timeout=None, allow_subannotations=None, trigger_specs=None, stdout=None, stderr=None, stdin=None, step_test_data=None):
Returns a step dictionary which is compatible with annotator.py.
Args:
Returns a types.StepData
for the running step.
@property
— def active_result(self):
The currently active (open) result from the last step that was run. This is a types.StepData
object.
Allows you to do things like:
try: api.step('run test', [..., api.json.output()]) finally: result = api.step.active_result if result.json.output: new_step_text = result.json.output['step_text'] api.step.active_result.presentation.step_text = new_step_text
This will update the step_text of the test, even if the test fails. Without this api, the above code would look like:
try: result = api.step('run test', [..., api.json.output()]) except api.StepFailure as f: result = f.result raise finally: if result.json.output: new_step_text = result.json.output['step_text'] api.step.active_result.presentation.step_text = new_step_text
@property
— def defer_results(self):
See recipe_api.py for docs.
@contextlib.contextmanager
— def nest(self, name):
Nest allows you to nest steps hierarchically on the build UI.
Calling
with api.step.nest(<name>): ...
will generate a dummy step with the provided name. All other steps run within this with statement will be hidden from the UI by default under this dummy step in a collapsible hierarchy. Nested blocks can also nest within each other.
The nesting is implemented by adjusting the ‘name’ and ‘nest_level’ fields of the context (see the context() method above).
Simplistic temporary directory manager (deprecated).
@contextlib.contextmanager
— def temp_dir(self, prefix):
This makes a temporary directory which lives for the scope of the with statement.
Example:
with api.tempfile.temp_dir("some_prefix") as path: # use path # path is deleted here.
Allows mockable access to the current time.
— def time(self):
Return current timestamp as a float number of seconds since epoch.
— def utcnow(self):
Return current UTC time as a datetime.datetime.
DEPS: context, json, path, python, raw_io
Methods for interacting with HTTP(s) URLs.
— def get_file(self, url, path, step_name=None, headers=None, transient_retry=True, strip_prefix=None, timeout=None):
GET data at given URL and writes it to file.
Args:
Returns (UrlApi.Response) - Response with “path” as its “output” value.
Raises:
— def get_json(self, url, step_name=None, headers=None, transient_retry=True, strip_prefix=None, log=False, timeout=None, default_test_data=None):
GET data at given URL and writes it to file.
Args:
Returns (UrlApi.Response) - Response with the JSON as its “output” value.
Raises:
— def get_text(self, url, step_name=None, headers=None, transient_retry=True, timeout=None, default_test_data=None):
GET data at given URL and writes it to file.
Args:
Returns (UrlApi.Response) - Response with the content as its output value.
Raises:
— def join(self, *parts):
Constructs a URL path from composite parts.
Args:
— def validate_url(self, v):
Validates that “v” is a valid URL.
A valid URL has a scheme and netloc, and must begin with HTTP or HTTPS.
Args:
Returns (bool) - True if the URL is considered secure, False if not.
Raises:
DEPS: context, path, raw_io, step
— def RunSteps(api):
— def RunSteps(api):
DEPS: context, path, raw_io, step
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
Tests that daemons that hang on to STDOUT can't cause the engine to hang.
— def RunSteps(api):
— def RunSteps(api):
DEPS: properties, raw_io, step
— def RunSteps(api, number):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api, to_pass):
Tests that step_data can accept multiple specs at once.
— def RunSteps(api):
Engine shouldn't explode when step_test_data gets functools.partial.
This is a regression test for a bug caused by this revision: http://src.chromium.org/viewvc/chrome?revision=298072&view=revision
When this recipe is run (by run_test.py), the _print_step code is exercised.
— def RunSteps(api):
Tests that deleting the current working directory doesn't immediately fail.
— def RunSteps(api):
Tests that step_data can accept multiple specs at once.
— def RunSteps(api):
— def RunSteps(api):
Tests that recipes have access to names, resources and their package.
— def RunSteps(api):
Tests that step presentation properties can be ordered.
— def RunSteps(api):
Tests that placeholders can't wreck the world by exhausting the step stack.
— def RunSteps(api):
DEPS: properties, python, step
— def RunSteps(api, from_recipe, attribute, module):
— def RunSteps(api):
DEPS: context, properties, step
Tests that step_data can accept multiple specs at once.
— def RunSteps(api, fakeit):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
DEPS: generator_script, json, path, properties, step
— def RunSteps(api, script_name):
DEPS: json, path, properties, python, raw_io, step
— def RunSteps(api):
— def RunSteps(api):
DEPS: path, platform, properties, step
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api, test_prop, param_name_test):
DEPS: path, python, raw_io, step
Launches the repo bundler.
— def RunSteps(api):
DEPS: path, properties, python, raw_io, step
— def RunSteps(api):
— def RunSteps(api):
DEPS: context, path, properties, step
— def RunSteps(api, bad_return, access_invalid_data, timeout):
— def RunSteps(api):
— def RunSteps(api):
DEPS: context, path, properties, step
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api):
— def RunSteps(api, timeout):
— def RunSteps(api, command):
— def RunSteps(api):
— def RunSteps(api):
DEPS: context, path, step, url
— def RunSteps(api):
— def RunSteps(api):
DEPS: properties, step, url
— def RunSteps(api):