Package documentation for infra

Table of Contents

Recipe Modules

Recipes

Recipe Modules

recipe_modules / conda

DEPS: depot_tools/cipd, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/python, recipe_engine/step, recipe_engine/url

Functions to work with Miniconda python environment.

See http://conda.pydata.org/miniconda.html

class CondaApi(RecipeApi):

def install(self, version, path):

Downloads Miniconda installer for given version and executes it.

Args: version: version of Miniconda to install, e.g. ‘Miniconda2-3.18.3’. path: prefix to install Miniconda into.

Returns: Instance of CondaEnv, that also optionally acts as context manager that deletes the environment on exit.

recipe_modules / infra_checkout

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/python, recipe_engine/runtime

class InfraCheckoutApi(RecipeApi):

Stateless API for using public infra gclient checkout.

def checkout(self, gclient_config_name, patch_root=None, path=None, internal=False, named_cache=None, **kwargs):

Fetches infra gclient checkout into a given path OR named_cache.

Arguments:

  • gclient_config_name (string) - name of gclient config.
  • patch_root (path or string) - path inside infra checkout to git repo in which to apply the patch. For example, ‘infra/luci’ for luci-py repo. If None (default), no patches will be applied.
  • path (path or string) - path to where to create/update infra checkout. If None (default) - path is cache with customizable name (see below).
  • internal (bool) - by default, False, meaning infra gclient checkout layout is assumed, else infra_internal. This has an effect on named_cache default and inside which repo's go corner the ./go/env.py command is run.
  • named_cache - if path is None, this allows to customize the name of the cache. Defaults to PUBLIC_NAMED_CACHE or INTERNAL_NAMED_CACHE, depending on internal argument value. Note: your cr-buildbucket.cfg should specify named_cache for swarming to prioritize bots which actually have this cache populated by prior runs. Otherwise, using named cache isn't particularly useful, unless your pool of builders is very small.
  • kwargs - passed as is to bot_update.ensure_checkout.

Returns: a Checkout object with commands for common actions on infra checkout.

recipe_modules / infra_cipd

DEPS: depot_tools/cipd, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/json, recipe_engine/python, recipe_engine/runtime, recipe_engine/step

class InfraCIPDApi(RecipeApi):

API for building packages defined in infra's public and intenral repos.

Essentially a shim around scripts in https://chromium.googlesource.com/infra/infra.git/+/master/build/ and its internal counterpart.

def build(self):

Builds packages.

@contextlib.contextmanager
def context(self, path_to_repo, goos=None, goarch=None):

Sets context building CIPD packages.

Arguments: path_to_repo (path): path infra or infra_internal repo root dir. Expects to find build/build.py inside provided dir. goos, goarch (str): allows for setting GOOS and GOARCH for cross-compiling Go code.

Doesn't support nesting.

def tags(self, git_repo_url, revision):

Returns tags to be attached to uploaded CIPD packages.

def test(self, skip_if_cross_compiling=False):

Tests previously built packages integrity.

def upload(self, tags, step_test_data=None):

Uploads previously built packages.

recipe_modules / infra_system

DEPS: recipe_engine/context, recipe_engine/path, recipe_engine/platform

class InfraSystemApi(RecipeApi):

API for interacting with a provisioned infrastructure system.

@property
def sys_bin_path(self):

@contextlib.contextmanager
def system_env(self):

Yields a context modified to operate on system paths.

recipe_modules / omahaproxy

DEPS: recipe_engine/raw_io, recipe_engine/url

class OmahaproxyApi(RecipeApi):

APIs for interacting with omahaproxy.

def history(self, min_major_version=None, exclude_platforms=None):

@staticmethod
def split_version(text):

recipe_modules / recipe_autoroller

DEPS: build/luci_config, depot_tools/git, depot_tools/git_cl, depot_tools/gsutil, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

class RecipeAutorollerApi(RecipeApi):

def roll_projects(self, projects):

Attempts to roll each project from the provided list.

If rolling any of the projects leads to failures, other projects are not affected.

recipe_modules / sync_submodules

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/infra_paths, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/step

class SyncSubmodulesApi(RecipeApi):

def __call__(self, source, source_repo_checkout_name, dest, source_ref=‘refs/heads/master’, dest_ref=‘refs/heads/master’, extra_submodules=None, deps_path_prefix=None, disable_path_prefix=False):

Args: source: URL of the git repository to mirror. source_repo_checkout_name: Name of the directory that the source repo should be checked out into. dest: URL of the git repository to push to. source_ref: git ref in the source repository to checkout. dest_ref: git ref in the destination repository to push to. extra_submodules: a list of “path=URL” strings. These are added as extra submodules. deps_path_prefix: path prefix used to filter out DEPS. DEPS with the prefix are included. disable_path_prefix: disable filtering out DEPS by path prefix.

recipe_modules / third_party_packages

DEPS: depot_tools/cipd, depot_tools/git, depot_tools/gitiles, windows_sdk, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/url

class ThirdPartyPackagesApi(RecipeApi):

def build_package(self, name, workdir, root, install_mode):

@property
def cmake(self):

@property
def cross_platform(self):

@property
def dep(self):

def does_package_exist(self, name, version):

@dry_run.setter
def dry_run(self, v):

def ensure_package(self, workdir, repo_url, package_name_prefix, install_fn, tag, version, cipd_install_mode, test_fn=None):

Ensures that the specified CIPD package exists.

@property
def firebase(self):

@property
def gcloud(self):

def get_latest_release_tag(self, repo_url, prefix=‘v’):

def get_package_name(self, package_name_prefix):

@property
def git(self):

@property
def go(self):

@property
def gsutil(self):

def init_cross_platform(self, cross_platform):

@property
def ninja(self):

@property
def python(self):

def register_package(self, package_file, name, version):

def support_prefix(self, base):

@property
def swig(self):

recipe_modules / third_party_packages_ng

DEPS: depot_tools/git, depot_tools/osx_sdk, depot_tools/windows_sdk, recipe_engine/archive, recipe_engine/cipd, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/python, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/step

Allows uniform cross-compiliation, version tracking and archival for third-party software packages (libs+tools) for distribution via CIPD.

The purpose of the Third Party Packages (3pp) recipe/module is to generate CIPD packages of statically-compiled software for distribution in our continuous integration fleets, as well as software distributed to our develepers (e.g. via depot_tools).

Target os and architecture uses the CIPD “${os}-${arch}” (a.k.a. “${platform}”) nomenclature, which is currently defined in terms of Go's GOOS and GOARCH runtime variables (with the unfortunate exception that CIPD uses ‘mac’ instead of ‘darwin’). This is somewhat arbitrary, but has worked well so far for us.

Package Definitions

The 3pp module loads package definitions from a folder containing subfolders. Each subfolder defines a single software package to fetch, build and upload. For example, you might have a folder in your repo like this:

my_repo.git/
  third_party_packages/  # "root folder"
    .vpython             # common vpython file for all package scripts
    zlib/                # zlib "package folder"
      3pp.pb             # REQUIRED: the Spec.proto definition for zlib
      install.sh         # a script to build zlib from source
      extra_resource_file
    other_package/
      3pp.pb             # REQUIRED
      fetch.py           # a script to fetch `other_package` in a custom way
      install.sh
      install-win.sh     # windows-specific build script
    ...

This defines two packages (zlib, and other_package). The 3pp.pb files have references to the fetch/build scripts, and describe what dependencies the packages have (if any).

NOTE: Only one layer of package folders is supported currently.

Packages are named by the folder that contains their definition file (3pp.pb) and build scripts. It's preferable to have package named after software that it contains. However, sometimes you want multiple major versions of the software to exist side-by-side (e.g. pcre and pcre2, python and python3, etc.). In this case, have two separate package definition folders.

Each package folder contains a package spec (3pp.pb), as well as scripts, patches and/or utility tools to build the software from source.

The spec is a Text Proto document specified by the spec.proto schema.

The spec is broken up into two main sections, “create” and “upload”. The create section allows you to specify how the package software gets created, and allows specifying differences in how it's fetched/built/tested on a per-target basis, and the upload section has some details on how the final result gets uploaded to CIPD.

Creation Stages

The 3pp.pb spec begins with a series of create messages, each with details on on how to fetch+build+test the package. Each create message contains a “platform_re” field which works as a regex on the ${platform} value. All matching patterns apply in order, and non-matching patterns are skipped. Each create message is applied with a dict.update for each member message (i.e. [‘source’].update, [‘build’].update, etc.) to build a singular create message for the current target platform. For list values (e.g. ‘tool’, ‘dep’ in the Build message), you can clear them by providing a new empty value (e.g. tool: "")

Once all the create messages are merged (see schema for all keys that can be present), the actual creation takes place.

Note that “source” is REQUIRED in the final merged instruction set. All other messages are optional and have defaults as documented in spec.proto.

The creation process is broken up into 4 different stages:

  • Source
  • Build
  • Package
  • Verify
Envvars

All scripts described below are invoked with the following environment variables set:

  • $_3PP_PACKAGE_NAME - the name of the package currently building
  • $_3PP_PATCH_VERSION - the patch_version set for the version we're building (if any patch version was set).
  • $_3PP_PLATFORM - the platform we're targeting
  • $_3PP_TOOL_PLATFORM - the platform that we‘re building on (will be different than _3PP_PLATFORM if we’re cross-compiling)
  • $_3PP_VERSION - the version we're building, e.g. 1.2.3
  • $GOOS - The golang OS name we're targeting
  • $GOARCH - The golang architecture we're targeting
  • $MACOSX_DEPLOYMENT_TARGET - On OS X, set to 10.10, for your semi-up-to-date OS X building needs. This needs to be consistently set for all packages or it will cause linker warnings/errors when linking in static libs that were targeting a newer version (e.g. if it was left unset). Binaries built with this set to 10.10 will not run on 10.9 or older systems.

Additionally, on cross-compile environments, the $CROSS_TRIPLE environment variable is set to a GCC cross compile target triplet of cpu-vendor-os.

Source

The source is used to fetch the raw sources for assembling the package. In some cases the sources may actually be binary artifacts (e.g. prebuilt windows installers).

The source is unpacked to a checkout directory, possibly in some specified subdirectory. Sources can either produce the actual source files, or they can produce a single archive file (e.g. zip or tarball), which can be unpacked with the ‘unpack_archive’ option. In addition, patches can be applied to the source with the ‘patch_dir’ option (the patches should be in git format-patch format, and will be applied with git apply).

  • git - This checks out a semver tag in the repo.
  • cipd - This fetches data from a CIPD package.
  • script - Used for “weird” packages which are distributed via e.g. an HTML download page or an API. The script must be able to return the ‘latest’ version of its source, as well as to actually fetch a specified version.

Additionally the Source message contains a patch_version field to allow symver disambiguation of the built packages when they contain patches or other alterations which need to be versioned. This string will be joined with a ‘.’ to the source version being built when uploading the result to CIPD.

Build

The build message allows you to specify deps, and tools, as well as a script install which contains your logic to transform the source into the result package.

Deps are libraries built for the target ${platform} and are typically used for linking your package.

Tools are binaries built for the host; they‘re things like automake or sed that are used during the configure/make phase of your build, but aren’t linked into the built product. These tools will be available on $PATH (both ‘$tools’ and ‘$tools/bin’ are added to $PATH, because many packages are set up with their binaries at the base of the package, and some are set up with them in a /bin folder)

Installation occurs by invoking the script indicated by the ‘install’ field (with the appropriate interpreter, depending on the file extension) like:

<interpreter> "$install[*]" "$PREFIX" "$DEPS_PREFIX"

Where:

  • The current working directory is the base of the source checkout w/o subdir.
  • $install[*] are all of the tokens in the ‘install’ field.
  • $PREFIX is the directory which the script should install everything to; this directory will be archived into CIPD verbatim.
  • $DEPS_PREFIX is the path to a prefix directory containing the union of all of your packages' transitive deps. For example, all of the headers of your deps are located at $DEPS_PREFIX/include.
  • All tools are in $PATH

If the ‘install’ script is omitted, it is assumed to be ‘install.sh’.

If the ENTIRE build message is omitted, no build takes place. Instead the result of the ‘source’ stage will be packaged.

During the execution of the build phase, the entire ‘root folder’ is copied into the source checkout in the .3pp directory, and the script will be invoked as /path/to/checkout/.3pp/$package_name/$script_name. Because the entire root folder is copied, you can have shared resources (like .vpython files or helper scripts) which are common to all package definitions and located relative to the install script.

Package

Once the build stage is complete, all files in the $PREFIX folder passed to the install script will be zipped into a CIPD package.

It is strongly recommended that if your package is a library or tool with many files that it be packaged in the standard POSIXey PREFIX format (e.g. bin, lib, include, etc.). If your package is a collection of one or more standalone binaries, it's permissible to just have the binaries in the root of the output $PREFIX.

If the build stage is skipped (i.e. the build message is omitted) then the output of the source stage will be packaged instead (this is mostly useful when using a ‘script’ source).

Verify

After the package is built it can be optionally tested. The recipe will run your test script in an empty directory with the path to the packaged-but-not-yet-uploaded cipd package file and it can do whatever testing it needs to it (exiting non-zero if something is wrong). You can use the cipd pkg-deploy command to deploy it (or whatever cipd commands you like, though I wouldn't recommend uploading it to CIPD, as the 3pp recipe will do that after the test exits 0).

Upload

Once the test comes back positive, the CIPD package will be uploaded to the CIPD server and registered with the prefix indicated in the upload message. The full CIPD package name is constructed as:

<prefix>/<pkg_name>/${platform}

So for example with the prefix infra, the bzip2 package on linux-amd64 would be uploaded to infra/bzip2/linux-amd64 and tagged with the version that was built (e.g. version:1.2.3.patch_version1).

Versions

Every package will try to build the latest identifiable semver of its source, or will attempt to build the semver requested as an input property to the third_party_packages recipe. This semver is also used to tag the uploaded artifacts in CIPD.

Because some of the packages here are used as dependencies for others (e.g. curl and zlib are dependencies of git, and zlib is a dependency of curl), each package used as a dependency for others should specify its version explicitly (currently this is only possible to do with the ‘cipd’ source type). So e.g. zlib and curl specify their source versions, but git and python float at ‘head’, always building the latest tagged version fetched from git.

When building a floating package (e.g. python, git) you may explicitly state the symver that you wish to build as part of the recipe invocation.

The symver of a package (either specified in the package definition, in the recipe properties or discovered while fetching its source code (e.g. latest git tag)) is also used to tag the package when it's uploaded to CIPD (plus the patch_version in the source message).

Cross Compilation

Third party packages are currently compiled on linux using the ‘infra.tools.dockerbuild’ tool from the infra.git repo. This uses a sligthly modified version of the dockcross Docker cross-compile environment. Windows and OS X targets are built using the ‘osx_sdk’ and ‘windows_sdk’ recipe modules, each of which provides a hermetic (native) build toolchain for those platforms.

For linux, we can support all the architectures implied by dockerbuild, including:

  • linux-arm64
  • linux-armv6l
  • linux-mips32
  • linux-mips64
  • linux-386
  • linux-amd64

Dry runs

If the recipe is run in experimental mode (according to the “recipe_engine/runtime” module), then this recipe will skip the final CIPD upload.

Examples

As an example of the package definition layout in action, take a look at the third_party_packages folder in this infra.git repo.

class ThirdPartyPackagesNGApi(RecipeApi):

def ensure_uploaded(self, packages=(), platform='', force_build=False):

Executes entire {fetch,build,package,verify,upload} pipeline for all the packages listed, targeting the given platform.

Args:

  • packages (seq[str]) - A sequence of packages to ensure are uploaded. Packages must be listed as either ‘pkgname’ or ‘pkgname@version’. If empty, builds all loaded packages.
  • platform (str) - If specified, the CIPD ${platform} to build for. If unspecified, this will be the appropriate CIPD ${platform} for the current host machine.

Returns (list[(cipd_pkg, cipd_version)], set[str]) of built CIPD packages and their tagged versions, as well as a list of unsupported packages.

def initialize(self):

def load_packages_from_path(self, path):

Loads all package definitions from the given path.

This will parse and intern all the 3pp.pb package definition files so that packages can be identified by their name. For example, if you pass:

path/ pkgname/ 3pp.pb install.sh

This would parse path/pkgname/3pp.pb and register the “pkgname” package.

Args:

  • path (Path) - A path to a directory full of package definitions. Each package definition is a directory containing at least a 3pp.pb file, whose behavior is defined by 3pp.proto.

Returns a set(str) containing the names of the packages which were loaded.

Raises a DuplicatePackage exception if this function encounters a package whose name is already registered. This could occur if you call load_packages_from_path multiple times, and one of the later calls tries to load a pacakge which was registered under one of the earlier calls.

recipe_modules / wct

DEPS: depot_tools/cipd, recipe_engine/context, recipe_engine/path, recipe_engine/platform, recipe_engine/step

class WCTApi(RecipeApi):

WCTApi provides support for running Web Component Tests

def install(self):

def run(self, root, prefix=‘test/’, step_name=‘Run WCT tests’):

recipe_modules / windows_sdk

DEPS: recipe_engine/cipd, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/step

class WindowsSDKApi(RecipeApi):

API for using Windows SDK distributed via CIPD.

@contextmanager
def __call__(self, path=None, version=None, enabled=True):

Setups the SDK environment when enabled.

Args: path (path): Path to a directory where to install the SDK (default is ‘[start_dir]/cipd/windows_sdk’) version (str): CIPD instance ID, tag or ref of the SDK (default is set via $infra/windows_sdk.version property) enabled (bool): Whether the SDK should be used or not.

Raises: StepFailure or InfraFailure.

Recipes

recipes / build_conda_cipd_pkg

DEPS: depot_tools/cipd, conda, recipe_engine/buildbucket, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties

Recipe to build CIPD package with sealed Conda environment.

Supposed to be used from manually triggered Buildbot builders. We aren't expecting rebuilding this environment often, so setting up and periodic schedule is a waste of resources.

To build a new package for all platforms:

  1. Manually trigger all builders by clicking buttons in Buildbot.
  2. Once they all complete, tag the with some release identifier by running: ./cipd set-tag infra/conda_python/scientific/ -tag=release: -version=latest
  3. Update Puppet configs to use ‘release:’ as a version.

def RunSteps(api):

recipes / build_from_tarball

DEPS: depot_tools/gsutil, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / build_gsutil_cipd_pkg

DEPS: build/zip, depot_tools/cipd, depot_tools/gsutil, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api):

recipes / depot_tools_builder

DEPS: build/zip, depot_tools/cipd, depot_tools/git, depot_tools/gsutil, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/tempfile

Recipe to build windows depot_tools bootstrap zipfile.

def RunSteps(api, revision):

recipes / docker_image_builder

DEPS: build/docker, depot_tools/bot_update, depot_tools/gclient, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/service_account, recipe_engine/step, recipe_engine/time

def RunSteps(api):

recipes / gae_sdk_cipd_packager

DEPS: build/gae_sdk, build/zip, depot_tools/cipd, depot_tools/gsutil, recipe_engine/path, recipe_engine/python, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/step

def RunSteps(api):

recipes / gerrit_cq_test

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/buildbucket, recipe_engine/properties, recipe_engine/step

def RunSteps(api):

recipes / gerrit_hello_world

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Pushes a trivial CL to Gerrit to verify git authentication works on LUCI.

def RunSteps(api):

recipes / goma_hello_world

DEPS: build/goma, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Compiles trivial C++ program using Goma.

Intended to be used as a very simple litmus test of Goma health on LUCI staging environment. Linux and OSX only.

def RunSteps(api):

recipes / gsubmodd

DEPS: depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python

Runs git submodule daemon (gsubmodd) against a given source repo.

Intended to be called periodically (see CYCLE_TIME_SEC). Runs several iterations of the daemon and then quits so that recipe has a chance to resync the source code.

def RunSteps(api, source_repo, target_repo, limit=''):

recipes / gsubtreed

DEPS: depot_tools/gclient, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

Runs git subtree daemon (gsubtreed) against Chromium src repo.

Intended to be called periodically (see CYCLE_TIME_SEC). Runs several iteration of the daemon and then quits so that recipe has a chance to resync the source code.

def RunSteps(api, target_repo, cycle_time_sec, max_error_count):

recipes / gsutil_hello_world

DEPS: depot_tools/depot_tools, depot_tools/gsutil, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step, recipe_engine/time

Pushes a trivial CL to Gerrit to verify git authentication works on LUCI.

def RunSteps(api):

recipes / infra_checkout:examples/full

DEPS: depot_tools/tryserver, infra_checkout, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/json, recipe_engine/platform, recipe_engine/python, recipe_engine/runtime

def RunSteps(api):

recipes / infra_cipd:examples/usage

DEPS: infra_cipd, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/runtime, recipe_engine/step

def RunSteps(api):

recipes / infra_continuous

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/depot_tools, depot_tools/gclient, depot_tools/infra_paths, infra_checkout, infra_cipd, infra_system, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/runtime, recipe_engine/step

def RunSteps(api):

def build_main(api, buildername, project_name, repo_url, rev):

recipes / infra_continuous_luci

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

Builds and tests infra.git code.

Very dumb for now, with no side effects. Runs continuously on LUCI in both staging and prod environments.

def RunSteps(api):

recipes / infra_repo_trybot

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, infra_checkout, infra_system, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api):

recipes / infra_system:examples/full

DEPS: infra_system, recipe_engine/context, recipe_engine/platform, recipe_engine/step

def RunSteps(api):

recipes / infra_wct_tester

DEPS: depot_tools/bot_update, depot_tools/gclient, wct, recipe_engine/path, recipe_engine/platform, recipe_engine/step

def RunSteps(api):

recipes / lkgr_finder

DEPS: build/chromium_checkout, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/gitiles, depot_tools/gsutil, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/step

def RunSteps(api):

recipes / luci_check

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python

def RunSteps(api):

recipes / luci_gae

DEPS: depot_tools/tryserver, infra_checkout, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/json, recipe_engine/properties, recipe_engine/runtime

def RunSteps(api):

recipes / luci_go

DEPS: depot_tools/tryserver, infra_checkout, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/json, recipe_engine/platform, recipe_engine/properties, recipe_engine/runtime

def RunSteps(api, GOARCH):

recipes / luci_py

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/buildbucket, recipe_engine/properties

def RunSteps(api):

recipes / master_manager_config

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/infra_paths, infra_checkout, recipe_engine/buildbucket, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / publish_tarball

DEPS: build/chromium, build/trigger, depot_tools/bot_update, depot_tools/depot_tools, depot_tools/gclient, depot_tools/git, depot_tools/gsutil, omahaproxy, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/scheduler, recipe_engine/step

def RunSteps(api):

@contextlib.contextmanager
def copytree_checkout(api):

@recipe_api.composite_step
def export_lite_tarball(api, version):

@recipe_api.composite_step
def export_nacl_tarball(api, version):

@recipe_api.composite_step
def export_tarball(api, args, source, destination):

def gsutil_upload(api, source, bucket, dest, args):

def published_all_tarballs(version, ls_result):

def published_full_tarball(version, ls_result):

def published_lite_tarball(version, ls_result):

def published_nacl_tarball(version, ls_result):

def published_test_tarball(version, ls_result):

recipes / recipe_autoroller

DEPS: build/luci_config, build/puppet_service_account, recipe_autoroller, recipe_engine/json, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

Rolls recipes.cfg dependencies for public projects.

def RunSteps(api, projects, auth_token, service_account):

recipes / recipe_bundler

DEPS: depot_tools/cipd, recipe_engine/path, recipe_engine/properties, recipe_engine/step

def RunSteps(api, recipe_bundler_pkg, recipe_bundler_vers, repo_specs, package_name_prefix, package_name_internal_prefix):

recipes / recipe_roll_tryjob

DEPS: build/luci_config, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/service_account, recipe_engine/step

def RunSteps(api, upstream_project, downstream_project):

recipes / recipe_simulation

DEPS: build/luci_config, build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

A continuous builder which runs recipe tests.

def RunSteps(api, project_under_test, auth_with_account):

recipes / recipes_py_continuous

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/gclient, recipe_engine/buildbucket, recipe_engine/path, recipe_engine/properties

def RunSteps(api):

recipes / remote_execute_dataflow_workflow

DEPS: build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

This recipe is used to execute Dataflow workflows.

If you want a workflow to run at regular intervals, you can configure a builder to run this recipe. Dataflow workflows run on an internal builder, so this step must be completed by a Google employee. See this change for an example: https://chrome-internal-review.googlesource.com/c/chrome/tools/build/+/412934

Builders configured with the name matching “dataflow-workflow-.*” will be automatically monitored for failures.

This recipe uses the dataflow-launcher service account. That account must have the permission to schedule a Dataflow job for your project.

def RunSteps(api, workflow, job_name, gcp_project_id, num_workers, timeout):

recipes / snapshot

DEPS: recipe_engine/cipd, recipe_engine/path, recipe_engine/platform, recipe_engine/properties

def RunSteps(api):

recipes / sync_submodules

DEPS: sync_submodules, recipe_engine/properties, recipe_engine/runtime

def RunSteps(api, disable_path_prefix):

recipes / third_party_packages

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/runtime, recipe_engine/step

This recipe builds and packages third party software, such as Git.

def RunSteps(api, cross_platform):

recipes / third_party_packages:examples/cmake

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘cmake’ building.

During testing, it may be useful to focus on building CMake. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/dep

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Go dependency management tool building.

During testing, it may be useful to focus on building dep. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/firebase

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for firebase-tools building.

During testing, it may be useful to focus on building the firebase-tools. This can be done by running this recipe directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/gcloud

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Google Cloud SDK building.

During testing, it may be useful to focus on building the SDK. This can be done by running this recipe module directly.

def RunSteps(api, platform_name, platform_bits, dry_run):

recipes / third_party_packages:examples/git

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘git’ building.

During testing, it may be useful to focus on building Git. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/go

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Go toolchain building.

During testing, it may be useful to focus on building Go. This can be done by running this recipe module directly.

def RunSteps(api, platform_name, platform_bits, dry_run):

recipes / third_party_packages:examples/gsutil

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for gsutil building.

During testing, it may be useful to focus on building the gsutil. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/ninja

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘ninja’ building.

During testing, it may be useful to focus on building Ninja. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/python

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties

Recipe for ‘python’ building.

During testing, it may be useful to focus on building Python. This can be done by running this recipe module directly.

def RunSteps(api, dry_run, cross_platform):

recipes / third_party_packages:examples/swig

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘swig’ building.

During testing, it may be useful to focus on building Swig. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages_ng

DEPS: depot_tools/git, third_party_packages_ng, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/runtime, recipe_engine/step

This recipe builds and packages third party software, such as Git.

def RunSteps(api, package_locations, to_build, platform, force_build):

recipes / third_party_packages_ng:tests/full

DEPS: third_party_packages_ng, recipe_engine/cipd, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io

def RunSteps(api, GOOS, GOARCH, load_dupe):

recipes / tricium_noop

DEPS: recipe_engine/properties, recipe_engine/tricium

Recipe for Tricium that outputs no results.

This is intended to be used as the default recipe for the tricium builder, which shouldn't actually be used in runs triggered by Tricium.

def RunSteps(api):

recipes / windows_sdk:examples/full

DEPS: windows_sdk, recipe_engine/platform, recipe_engine/properties, recipe_engine/step

def RunSteps(api):

recipes / wpt_export

DEPS: build/chromium, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/runtime

Exports commits in Chromium to the web-platform-tests repo.

This recipe runs the wpt-export script; it is expected to be run as a recurring job at a short interval. It creates pull requests on GitHub for Chromium commits that contain exportable changes, merges these pull requests.

See: //docs/testing/web_platform_tests.md (https://goo.gl/rSRGmZ)

def RunSteps(api):

recipes / wpt_import

DEPS: build/chromium, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/runtime, recipe_engine/step

Imports changes from web-platform-tests into Chromium.

This recipe runs the wpt-import script. The import process involves first fetching the latest changes from web-platform-tests, then running the tests via try jobs and and updating any baselines and expectations, before committing to Chromium.

See: //docs/testing/web_platform_tests.md (https://goo.gl/rSRGmZ)

def RunSteps(api):

def git_cl_issue_link(api):

Runs a step which adds a link to the current CL if there is one.