Package documentation for infra

Table of Contents

Recipe Modules

Recipes

Recipe Modules

recipe_modules / conda

DEPS: depot_tools/cipd, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/python, recipe_engine/step, recipe_engine/url

Functions to work with Miniconda python environment.

See http://conda.pydata.org/miniconda.html

class CondaApi(RecipeApi):

def install(self, version, path):

Downloads Miniconda installer for given version and executes it.

Args: version: version of Miniconda to install, e.g. ‘Miniconda2-3.18.3’. path: prefix to install Miniconda into.

Returns: Instance of CondaEnv, that also optionally acts as context manager that deletes the environment on exit.

recipe_modules / infra_checkout

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python

class InfraCheckoutApi(RecipeApi):

Stateless API for using public infra gclient checkout.

def checkout(self, gclient_config_name, patch_root=None, path=None, internal=False, named_cache=None, **kwargs):

Fetches infra gclient checkout into a given path OR named_cache.

Arguments:

  • gclient_config_name (string) - name of gclient config.
  • patch_root (path or string) - path inside infra checkout to git repo in which to apply the patch. For example, ‘infra/luci’ for luci-py repo. If None (default), no patches will be applied.
  • path (path or string) - path to where to create/update infra checkout. If None (default) - path is cache with customizable name (see below).
  • internal (bool) - by default, False, meaning infra gclient checkout layout is assumed, else infra_internal. This has an effect on named_cache default and inside which repo's go corner the ./go/env.py command is run.
  • named_cache - if path is None, this allows to customize the name of the cache. Defaults to PUBLIC_NAMED_CACHE or INTERNAL_NAMED_CACHE, depending on internal argument value. Note: your cr-buildbucket.cfg should specify named_cache for swarming to prioritize bots which actually have this cache populated by prior runs. Otherwise, using named cache isn't particularly useful, unless your pool of builders is very small.
  • kwargs - passed as is to bot_update.ensure_checkout.

Returns: a Checkout object with commands for common actions on infra checkout.

recipe_modules / infra_cipd

DEPS: depot_tools/cipd, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/json, recipe_engine/python, recipe_engine/runtime, recipe_engine/step

class InfraCIPDApi(RecipeApi):

API for building packages defined in infra's public and intenral repos.

Essentially a shim around scripts in https://chromium.googlesource.com/infra/infra.git/+/master/build/ and its internal counterpart.

def build(self):

Builds packages.

@contextlib.contextmanager
def context(self, path_to_repo, goos=None, goarch=None):

Sets context building CIPD packages.

Arguments: path_to_repo (path): path infra or infra_internal repo root dir. Expects to find build/build.py inside provided dir. goos, goarch (str): allows for setting GOOS and GOARCH for cross-compiling Go code.

Doesn't support nesting.

def tags(self, git_repo_url, revision):

Returns tags to be attached to uploaded CIPD packages.

def test(self, skip_if_cross_compiling=False):

Tests previously built packages integrity.

def upload(self, tags, step_test_data=None):

Uploads previously built packages.

recipe_modules / infra_system

DEPS: recipe_engine/context, recipe_engine/path, recipe_engine/platform

class InfraSystemApi(RecipeApi):

API for interacting with a provisioned infrastructure system.

@property
def sys_bin_path(self):

@contextlib.contextmanager
def system_env(self):

Yields a context modified to operate on system paths.

recipe_modules / omahaproxy

DEPS: recipe_engine/raw_io, recipe_engine/url

class OmahaproxyApi(RecipeApi):

APIs for interacting with omahaproxy.

def history(self, min_major_version=None, exclude_platforms=None):

@staticmethod
def split_version(text):

recipe_modules / recipe_autoroller

DEPS: build/luci_config, depot_tools/cipd, depot_tools/gclient, depot_tools/git, depot_tools/git_cl, depot_tools/gsutil, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/tempfile, recipe_engine/time

class RecipeAutorollerApi(RecipeApi):

def roll_projects(self, projects):

Attempts to roll each project from the provided list.

If rolling any of the projects leads to failures, other projects are not affected.

recipe_modules / sync_submodules

DEPS: depot_tools/gclient, depot_tools/git, depot_tools/infra_paths, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/step

class SyncSubmodulesApi(RecipeApi):

def __call__(self, source, dest, source_ref=‘refs/heads/master’, dest_ref=‘refs/heads/master’, extra_submodules=None, deps_path_prefix=None, enable_recurse_deps=False, disable_path_prefix=False):

Args: source: URL of the git repository to mirror. dest: URL of the git repository to push to. source_ref: git ref in the source repository to checkout. dest_ref: git ref in the destination repository to push to. extra_submodules: a list of “path=URL” strings. These are added as extra submodules. deps_path_prefix: path prefix used to filter out DEPS. DEPS with the prefix are included. enable_recurse_deps: enable collecting submodules for recurse deps repos disable_path_prefix: disable filtering out DEPS by path prefix.

recipe_modules / third_party_packages

DEPS: depot_tools/cipd, depot_tools/git, depot_tools/gitiles, windows_sdk, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/url

class ThirdPartyPackagesApi(RecipeApi):

def build_package(self, name, workdir, root, install_mode):

@property
def cmake(self):

@property
def dep(self):

def does_package_exist(self, name, version):

@dry_run.setter
def dry_run(self, v):

def ensure_package(self, workdir, repo_url, package_name_prefix, install_fn, tag, version, cipd_install_mode, test_fn=None):

Ensures that the specified CIPD package exists.

@property
def gcloud(self):

def get_latest_release_tag(self, repo_url, prefix=‘v’):

def get_package_name(self, package_name_prefix):

@property
def git(self):

@property
def go(self):

@property
def gsutil(self):

@property
def ninja(self):

@property
def python(self):

def register_package(self, package_file, name, version):

def support_prefix(self, base):

@property
def swig(self):

recipe_modules / wct

DEPS: depot_tools/cipd, recipe_engine/context, recipe_engine/path, recipe_engine/platform, recipe_engine/step

class WCTApi(RecipeApi):

WCTApi provides support for running Web Component Tests

def install(self):

def run(self, root, prefix=‘test/’, step_name=‘Run WCT tests’):

recipe_modules / windows_sdk

DEPS: recipe_engine/cipd, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/step

class WindowsSDKApi(RecipeApi):

API for using Windows SDK distributed via CIPD.

@contextmanager
def __call__(self, path=None, version=None, enabled=True):

Setups the SDK environment when enabled.

Args: path (path): Path to a directory where to install the SDK (default is ‘[start_dir]/cipd/windows_sdk’) version (str): CIPD instance ID, tag or ref of the SDK (default is set via $infra/windows_sdk.version property) enabled (bool): Whether the SDK should be used or not.

Raises: StepFailure or InfraFailure.

Recipes

recipes / build_conda_cipd_pkg

DEPS: depot_tools/cipd, conda, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties

Recipe to build CIPD package with sealed Conda environment.

Supposed to be used from manually triggered Buildbot builders. We aren't expecting rebuilding this environment often, so setting up and periodic schedule is a waste of resources.

To build a new package for all platforms:

  1. Manually trigger all builders by clicking buttons in Buildbot.
  2. Once they all complete, tag the with some release identifier by running: ./cipd set-tag infra/conda_python/scientific/ -tag=release: -version=latest
  3. Update Puppet configs to use ‘release:’ as a version.

def RunSteps(api):

recipes / build_from_tarball

DEPS: depot_tools/gsutil, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / build_gsutil_cipd_pkg

DEPS: build/zip, depot_tools/cipd, depot_tools/gsutil, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api):

recipes / depot_tools_builder

DEPS: build/zip, depot_tools/cipd, depot_tools/git, depot_tools/gsutil, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/tempfile

Recipe to build windows depot_tools bootstrap zipfile.

def RunSteps(api, revision):

recipes / docker_image_builder

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

def RunSteps(api):

recipes / gae_sdk_cipd_packager

DEPS: build/gae_sdk, build/zip, depot_tools/cipd, depot_tools/gsutil, recipe_engine/path, recipe_engine/python, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/step

def RunSteps(api):

recipes / gerrit_cq_test

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/properties, recipe_engine/step

def RunSteps(api):

recipes / gerrit_hello_world

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Pushes a trivial CL to Gerrit to verify git authentication works on LUCI.

def RunSteps(api):

recipes / git_hash_metric

DEPS: recipe_engine/properties, recipe_engine/python

def RunSteps(api):

recipes / goma_hello_world

DEPS: build/goma, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Compiles trivial C++ program using Goma.

Intended to be used as a very simple litmus test of Goma health on LUCI staging environment. Linux and OSX only.

def RunSteps(api):

recipes / gsutil_hello_world

DEPS: depot_tools/depot_tools, depot_tools/gsutil, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step, recipe_engine/time

Pushes a trivial CL to Gerrit to verify git authentication works on LUCI.

def RunSteps(api):

recipes / infra_checkout:examples/full

DEPS: depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/json, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / infra_cipd:examples/usage

DEPS: infra_cipd, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/runtime, recipe_engine/step

def RunSteps(api):

recipes / infra_continuous

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/depot_tools, depot_tools/gclient, depot_tools/infra_paths, infra_checkout, infra_cipd, infra_system, recipe_engine/buildbucket, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/runtime, recipe_engine/step

def RunSteps(api):

def build_main(api, buildername, project_name, repo_url, rev):

recipes / infra_continuous_luci

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

Builds and tests infra.git code.

Very dumb for now, with no side effects. Runs continuously on LUCI in both staging and prod environments.

def RunSteps(api):

recipes / infra_repo_trybot

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, infra_checkout, infra_system, recipe_engine/context, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api):

recipes / infra_system:examples/full

DEPS: infra_system, recipe_engine/context, recipe_engine/platform, recipe_engine/step

def RunSteps(api):

recipes / infra_wct_tester

DEPS: depot_tools/bot_update, depot_tools/gclient, wct, recipe_engine/path, recipe_engine/platform, recipe_engine/step

def RunSteps(api):

recipes / lkgr_finder

DEPS: build/chromium_checkout, build/v8, build/webrtc, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/gitiles, depot_tools/gsutil, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/step

def RunSteps(api, buildername):

recipes / luci_check

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python

def RunSteps(api):

recipes / luci_gae

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / luci_go

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api, presubmit, GOARCH):

recipes / luci_py

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/properties

def RunSteps(api):

recipes / master_manager_config

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/infra_paths, infra_checkout, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / publish_tarball

DEPS: build/chromium, build/trigger, depot_tools/bot_update, depot_tools/depot_tools, depot_tools/gclient, depot_tools/git, depot_tools/gsutil, omahaproxy, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api):

@contextlib.contextmanager
def copytree_checkout(api):

@recipe_api.composite_step
def export_lite_tarball(api, version):

@recipe_api.composite_step
def export_nacl_tarball(api, version):

@recipe_api.composite_step
def export_tarball(api, args, source, destination):

def gsutil_upload(api, source, bucket, dest, args):

def published_all_tarballs(version, ls_result):

def published_full_tarball(version, ls_result):

def published_lite_tarball(version, ls_result):

def published_nacl_tarball(version, ls_result):

def published_test_tarball(version, ls_result):

recipes / recipe_autoroller

DEPS: build/luci_config, build/puppet_service_account, recipe_autoroller, recipe_engine/json, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

Rolls recipes.cfg dependencies for public projects.

def RunSteps(api, projects, auth_token, service_account):

recipes / recipe_bundler

DEPS: depot_tools/cipd, recipe_engine/path, recipe_engine/properties, recipe_engine/step

def RunSteps(api, recipe_bundler_pkg, recipe_bundler_vers, repo_specs, package_name_prefix, package_name_internal_prefix):

recipes / recipe_roll_tryjob

DEPS: build/luci_config, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/runtime, recipe_engine/service_account, recipe_engine/step

def RunSteps(api, upstream_project, downstream_project):

recipes / recipe_simulation

DEPS: build/luci_config, build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

A continuous builder which runs recipe tests.

def RunSteps(api, project_under_test, auth_with_account):

recipes / recipes_py_continuous

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/gclient, recipe_engine/path, recipe_engine/properties

def RunSteps(api, mastername, buildername, buildnumber):

recipes / remote_execute_dataflow_workflow

DEPS: build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

This recipe is used to execute Dataflow workflows.

If you want a workflow to run at regular intervals, you can configure a builder to run this recipe. Dataflow workflows run on an internal builder, so this step must be completed by a Google employee. See this change for an example: https://chrome-internal-review.googlesource.com/c/chrome/tools/build/+/412934

Builders configured with the name matching “dataflow-workflow-.*” will be automatically monitored for failures.

This recipe uses the dataflow-launcher service account. That account must have the permission to schedule a Dataflow job for your project.

def RunSteps(api, workflow, job_name, gcp_project_id, num_workers, timeout):

recipes / sync_submodules

DEPS: sync_submodules, recipe_engine/properties, recipe_engine/runtime

def RunSteps(api, enable_recurse_deps, disable_path_prefix):

recipes / third_party_packages

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/runtime, recipe_engine/step

This recipe builds and packages third party software, such as Git.

def RunSteps(api):

recipes / third_party_packages:examples/cmake

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘cmake’ building.

During testing, it may be useful to focus on building CMake. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/dep

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Go dependency management tool building.

During testing, it may be useful to focus on building dep. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/gcloud

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Google Cloud SDK building.

During testing, it may be useful to focus on building the SDK. This can be done by running this recipe module directly.

def RunSteps(api, platform_name, platform_bits, dry_run):

recipes / third_party_packages:examples/git

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘git’ building.

During testing, it may be useful to focus on building Git. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/go

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Go toolchain building.

During testing, it may be useful to focus on building Go. This can be done by running this recipe module directly.

def RunSteps(api, platform_name, platform_bits, dry_run):

recipes / third_party_packages:examples/gsutil

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for gsutil building.

During testing, it may be useful to focus on building the gsutil. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/ninja

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘ninja’ building.

During testing, it may be useful to focus on building Ninja. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/python

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties

Recipe for ‘python’ building.

During testing, it may be useful to focus on building Python. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/swig

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘swig’ building.

During testing, it may be useful to focus on building Swig. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / windows_sdk:examples/full

DEPS: windows_sdk, recipe_engine/platform, recipe_engine/properties, recipe_engine/step

def RunSteps(api):

recipes / wpt_export

DEPS: build/chromium, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python

Exports commits in Chromium to the web-platform-tests repo.

This recipe runs the wpt-export script; it is expected to be run as a recurring job at a short interval. It creates pull requests on GitHub for Chromium commits that contain exportable changes, merges these pull requests.

See: //docs/testing/web_platform_tests.md (https://goo.gl/rSRGmZ)

def RunSteps(api):

recipes / wpt_import

DEPS: build/chromium, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

Imports changes from web-platform-tests into Chromium.

This recipe runs the wpt-import script. The import process involves first fetching the latest changes from web-platform-tests, then running the tests via try jobs and and updating any baselines and expectations, before committing to Chromium.

See: //docs/testing/web_platform_tests.md (https://goo.gl/rSRGmZ)

def RunSteps(api):

def git_cl_issue_link(api):

Runs a step which adds a link to the current CL if there is one.