Package documentation for infra

Table of Contents

Recipe Modules

Recipes

Recipe Modules

recipe_modules / conda

DEPS: depot_tools/cipd, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/python, recipe_engine/step, recipe_engine/url

Functions to work with Miniconda python environment.

See http://conda.pydata.org/miniconda.html

class CondaApi(RecipeApi):

def install(self, version, path):

Downloads Miniconda installer for given version and executes it.

Args: version: version of Miniconda to install, e.g. ‘Miniconda2-3.18.3’. path: prefix to install Miniconda into.

Returns: Instance of CondaEnv, that also optionally acts as context manager that deletes the environment on exit.

recipe_modules / infra_checkout

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python

class InfraCheckoutApi(RecipeApi):

Stateless API for using public infra gclient checkout.

def checkout(self, gclient_config_name, patch_root, path=None, named_cache=DEFAULT_NAMED_CACHE, **kwargs):

Fetches infra gclient checkout into a given path OR named_cache.

Arguments:

  • gclient_config_name (string) - name of gclient config.
  • patch_root (path or string) - path inside infra checkout to git repo in which to apply the patch. For example, ‘infra/luci’ for luci-py repo.
  • path (path or string) - path to where to create/update infra checkout. If None (default) - path is cache with customizable name (see below).
  • named_cache - if path is None, this allows to customize the name of the cache. Defaults to DEFAULT_NAMED_CACHE. Note: your cr-buildbucket.cfg should specify named_cache for swarming to prioritize bots which actually have this cache populated by prior runs. Otherwise, using named cache isn't particularly useful, unless your pool of builders is very small.
  • kwargs - passed as is to bot_update.ensure_checkout.

Returns: a Checkout object with commands for common actions on infra checkout.

recipe_modules / infra_system

DEPS: recipe_engine/context, recipe_engine/path, recipe_engine/platform

class InfraSystemApi(RecipeApi):

API for interacting with a provisioned infrastructure system.

@property
def sys_bin_path(self):

@contextlib.contextmanager
def system_env(self):

Yields a context modified to operate on system paths.

recipe_modules / omahaproxy

DEPS: recipe_engine/raw_io, recipe_engine/url

class OmahaproxyApi(RecipeApi):

APIs for interacting with omahaproxy.

def history(self, min_major_version=None, exclude_platforms=None):

@staticmethod
def split_version(text):

recipe_modules / recipe_autoroller

DEPS: build/luci_config, depot_tools/cipd, depot_tools/gclient, depot_tools/git, depot_tools/git_cl, depot_tools/gsutil, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/tempfile, recipe_engine/time

class RecipeAutorollerApi(RecipeApi):

def roll_projects(self, projects):

Attempts to roll each project from the provided list.

If rolling any of the projects leads to failures, other projects are not affected.

recipe_modules / sync_submodules

DEPS: depot_tools/gclient, depot_tools/git, depot_tools/infra_paths, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step

class SyncSubmodulesApi(RecipeApi):

def __call__(self, source, dest, source_ref=‘refs/heads/master’, dest_ref=‘refs/heads/master’, extra_submodules=None, deps_path_prefix=None, enable_recurse_deps=False, disable_path_prefix=False):

Args: source: URL of the git repository to mirror. dest: URL of the git repository to push to. source_ref: git ref in the source repository to checkout. dest_ref: git ref in the destination repository to push to. extra_submodules: a list of “path=URL” strings. These are added as extra submodules. deps_path_prefix: path prefix used to filter out DEPS. DEPS with the prefix are included. enable_recurse_deps: enable collecting submodules for recurse deps repos disable_path_prefix: disable filtering out DEPS by path prefix.

recipe_modules / third_party_packages

DEPS: depot_tools/cipd, depot_tools/git, depot_tools/gitiles, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/url

class ThirdPartyPackagesApi(RecipeApi):

def build_package(self, name, workdir, root, install_mode):

@property
def cmake(self):

def does_package_exist(self, name, version):

@dry_run.setter
def dry_run(self, v):

def ensure_package(self, workdir, repo_url, package_name_prefix, install_fn, tag, version, cipd_install_mode, test_fn=None):

Ensures that the specified CIPD package exists.

@property
def gcloud(self):

def get_latest_release_tag(self, repo_url, prefix=‘v’):

def get_package_name(self, package_name_prefix):

@property
def git(self):

@property
def go(self):

@property
def gsutil(self):

@property
def ninja(self):

@property
def python(self):

def register_package(self, package_file, name, version):

def support_prefix(self, base):

@property
def swig(self):

recipe_modules / wct

DEPS: depot_tools/cipd, recipe_engine/context, recipe_engine/path, recipe_engine/platform, recipe_engine/step

class WCTApi(RecipeApi):

WCTApi provides support for running Web Component Tests

def install(self):

def run(self, root):

Recipes

recipes / build_conda_cipd_pkg

DEPS: depot_tools/cipd, conda, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties

Recipe to build CIPD package with sealed Conda environment.

Supposed to be used from manually triggered Buildbot builders. We aren't expecting rebuilding this environment often, so setting up and periodic schedule is a waste of resources.

To build a new package for all platforms:

  1. Manually trigger all builders by clicking buttons in Buildbot.
  2. Once they all complete, tag the with some release identifier by running: ./cipd set-tag infra/conda_python/scientific/ -tag=release: -version=latest
  3. Update Puppet configs to use ‘release:’ as a version.

def RunSteps(api):

recipes / build_gsutil_cipd_pkg

DEPS: build/zip, depot_tools/cipd, depot_tools/gsutil, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api):

recipes / depot_tools_builder

DEPS: build/zip, depot_tools/cipd, depot_tools/git, depot_tools/gsutil, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/tempfile

Recipe to build windows depot_tools bootstrap zipfile.

def RunSteps(api, revision):

recipes / docker_image_builder

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

def RunSteps(api):

recipes / gerrit_cq_test

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/properties, recipe_engine/step

def RunSteps(api):

recipes / gerrit_hello_world

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Pushes a trivial CL to Gerrit to verify git authentication works on LUCI.

def RunSteps(api):

recipes / git_hash_metric

DEPS: recipe_engine/properties, recipe_engine/python

def RunSteps(api):

recipes / goma_hello_world

DEPS: build/goma, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Compiles trivial C++ program using Goma.

Intended to be used as a very simple litmus test of Goma health on LUCI staging environment. Linux and OSX only.

def RunSteps(api):

recipes / gsutil_hello_world

DEPS: depot_tools/gsutil, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Pushes a trivial CL to Gerrit to verify git authentication works on LUCI.

def RunSteps(api):

recipes / infra_checkout:examples/full

DEPS: depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/json, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / infra_continuous

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/depot_tools, depot_tools/gclient, depot_tools/infra_paths, infra_system, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api, mastername, buildername, buildnumber):

def build_cipd_packages(api, repo, rev, mastername, buildername, buildnumber, goos, goarch):

def build_luci(api):

def build_main(api, mastername, buildername, buildnumber, project_name, repo_name, rev):

recipes / infra_continuous_luci

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

Builds and tests infra.git code.

Very dumb for now, with no side effects. Runs continuously on LUCI in both staging and prod environments.

def RunSteps(api):

recipes / infra_repo_trybot

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, infra_system, recipe_engine/context, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api):

recipes / infra_system:examples/full

DEPS: infra_system, recipe_engine/context, recipe_engine/platform, recipe_engine/step

def RunSteps(api):

recipes / isolate_go_perf

DEPS: build/perf_dashboard, depot_tools/bot_update, depot_tools/depot_tools, depot_tools/gclient, depot_tools/git, depot_tools/gsutil, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

def RunSteps(api):

def build_isolate(api):

def download_test_data(api):

def get_file_size(api, name, path):

Returns file size if file exists, else None

def post_to_perf_dashboard(api, taken_seconds, revision, tstamp_iso_str, tstamp_unix):

def upload_isolate(api, isolate):

recipes / lkgr_finder

DEPS: build/chromium_checkout, build/v8, build/webrtc, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/gitiles, depot_tools/gsutil, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io

def RunSteps(api, buildername):

recipes / luci_gae

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / luci_go

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api, presubmit, GOARCH):

recipes / luci_py

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/properties

def RunSteps(api):

recipes / master_manager_config

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/infra_paths, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api):

recipes / publish_tarball

DEPS: build/chromium, build/trigger, depot_tools/bot_update, depot_tools/depot_tools, depot_tools/gclient, depot_tools/git, depot_tools/gsutil, omahaproxy, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api):

@contextlib.contextmanager
def copytree_checkout(api):

@recipe_api.composite_step
def export_lite_tarball(api, version):

@recipe_api.composite_step
def export_nacl_tarball(api, version):

@recipe_api.composite_step
def export_tarball(api, args, source, destination):

def gsutil_upload(api, source, bucket, dest, args):

recipes / recipe_autoroller

DEPS: build/luci_config, build/puppet_service_account, recipe_autoroller, recipe_engine/json, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

Rolls recipes.cfg dependencies for public projects.

def RunSteps(api, projects, auth_token, service_account):

recipes / recipe_roll_tryjob

DEPS: build/luci_config, build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

def RunSteps(api, upstream_project, downstream_project, auth_token, service_account):

recipes / recipe_simulation

DEPS: build/luci_config, build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

A continuous builder which runs recipe tests.

def RunSteps(api, project_under_test, auth_with_account):

recipes / recipes_py_continuous

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/gclient, recipe_engine/path, recipe_engine/properties

def RunSteps(api, mastername, buildername, buildnumber):

recipes / remote_execute_dataflow_workflow

DEPS: build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

def RunSteps(api, workflow, job_name):

recipes / sheriff-o-matic-wct

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/gclient, wct, recipe_engine/path, recipe_engine/platform, recipe_engine/step

def RunSteps(api):

recipes / sync_submodules

DEPS: sync_submodules, recipe_engine/properties

def RunSteps(api, enable_recurse_deps, disable_path_prefix):

recipes / third_party_packages

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/step

This recipe builds and packages third party software, such as Git.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/cmake

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘cmake’ building.

During testing, it may be useful to focus on building CMake. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/gcloud

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Google Cloud SDK building.

During testing, it may be useful to focus on building the SDK. This can be done by running this recipe module directly.

def RunSteps(api, platform_name, platform_bits, dry_run):

recipes / third_party_packages:examples/git

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘git’ building.

During testing, it may be useful to focus on building Git. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/go

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Go toolchain building.

During testing, it may be useful to focus on building Go. This can be done by running this recipe module directly.

def RunSteps(api, platform_name, platform_bits, dry_run):

recipes / third_party_packages:examples/gsutil

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for gsutil building.

During testing, it may be useful to focus on building the gsutil. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/ninja

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘ninja’ building.

During testing, it may be useful to focus on building Ninja. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/python

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties

Recipe for ‘python’ building.

During testing, it may be useful to focus on building Python. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / third_party_packages:examples/swig

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘swig’ building.

During testing, it may be useful to focus on building Swig. This can be done by running this recipe module directly.

def RunSteps(api, dry_run):

recipes / wpt_export

DEPS: build/chromium, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python

Exports commits in Chromium to the web-platform-tests repo.

This recipe runs the wpt-export script; it is expected to be run as a recurring job at a short interval. It creates pull requests on GitHub for Chromium commits that contain exportable changes, merges these pull requests.

See: //docs/testing/web_platform_tests.md (https://goo.gl/rSRGmZ)

def RunSteps(api):

recipes / wpt_import

DEPS: build/chromium, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

Imports changes from web-platform-tests into Chromium.

This recipe runs the wpt-import script. The import process involves first fetching the latest changes from web-platform-tests, then running the tests via try jobs and and updating any baselines and expectations, before committing to Chromium.

See: //docs/testing/web_platform_tests.md (https://goo.gl/rSRGmZ)

def RunSteps(api):

def git_cl_issue_link(api):

Runs a step which adds a link to the current CL if there is one.