Chrome infrastructure

Package documentation for infra

Table of Contents

Recipe Modules

Recipes

Recipe Modules

recipe_modules / conda

DEPS: depot_tools/cipd, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/python, recipe_engine/step, recipe_engine/url

Functions to work with Miniconda python environment.

See http://conda.pydata.org/miniconda.html

class CondaApi(RecipeApi):

— def install(self, version, path):

Downloads Miniconda installer for given version and executes it.

Args: version: version of Miniconda to install, e.g. ‘Miniconda2-3.18.3’. path: prefix to install Miniconda into.

Returns: Instance of CondaEnv, that also optionally acts as context manager that deletes the environment on exit.

recipe_modules / infra_checkout

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/python

class InfraCheckoutApi(RecipeApi):

Stateless API for using public infra gclient checkout.

— def checkout(self, gclient_config_name, patch_root, path=None, named_cache=DEFAULT_NAMED_CACHE, **kwargs):

Fetches infra gclient checkout into a given path OR named_cache.

Arguments:

  • gclient_config_name (string) - name of gclient config.
  • patch_root (path or string) - path inside infra checkout to git repo in which to apply the patch. For example, ‘infra/luci’ for luci-py repo.
  • path (path or string) - path to where to create/update infra checkout. If None (default) - path is cache with customizable name (see below).
  • named_cache - if path is None, this allows to customize the name of the cache. Defaults to DEFAULT_NAMED_CACHE. Note: your cr-buildbucket.cfg should specify named_cache for swarming to prioritize bots which actually have this cache populated by prior runs. Otherwise, using named cache isn't particularly useful, unless your pool of builders is very small.
  • kwargs - passed as is to bot_update.ensure_checkout.

Returns: a Checkout object with commands for common actions on infra checkout.

recipe_modules / infra_system

DEPS: recipe_engine/context, recipe_engine/path, recipe_engine/platform

class InfraSystemApi(RecipeApi):

API for interacting with a provisioned infrastructure system.

  @property
— def sys_bin_path(self):

  @contextlib.contextmanager
— def system_env(self):

Yields a context modified to operate on system paths.

recipe_modules / omahaproxy

DEPS: recipe_engine/raw_io, recipe_engine/url

class OmahaproxyApi(RecipeApi):

APIs for interacting with omahaproxy.

— def history(self, min_major_version=None, exclude_platforms=None):

  @staticmethod
— def split_version(text):

recipe_modules / recipe_autoroller

DEPS: build/luci_config, depot_tools/cipd, depot_tools/gclient, depot_tools/git, depot_tools/git_cl, depot_tools/gsutil, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/tempfile, recipe_engine/time

class RecipeAutorollerApi(RecipeApi):

— def roll_projects(self, projects):

Attempts to roll each project from the provided list.

If rolling any of the projects leads to failures, other projects are not affected.

recipe_modules / sync_submodules

DEPS: depot_tools/gclient, depot_tools/git, depot_tools/infra_paths, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step

class SyncSubmodulesApi(RecipeApi):

— def __call__(self, source, dest, source_ref=‘refs/heads/master’, dest_ref=‘refs/heads/master’, extra_submodules=None, deps_path_prefix=None, enable_recurse_deps=False, disable_path_prefix=False):

Args: source: URL of the git repository to mirror. dest: URL of the git repository to push to. source_ref: git ref in the source repository to checkout. dest_ref: git ref in the destination repository to push to. extra_submodules: a list of “path=URL” strings. These are added as extra submodules. deps_path_prefix: path prefix used to filter out DEPS. DEPS with the prefix are included. enable_recurse_deps: enable collecting submodules for recurse deps repos disable_path_prefix: disable filtering out DEPS by path prefix.

recipe_modules / third_party_packages

DEPS: depot_tools/cipd, depot_tools/git, depot_tools/gitiles, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/url

class ThirdPartyPackagesApi(RecipeApi):

— def build_package(self, name, workdir, root, install_mode):

  @property
— def cmake(self):

  @property
— def dep(self):

— def does_package_exist(self, name, version):

  @dry_run.setter
— def dry_run(self, v):

— def ensure_package(self, workdir, repo_url, package_name_prefix, install_fn, tag, version, cipd_install_mode, test_fn=None):

Ensures that the specified CIPD package exists.

  @property
— def gcloud(self):

— def get_latest_release_tag(self, repo_url, prefix=‘v’):

— def get_package_name(self, package_name_prefix):

  @property
— def git(self):

  @property
— def go(self):

  @property
— def gsutil(self):

  @property
— def ninja(self):

  @property
— def python(self):

— def register_package(self, package_file, name, version):

— def support_prefix(self, base):

  @property
— def swig(self):

recipe_modules / wct

DEPS: depot_tools/cipd, recipe_engine/context, recipe_engine/path, recipe_engine/platform, recipe_engine/step

class WCTApi(RecipeApi):

WCTApi provides support for running Web Component Tests

— def install(self):

— def run(self, root):

Recipes

recipes / build_conda_cipd_pkg

DEPS: depot_tools/cipd, conda, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties

Recipe to build CIPD package with sealed Conda environment.

Supposed to be used from manually triggered Buildbot builders. We aren't expecting rebuilding this environment often, so setting up and periodic schedule is a waste of resources.

To build a new package for all platforms:

  1. Manually trigger all builders by clicking buttons in Buildbot.
  2. Once they all complete, tag the with some release identifier by running: ./cipd set-tag infra/conda_python/scientific/ -tag=release: -version=latest
  3. Update Puppet configs to use ‘release:’ as a version.

— def RunSteps(api):

recipes / build_gsutil_cipd_pkg

DEPS: build/zip, depot_tools/cipd, depot_tools/gsutil, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

— def RunSteps(api):

recipes / depot_tools_builder

DEPS: build/zip, depot_tools/cipd, depot_tools/git, depot_tools/gsutil, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/tempfile

Recipe to build windows depot_tools bootstrap zipfile.

— def RunSteps(api, revision):

recipes / docker_image_builder

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/file, recipe_engine/path, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

— def RunSteps(api):

recipes / gerrit_cq_test

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/properties, recipe_engine/step

— def RunSteps(api):

recipes / gerrit_hello_world

DEPS: recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Pushes a trivial CL to Gerrit to verify git authentication works on LUCI.

— def RunSteps(api):

recipes / git_hash_metric

DEPS: recipe_engine/properties, recipe_engine/python

— def RunSteps(api):

recipes / goma_hello_world

DEPS: build/goma, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Compiles trivial C++ program using Goma.

Intended to be used as a very simple litmus test of Goma health on LUCI staging environment. Linux and OSX only.

— def RunSteps(api):

recipes / gsutil_hello_world

DEPS: depot_tools/gsutil, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/step, recipe_engine/time

Pushes a trivial CL to Gerrit to verify git authentication works on LUCI.

— def RunSteps(api):

recipes / infra_checkout:examples/full

DEPS: depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/json, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

— def RunSteps(api):

recipes / infra_continuous

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/depot_tools, depot_tools/gclient, depot_tools/infra_paths, infra_system, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

— def RunSteps(api, mastername, buildername, buildnumber):

— def build_cipd_packages(api, repo, rev, mastername, buildername, buildnumber, goos, goarch):

— def build_luci(api):

— def build_main(api, mastername, buildername, buildnumber, project_name, repo_name, rev):

recipes / infra_continuous_luci

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

Builds and tests infra.git code.

Very dumb for now, with no side effects. Runs continuously on LUCI in both staging and prod environments.

— def RunSteps(api):

recipes / infra_repo_trybot

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, infra_system, recipe_engine/context, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

— def RunSteps(api):

recipes / infra_system:examples/full

DEPS: infra_system, recipe_engine/context, recipe_engine/platform, recipe_engine/step

— def RunSteps(api):

recipes / isolate_go_perf

DEPS: build/perf_dashboard, depot_tools/bot_update, depot_tools/depot_tools, depot_tools/gclient, depot_tools/git, depot_tools/gsutil, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

— def RunSteps(api):

— def build_isolate(api):

— def download_test_data(api):

— def get_file_size(api, name, path):

Returns file size if file exists, else None

— def post_to_perf_dashboard(api, taken_seconds, revision, tstamp_iso_str, tstamp_unix):

— def upload_isolate(api, isolate):

recipes / lkgr_finder

DEPS: build/chromium_checkout, build/v8, build/webrtc, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/gitiles, depot_tools/gsutil, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

— def RunSteps(api, buildername):

recipes / luci_check

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python

— def RunSteps(api):

recipes / luci_gae

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

— def RunSteps(api):

recipes / luci_go

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/presubmit, depot_tools/tryserver, infra_checkout, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

— def RunSteps(api, presubmit, GOARCH):

recipes / luci_py

DEPS: depot_tools/bot_update, depot_tools/gclient, recipe_engine/properties

— def RunSteps(api):

recipes / master_manager_config

DEPS: depot_tools/bot_update, depot_tools/gclient, depot_tools/infra_paths, recipe_engine/json, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/step

— def RunSteps(api):

recipes / publish_tarball

DEPS: build/chromium, build/trigger, depot_tools/bot_update, depot_tools/depot_tools, depot_tools/gclient, depot_tools/git, depot_tools/gsutil, omahaproxy, recipe_engine/context, recipe_engine/file, recipe_engine/path, recipe_engine/platform, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

— def RunSteps(api):

  @contextlib.contextmanager
— def copytree_checkout(api):

  @recipe_api.composite_step
— def export_lite_tarball(api, version):

  @recipe_api.composite_step
— def export_nacl_tarball(api, version):

  @recipe_api.composite_step
— def export_tarball(api, args, source, destination):

— def gsutil_upload(api, source, bucket, dest, args):

recipes / recipe_autoroller

DEPS: build/luci_config, build/puppet_service_account, recipe_autoroller, recipe_engine/json, recipe_engine/properties, recipe_engine/raw_io, recipe_engine/step, recipe_engine/time

Rolls recipes.cfg dependencies for public projects.

— def RunSteps(api, projects, auth_token, service_account):

recipes / recipe_roll_tryjob

DEPS: build/luci_config, build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, depot_tools/tryserver, recipe_engine/context, recipe_engine/file, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/raw_io, recipe_engine/step

— def RunSteps(api, upstream_project, downstream_project, auth_token, service_account):

recipes / recipe_simulation

DEPS: build/luci_config, build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

A continuous builder which runs recipe tests.

— def RunSteps(api, project_under_test, auth_with_account):

recipes / recipes_py_continuous

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/gclient, recipe_engine/path, recipe_engine/properties

— def RunSteps(api, mastername, buildername, buildnumber):

recipes / remote_execute_dataflow_workflow

DEPS: build/puppet_service_account, depot_tools/bot_update, depot_tools/gclient, recipe_engine/context, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

This recipe is used to execute Dataflow workflows.

If you want a workflow to run at regular intervals, you can configure a builder to run this recipe. Dataflow workflows run on an internal builder, so this step must be completed by a Google employee. See this change for an example: https://chrome-internal-review.googlesource.com/c/chrome/tools/build/+/412934

Builders configured with the name matching “dataflow-workflow-.*” will be automatically monitored for failures.

This recipe uses the dataflow-launcher service account. That account must have the permission to schedule a Dataflow job for your project.

— def RunSteps(api, workflow, job_name, gcp_project_id, num_workers, timeout):

recipes / sheriff-o-matic-wct

DEPS: depot_tools/bot_update, depot_tools/cipd, depot_tools/gclient, wct, recipe_engine/path, recipe_engine/platform, recipe_engine/step

— def RunSteps(api):

recipes / sync_submodules

DEPS: sync_submodules, recipe_engine/properties

— def RunSteps(api, enable_recurse_deps, disable_path_prefix):

recipes / third_party_packages

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/step

This recipe builds and packages third party software, such as Git.

— def RunSteps(api, dry_run):

recipes / third_party_packages:examples/cmake

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘cmake’ building.

During testing, it may be useful to focus on building CMake. This can be done by running this recipe module directly.

— def RunSteps(api, dry_run):

recipes / third_party_packages:examples/dep

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Go dependency management tool building.

During testing, it may be useful to focus on building dep. This can be done by running this recipe module directly.

— def RunSteps(api, dry_run):

recipes / third_party_packages:examples/gcloud

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Google Cloud SDK building.

During testing, it may be useful to focus on building the SDK. This can be done by running this recipe module directly.

— def RunSteps(api, platform_name, platform_bits, dry_run):

recipes / third_party_packages:examples/git

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘git’ building.

During testing, it may be useful to focus on building Git. This can be done by running this recipe module directly.

— def RunSteps(api, dry_run):

recipes / third_party_packages:examples/go

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for Go toolchain building.

During testing, it may be useful to focus on building Go. This can be done by running this recipe module directly.

— def RunSteps(api, platform_name, platform_bits, dry_run):

recipes / third_party_packages:examples/gsutil

DEPS: depot_tools/cipd, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for gsutil building.

During testing, it may be useful to focus on building the gsutil. This can be done by running this recipe module directly.

— def RunSteps(api, dry_run):

recipes / third_party_packages:examples/ninja

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘ninja’ building.

During testing, it may be useful to focus on building Ninja. This can be done by running this recipe module directly.

— def RunSteps(api, dry_run):

recipes / third_party_packages:examples/python

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties

Recipe for ‘python’ building.

During testing, it may be useful to focus on building Python. This can be done by running this recipe module directly.

— def RunSteps(api, dry_run):

recipes / third_party_packages:examples/swig

DEPS: depot_tools/cipd, depot_tools/gitiles, third_party_packages, recipe_engine/platform, recipe_engine/properties, recipe_engine/url

Recipe for ‘swig’ building.

During testing, it may be useful to focus on building Swig. This can be done by running this recipe module directly.

— def RunSteps(api, dry_run):

recipes / wpt_export

DEPS: build/chromium, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python

Exports commits in Chromium to the web-platform-tests repo.

This recipe runs the wpt-export script; it is expected to be run as a recurring job at a short interval. It creates pull requests on GitHub for Chromium commits that contain exportable changes, merges these pull requests.

See: //docs/testing/web_platform_tests.md (https://goo.gl/rSRGmZ)

— def RunSteps(api):

recipes / wpt_import

DEPS: build/chromium, depot_tools/bot_update, depot_tools/gclient, depot_tools/git, recipe_engine/context, recipe_engine/json, recipe_engine/path, recipe_engine/properties, recipe_engine/python, recipe_engine/step

Imports changes from web-platform-tests into Chromium.

This recipe runs the wpt-import script. The import process involves first fetching the latest changes from web-platform-tests, then running the tests via try jobs and and updating any baselines and expectations, before committing to Chromium.

See: //docs/testing/web_platform_tests.md (https://goo.gl/rSRGmZ)

— def RunSteps(api):

— def git_cl_issue_link(api):

Runs a step which adds a link to the current CL if there is one.