tree: b09ac375178c9f35395ef9df998adbb5fcea10fb [path history] [tgz]
  1. gae/
  2. images/
  3. out/
  4. packages/
  5. tests/
  6. .vpython
  7. .vpython3


Scripts and files in this directory describe how to build CIPD packages from the source code in infra.git repo.

There are two flavors of packages:

  • Packages with executables compiled from Go code.
  • Single giant package with all python code and archived virtual environment needed to run it.

Package definition

A package is defined in a YAML file that is parsed by script before being passed to the CIPD client.

The package file has the following structure:

# Name of the package in CIPD repository.
package: infra/example/package/${platform}

# Human readable description of the package.
description: Example package

# Set to true to exclude the package from the build.
disabled: false

# Set to false to not update the 'latest' CIPD ref.
update_latest_ref: false

# Optional filter with a list of CIPD platform suffixes for which to build this
# package. If not specified, will be built only for the host platform. Note that
# this is a filter: whenever script is invoked with some GOOS and
# GOARCH, it looks at this list to decide whether to build the package or not.
# The list of platforms we attempt to build for is thus outside of
# control: it is specified by whoever calls (see
# recipe).
  - android-amd64
  - android-armv6l
  - linux-386
  - linux-amd64
  - linux-arm64
  - linux-armv6l
  - linux-mipsle
  - linux-mips64
  - linux-mips64le
  - linux-ppc64
  - linux-ppc64le
  - linux-s390x
  - mac-amd64
  - windows-386
  - windows-amd64
  - windows-arm64

# Optional list of go packages to 'go install' before zipping this package.
  - ...

# Environment variables to set when building go code. Only CGO_ENABLED is
# recognized currently.
  # If given, overrides CGO_ENABLED env var when building a **native** variant
  # of this package (i.e. when not cross-compiling).
  # Note that it is also possible to specify this on per-target GOOS basis, by
  # using a dictionary as a value, e.g. {'darwin': 1, 'windows': 0, 'linux': 0}.
  # When cross-compiling CGO_ENABLED is taken from the host environment as
  # usual, defaulting to 0.

  # If given, attach the flags to the go build command.
  # Note that it is also possible to specify this on per-target GOOS basis, by
  # using a dictionary as a value, e.g. {'darwin': 1, 'windows': 0, 'linux': 0}.
  # The flag will only be attached to the go build command if:
  #   * the flag is given;
  #   * it is supported on the platform:
  #     * As of Aug 2022, it is only supported on
  #       * linux/amd64
  #       * freebsd/amd64
  #       * darwin/amd64
  #       * windows/amd64
  #       * linux/ppc64le
  #       * linux/arm64 (only for 48-bit VMA)
  #   * CGO is enabled.
  race: 0

# Path to the root of the package source files on the system we're building
# the package from. Can be absolute or relative to the path of the *.yaml
# file itself.
root: ../..

# Copies apply after the build, but before creating the package. `dst` files
# will automatically be added as data 'file' entries.
# generate_bat_shim works as it does for data 'file' entries below, and is
# optional. platforms optionally specifies which platforms to target. If
# unspecified, targets all platforms.
  - src: some_built_file
    dst: some_other_file
    generate_bat_shim: true
    - windows-386
    - windows-amd64
    - windows-arm64

# Symlinks apply after the build, but before creating the package. `dst` links
# are automatically be added as data 'file' entries. These will only run when
# targetting posix platforms (mac, linux), and are otherwise ignored.
  - src: some_file
    dst: some_link_name

# CIPD packages may be exported into the build directory by embedding an
# ensure file (
# In this specification, ${target_platform} may be used for the cross-compile
# target, where as ${platform} refers to the host platform.
# Make sure to also include this path in `data` below.
cipd_export: |
  @Subdir pkg/path
  cipd/pkg/path/${target_platform} version:2.0

  # 'dir' section adds a subdirectory of 'root' to the package. In this case
  # it will scan directory <yaml_path>/../../a/b/c and put files into a/b/c
  # directory of the package.
  - dir: a/b/c
    # A list of regular expressions for files to exclude from the package.
    # Syntax is defined at Each expression
    # is implicitly wrapped into ^...$. The tests are applied to paths relative
    # to 'dir', e.g. 'bin/active' regexp matches only single file
    # <yaml_path>/../../a/b/c/bin/active.
      - bin/activate
      - .*\.pyc

  # 'file' section adds a single file to the package.
  - file:

  # Exe files can also be augmented with *.bat shims, residing in same dir.
  # Kicks in only when the package is targeting Windows.
  - file: cipd.exe
    generate_bat_shim: true

  # Only uploading the package when specified files in the cipd package changed.
  # It will generate on-change tags and search if there is any package with
  # same set of tags. If no such package available, package will be uploaded. In
  # other case only tag will be updated on the package with same set of tags.
  - file: vpython
    upload_on_change: true

  # Include the CIPD package installed above.
  - dir: pkg/path

Following features of the package definition are implemented by (basically anything related to the process of building the code and preparing all necessary files for packaging):

  • platforms
  • go_build_environ
  • go_packages
  • copies
  • posix_symlinks
  • generate_bat_shim
  • upload_on_change

Strings interpolation

Any string in package definition can reference a variable via ${var_name}, for example:

package: infra/tools/cipd/${platform}

Available variables are defined in in get_package_vars:

  • ${exe_suffix} is .exe on Windows and empty string on other platforms. If cross-compiling to Windows, it is also set to .exe regardless of the host platform.
  • ${platform} defines where is running (if not cross-compiling) or what the target platform is (when cross-compiling), as (flavor)-(bitness) string. It is suitable for packages that do not depend much on the exact version of the OS, for example packages with statically linked binaries. Examples of possible combinations:
    • linux-amd64
    • linux-386
    • linux-armv6l
    • mac-amd64
    • mac-386
    • windows-amd64
    • windows-arm64
    • windows-386
    • ...

See packages for examples of package definitions.

Build script script does the following:

  • Ensures python virtual environment directory (ENV) is up to date.
  • Rebuilds all necessary Go code from scratch and installs binaries into GOBIN.
  • Enumerates packages/ directory for package definition files, builds and (if --upload option is passed) uploads CIPD packages to the repository.
  • Stores built packages into out/ (as *.cipd files).

Package definition files can assume that Go infra code is built and all artifacts are installed in GOBIN (which is go/bin).

You can also pass one or more *.yaml file names to build only specific packages: infra_python cipd_client

Verifying a package

To install a built package locally use cipd client binary (it is built by as well). For example, to rebuild and install infra_python.cipd into ./install_dir, run:

cd infra.git/
rm -rf install_dir
./build/ infra_python
./go/bin/cipd pkg-deploy -root=install_dir build/out/infra_python.cipd
cd install_dir

Package tests script can be used to run simple package integrity tests to verify a built package looks good after deploy.

For each .yaml in packages/ there can be corresponding .py file in tests/ that is invoked by to check that deployed package looks good.

Basically does the following:

  • Installs a CIPD file to a local directory or update currently installed version there (if --work-dir is used).
  • Runs python test/<name>.py with cwd == installation directory.
  • If test returns 0, considers it success, otherwise - failure.

Thus to test that infra_python.cipd package works, one can do the following:

./build/ infra_python
./build/ infra_python is used on CI builders to verify packages look good before uploading them.

Cross compilation of Go code script recognizes GOOS and GOARCH environment variables used to specify a target platform when cross-compiling Go code. When it detects them, it builds only Go packages that have the target CIPD platform specified in the platforms list in the package definition YAML. It also changes the meaning of ${platform} and ${exe_suffix} to match the values for the target platform.

Built packages have +${platform} suffix in file names and coexist with native package in build output directory. When uploading packages (via --no-rebuild --upload), GOOS and GOARCH are used to figure out what flavor of built packages to pick (what +${platform} to search for).

Cross compiling toolset doesn't include C compiler, so the binaries are built in CGO_ENABLED=0 mode, meaning some stdlib functions that depend on libc are not working or working differently compared to natively built executables.