gclient sync.You‘ll first want somewhere in the repository in which your dependency will live. For third-party dependencies, this should typically be a subdirectory of //third_party. You’ll need to add the same set of things to that directory that you'd add for a non-CIPD dependency -- OWNERS, README.chromium, etc.
For example, if you want to add a package named sample_cipd_dep, you might create the following:
third_party/
sample_cipd_dep/
LICENSE
OWNERS
README.chromium
For more on third-party dependencies, see adding_to_third_party.md.
The 3pp subdirectory will store all the 3pp related files, including a 3pp spec (3pp.pb), as well as scripts, patches and/or tools to build the software from source. It should be placed directly under the package directory.
Staying with the example from above, the sample_cipd_dep directory may be like the following.
Note that among the files in 3pp subdirectory, the
3pp.pbis always required. The rest are optional, depending on how the3pp.pbis specified.
third_party/
sample_cipd_dep/
LICENSE
OWNERS
README.chromium
3pp/
3pp.pb # REQUIRED
bootstrap.py
fetch.py
install.sh
install_win.sh
patches/
0001-foo.patch
3pp.pb (Required)3pp.pb is a text proto specified by the spec.proto schema. It is broken up into two main sections:
create: allows you to specify how the package software gets created, and allows specifying differences in how it's fetched/built/tested on a per-target basis. See here for more details.upload: contains some details on how the final result gets uploaded to CIPD. See here for more details.Staying with the example from above, the file sample_cipd_dep/3pp/3pp.pb may be like the following:
create {
source {
url {
download_url: "https://some_url_link/foo.zip"
version: "1.0.0"
extension: ".zip"
}
patch_version: "cr0"
unpack_archive: true
}
}
upload {
pkg_prefix: "tools"
universal: true
}
While the above example could meet most of the use case, 3pp.pb is capable of handling more complicated use case like the following:
# create section that is shared by linux-.* and mac-.* platforms
create {
platform_re: "linux-.*|mac-.*"
source {
git {
repo: "<one_git_repo>"
tag_pattern: "v%s",
# Fixed to 3.8.x for now.
version_restriction: { op: LT val: "3.9a0"}
}
patch_dir: "patches"
}
build {
# Can also leave as blank since the script name defaults to "install.sh"
install: "install.sh"
}
}
# create section that is specific to linux-.* platforms
create {
platform_re: "linux-.*"
build {
dep: "<dep_foo>"
dep: "<dep_bar>"
tool: "<tool_foo>"
}
}
# create section that is specific to linux-arm.* and linux-mips.* platforms
create {
platform_re: "linux-arm.*|linux-mips.*"
build {
tool: "<tool_bar>"
}
}
# create section that is specific to windows-*
create {
platform_re: "windows-.*"
source { script { name: "fetch.py" } }
build {
install: "install_win.sh"
}
}
upload { pkg_prefix: "tools" }
install.sh (Optional)When the build message is specified in 3pp.pb, the file specified in “build.install” (Default to “install.sh”) will be run to transform the source into the built product.
Staying with the example from above, the file sample_cipd_dep/3pp/install.sh may be like the following.
Note that during the build stage, the 3pp directory and all its dependent 3pp directories (i.e. the
toolanddepfrom thebuildmessage in3pp.pb) will be copied to a different directory. So commands ininstall.shshould not refer to files that are outside of these directories.
#!/bin/bash
# Copyright 2021 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
set -e
set -x
set -o pipefail
# An auto-created directory whose content will ultimately be uploaded to CIPD.
# So the commands below should output the built product to this directory.
PREFIX="$1"
# Commands to transform the source into the built product and move it to $PREFIX
./configure
make install
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
cp -a bin_foo "$SCRIPT_DIR/bootstrap.py" "$PREFIX"
fetch.py (Optional)When specifying the source in 3pp.pb, it is possible to use a custom catch-all script to probe for the latest version and obtain the latest sources. A simple example can be like the following:
#!/usr/bin/env python
# Copyright 2021 The Chromium Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
from __future__ import print_function
import argparse
import json
import os
import urllib
def do_latest():
print(urllib.urlopen('some_url/master/VERSION').read().strip())
def get_download_url(version, platform):
target_os, target_arch = platform.split('-')
ext = '.zip' if target_os == 'windows' else '.tar.gz'
partial_manifest = {
'url': ['download_url1', 'download_url2'],
'ext': ext,
}
print(json.dumps(partial_manifest))
def main():
ap = argparse.ArgumentParser()
sub = ap.add_subparsers()
latest = sub.add_parser("latest")
latest.set_defaults(func=lambda _opts: do_latest())
download = sub.add_parser("get_url")
download.set_defaults(
func=lambda _opts: get_download_url(
os.environ['_3PP_VERSION'], os.environ['_3PP_PLATFORM']
)
)
opts = ap.parse_args()
opts.func(opts)
if __name__ == '__main__':
main()
chromium/src repoThe following are the optional CQ builders to run the presubmit check for CLs that have the directory “3pp” in the patchset.
Once the CLs pass the CQ and get landed, the following CI builders will periodically build all the 3pp packages that match the given platforms and upload any new results to CIPD.
Once your CIPD package is created by the 3pp CI builders, you can add it to DEPS by adding an entry of the following form to the deps dict:
deps = {
# ...
# This is the installation directory.
'src/third_party/sample_cipd_dep': {
# In this example, we're only installing one package in this location,
# but installing multiple package in a location is supported.
'packages': [
{
'package': 'chromium/third_party/sample_cipd_dep',
'version': 'TX7HeY1_1JLwFVx-xiETOpT8YK4W5CbyO26SpmaMA0IC',
},
],
# As with git-based DEPS entries, 'condition' is optional.
'condition': 'checkout_android',
'dep_type': 'cipd',
},
# ...
}
This will result in CIPD package chromium/third_party/sample_cipd_dep at TX7HeY1_1JLwFVx-xiETOpT8YK4W5CbyO26SpmaMA0IC being installed in src/third_party/sample_cipd_dep (relative to the gclient root directory).
To modify a CIPD dependency, follow steps 2 and 3 above, then modify the version listed in DEPS.
While it is strongly suggested to use 3pp infrastructure, there are existing flows that create a cipd.yaml file by a GN template or a script, and upload it to CIPD by builders with custom recipes.
Examples are:
The cipd_package_definition template in build/cipd/cipd.gni can be used to create the yaml definition as part of Chromium's normal build process. Declare a target like:
cipd_package_definition("my_cipd_package") {
package = "path/to/cipd/package"
description = "Prebuilt test binary."
install_mode = "copy"
deps = [ "//path/to:test_binary_target" ]
sources = [ "//path/to:test_binary_file" ]
}
You can check a package's ACLs with cipd acl-list:
$ cipd acl-list chromium/third_party/sample_cipd_dep ...
Permissions in CIPD are handled hierarchically. You can check entries higher in the package hierarchy with cipd acl-list, too:
$ cipd acl-list chromium ...
By default, cria/project-chromium-cipd-owners own all CIPD packages under chromium/. If you're adding a package, talk to one of them.
To obtain write access to a new package, ask an owner to run:
$ cipd acl-edit chromium/third_party/sample_cipd_dep -owner user:email@address.com
CIPD currently caches installation state. Modifying packages managed by CIPD will invalidate this cache in a way that CIPD doesn't detect - i.e., CIPD will assume that anything it installed is still installed, even if you deleted it. To clear the cache and force a full reinstallation, delete your $GCLIENT_ROOT/.cipd directory.
Note that there is a bug on file where gclient sync does not reset CIPD entries that are changed locally.