docs/cipd.md

# CIPD for chromium dependencies

[TOC]

## What is CIPD?
* CIPD stands for "Chrome Infrastructure Package Deployment".
* Its code and docs [live within the luci-go project][CIPD].
* Chromium uses CIPD to avoid checking large binary files into git, which git
  does not handle well.
* gclient supports CIPD packages in the same way as git repositories. They are
  specified in [DEPS] and updated via `gclient sync`.
* You can [browse Chromium's CIPD repository][browse] online.

[CIPD]: https://chromium.googlesource.com/infra/luci/luci-go/+/master/cipd/README.md
[DEPS]: /DEPS
[browse]: https://chrome-infra-packages.appspot.com/p/chromium

## Adding a new CIPD dependency

### 1. Set up a new directory for your dependency

You'll first want somewhere in the repository in which your dependency will
live. For third-party dependencies, this should typically be a subdirectory
of `//third_party`. You'll need to add the same set of things to that
directory that you'd add for a non-CIPD dependency -- OWNERS, README.chromium,
etc.

For example, if you want to add a package named `sample_cipd_dep`, you might
create the following:

```
  third_party/
    sample_cipd_dep/
      LICENSE
      OWNERS
      README.chromium
```

For more on third-party dependencies, see [adding_to_third_party.md].

[adding_to_third_party.md]: /docs/adding_to_third_party.md

### 2. Acquire whatever you want to package

Build it, download it, whatever. Once you've done that, lay it out
in your local checkout the way you want it to be laid out in a typical
checkout.

Staying with the example from above, if you want to add a package named
`sample_cipd_dep` that consists of two JARs, `foo.jar` and `bar.jar`, you might
lay them out like so:

```
  third_party/
    sample_cipd_dep/
      ...
      lib/
        bar.jar
        foo.jar
```

### 3. Create a cipd.yaml file

CIPD knows how to create your package based on a .yaml file you provide to it.
This file can either be generated by a GN template, or manually.

#### 3a. Generating cipd.yaml via GN Template:

The `cipd_package_definition` template in [build/cipd/cipd.gni] can be used to
create the yaml definition as part of Chromium's normal build process. Declare
a target like:
```
cipd_package_definition("my_cipd_package") {
  package = "path/to/cipd/package"
  description = "Prebuilt test binary."
  install_mode = "copy"
  deps = [ "//path/to:test_binary_target" ]
  sources = [ "//path/to:test_binary_file" ]
}
```

After generating build files, you can then run
`ninja -C out/Default my_cipd_package`, which creates the .yaml file under
`out/Default/my_cipd_package/package.yaml`.

[build/cipd/cipd.gni]: https://source.chromium.org/chromium/chromium/src/+/master:build/cipd/cipd.gni

#### 3b. Generating cipd.yaml by hand:

You can also write the .yaml file by hand. The file should take the following
form:

```
# Comments are allowed.

# The package name is required. Third-party chromium dependencies should
# unsurprisingly all be prefixed with chromium/third_party/.
package: chromium/third_party/sample_cipd_dep

# The description is optional and is solely for the reader's benefit. It
# isn't used in creating the CIPD package.
description: A sample CIPD dependency.

# The root is optional and, if unspecified, defaults to ".". It specifies the
# root directory of the files and directories specified below in "data".
#
# You won't typically need to specify this explicitly.
root: "."

# The install mode is optional. If provided, it specifies how CIPD should
# install a package: "copy", which will copy the contents of the package
# to the installation directory; and "symlink", which will create symlinks
# to the contents of the package in the CIPD root inside the installation
# directory.
#
# You won't typically need to specify this explicitly.
install_mode: "symlink"

# The data is required and described what should be included in the CIPD
# package.
data:
  # Data can include directories, files, or a version file.

  - dir: "directory_name"

    # Directories can include an optional "exclude" list of regexes.
    # Files or directories within the given directory that match any of
    # the provided regexes will not be included in the CIPD package.
    exclude:
      - .*\.pyc
      - exclude_me
      - keep_this/but_not_this

  - file: keep_this_file.bin

  # If included, CIPD will dump package version information to this path
  # at package installation.
  - version_file: CIPD_VERSION.json
```

For example, for `sample_cipd_dep`, we might write the following .yaml file:

```
package: chromium/third_party/sample_cipd_dep
description: A sample CIPD dependency.
data:
  - file: bar.jar
  - file: foo.jar
```

For more information about the package definition spec, see [the code].

> **Note:** To make the package private (Googler-only), prefix the package name
> with `chrome_internal`. For example:
> ```
> package: chrome_internal/third_party/sample_cipd_dep
> ```

[the code]: https://chromium.googlesource.com/infra/luci/luci-go/+/master/cipd/client/cipd/builder/pkgdef.go

### 4. Create your CIPD package

To actually create your package, you'll need:

 - the `cipd.yaml` file (described above)
 - [permission](#permissions-in-cipd).

Once you have those, you can create your package like so:

```
# Assuming that the third-party dependency in question is at version 1.2.3
# and this is the first chromium revision of that version.
$ cipd auth-login  # One-time auth.
$ cipd create --pkg-def cipd.yaml
...
[P114210 10:14:17.215 client.go:931 I] cipd: instance
chromium/third_party/sample_cipd_dep:TX7HeY1_1JLwFVx-xiETOpT8YK4W5CbyO26SpmaMA0IC was
successfully registered
```

Take note of the instance ID printed in the log
(`TX7HeY1_1JLwFVx-xiETOpT8YK4W5CbyO26SpmaMA0IC` in the example above).
You'll be adding it to DEPS momentarily.

### 5. Add your CIPD package to DEPS

You can add your package to `DEPS` by adding an entry of the following form to
the `deps` dict:

```
deps = {
  # ...

  # This is the installation directory.
  'src/third_party/sample_cipd_dep': {

    # In this example, we're only installing one package in this location,
    # but installing multiple package in a location is supported.
    'packages': [
      {
        'package': 'chromium/third_party/sample_cipd_dep',
        'version': 'TX7HeY1_1JLwFVx-xiETOpT8YK4W5CbyO26SpmaMA0IC',
      },
    ],

    # As with git-based DEPS entries, 'condition' is optional.
    'condition': 'checkout_android',
    'dep_type': 'cipd',
  },

  # ...
}
```

This will result in CIPD package `chromium/third_party/sample_cipd_dep` at
`TX7HeY1_1JLwFVx-xiETOpT8YK4W5CbyO26SpmaMA0IC` being installed in
`src/third_party/sample_cipd_dep` (relative to the gclient root directory).

## Updating a CIPD dependency

To modify a CIPD dependency, follow steps 2, 3, and 4 above, then modify the
version listed in DEPS.

## Miscellaneous

### Permissions in CIPD

You can check a package's ACLs with `cipd acl-list`:

```
$ cipd acl-list chromium/third_party/sample_cipd_dep
...
```

Permissions in CIPD are handled hierarchically. You can check entries higher
in the package hierarcy with `cipd acl-list`, too:

```
$ cipd acl-list chromium
...
```

By default, [cria/project-chromium-cipd-owners][cria] own all CIPD packages
under `chromium/`. If you're adding a package, talk to one of them.

To obtain write access to a new package, ask an owner to run:

```
$ cipd acl-edit chromium/third_party/sample_cipd_dep -owner user:email@address.com
```

[cria]: https://chrome-infra-auth.appspot.com/auth/groups/project-chromium-cipd-owners

## Troubleshooting

 - **A file maintained by CIPD is missing, and gclient sync doesn't recreate it.**

CIPD currently caches installation state. Modifying packages managed by CIPD
will invalidate this cache in a way that CIPD doesn't detect - i.e., CIPD will
assume that anything it installed is still installed, even if you deleted it.
To clear the cache and force a full reinstallation, delete your
`$GCLIENT_ROOT/.cipd` directory.

Note that there is a [bug](https://crbug.com/794764) on file to add a mode to CIPD
that is not so trusting of its own cache.