blob: bbd22f49dbcc7e3a4f819af50a2a27e33c320f1a [file] [log] [blame] [view]
# Updating Rust crates used by Chromium
This document describes how Chromium updates crates.io Rust crates that Chromium
depends on.
## Staffing
We have a
[weekly rotation](https://goto.google.com/chromium-crates-update-rotation) of
Google engineers responsible for creating and landing CLs that update Rust
crates.
Google engineers can join the rotation by emailing
[chrome-safe-coding@google.com](mailto:chrome-safe-coding@google.com).
## Initial setup
The "Rust: periodic update of 3rd-party crates" rotation requires access to an
up-to-date Chromium repo. One way to start a shift is to run `git fetch`,
`git checkout origin/main`, and `gclient sync` (but other workflows should also
work - e.g. ones based on `git-new-workdir`).
## Checking the state of the world
Before creating a CL stack, check for open CLs with the [`cratesio-autoupdate`
tag](https://chromium-review.googlesource.com/q/hashtag:%22cratesio-autoupdate%22+(status:open%20OR%20status:merged)).
Such CLs tend to conflict, so coordinate with owners of any open CLs.
You may also check a doc with notes from previous rotations, where we may note
known issues and their workarounds. See (Google-internal, sorry):
https://docs.google.com/document/d/1S7gsrJFsgoU5CH0K7-X_gL55zIIgd6UsFpCGrJqjdAg/edit?usp=sharing
## Automated step: `create_update_cl.py`
The first actual step of the rotation is running `create_update_cl.py`. You must
invoke it from within the `src/` directory of a Chromium repository checkout,
and it depends on `depot_tools` and `git` being present in the `PATH`.
```sh
$ cd ~/chromium/src # or wherever you have your checkout
$ tools/crates/create_update_cl.py auto
```
In `auto` mode, it runs `gnrt update` to discover crate updates and then for
each update creates a new local git branch (and a Gerrit CL unless invoked with
`--no-upload`). Each branch contains an update created by `gnrt update <old
crate id>`, `gnrt vendor`, and `gnrt gen`. Depending on how many crates are
updated, the script may need 10-15 minutes to run.
The script should Just Work in most cases, but sometimes it may fail when
dealing with a specific crate update. See [Recovering from script
failures](#recovering-from-script-failures) below for what to do when that
happens.
Before the auto-generated CLs can be landed, you will need to get an LGTM from
`//third_party/rust/OWNERS`. A review checklist can be found at
`//third_party/rust/OWNERS-review-checklist.md`. If you add
chrome-third-party-rust-reviews@google.com to the "Reviewers" line, an
OWNER will be automatically assigned.
## New transitive dependencies
Notes from `//third_party/rust/OWNERS-review-checklist.md` apply:
* The dependency will need to go through security review.
* An FYI email should be sent to
[chrome-atls-discuss@google.com](mailto:chrome-atls-discuss@google.com)
in order to record the addition.
### Optional: Adding the transitive dependency in its own CL
If the new crate is non-trivial, it's possible to split the
additional crate into its own CL, however then it will default to global
visibility and allowing non-test use.
* `gnrt add` and `gnrt vendor` can add the dependency to a fresh checkout.
* Mark the crate as being for third-party code only by setting
`allow_first_party_usage` to `false` for the crate in
`third_party/rust/chromium_crates_io/gnrt_config.toml`.
* If the crates making use of the transitive dependency are only allowed
in tests, then set `group = 'test'` for the crate in
`third_party/rust/chromium_crates_io/gnrt_config.toml`. This reduces
the level of security review required for the library.
* `gnrt gen` will then generate the GN rules.
* Rebase the roll CL on top of the changes to make sure the choices made above
are correct. `gn gen` will fail in CQ if the crate was placed in the `'test'`
group but needs to be visible outside of tests.
## Potential additional steps
* The `create_update_cl.py` script may stop early if it detects that `gnrt
vendor` or `gnrt gen` have reported any warnings or errors (e.g. a "License
file not found for crate foo" warning). In this case, manual intervention is
needed to finish the update CL. It's probably best to finish and land the CLs
created so far before trying to restart the script in order to create the
remaining CLs.
## Landing the CL
Other than the above, the CL can go through the normal, plain-vanilla, manual
review and landing process.
1. `git cl upload`
1. Get a review from one of `//third_party/rust/OWNERS`
1. Land the CL using CQ+2
## Checking for new major versions
Note that `create_update_cl.py auto` will by default only handle minor version
updates (e.g. 123.1 => 123.2, or 0.123.1 => 0.123.2). Major version changes
(e.g. 1.0 => 2.0, which may include breaking API changes and other breaking
changes) need to be handled separately - this section describes what to do.
### Detecting available major version updates (and doing the updates)
As part of the rotation, please do the following:
1. Check for new major versions of
_direct_ Chromium dependencies (i.e. dependencies directly listed in
`third_party/rust/chromium_crates_io/Cargo.toml`). To discover direct _and_
transitive dependencies with a new major version, you can use the command below
(running it in the final update CL branch - after all the minor version
updates):
```sh
$ tools/crates/run_gnrt.py update -- --verbose --dry-run
...
Unchanged serde_json_lenient v0.1.8 (latest: v0.2.0)
Unchanged syn v1.0.109 (latest: v2.0.53)
...
```
2. For each detected available major version update, use the script to
put together a CL that updates the crate (see the "Major version update:
Workflow A: Single update CL" section below) and then kick of CQ dry run.
Example invocation:
```
$ tools/crates/create_update_cl.py auto -- font-types read-fonts skrifa --breaking
Checking out the `origin/main` branch...
...
Creating a major version update CL...
Running `gnrt update -- font-types read-fonts skrifa --breaking` ...
...
Issue number: 6990318 (https://chromium-review.googlesource.com/6990318)
```
3. Depending on whether the CQ dry run succeeds
(or it's relatively easy to fix errors and make it succeed):
* If CQ succeeds then get a review from the crate owner (look
for `//third_party/rust/some_crate_name/OWNERS`) and land the CL.
Getting a review from the crate owner is important, because sometimes a
major version bump indicates a breaking behavior change (and CQ may pass
if there are no breaking API changes and Chromium test coverage doesn't
exercise the breaking behavior change).
* Otherwise, if fixing CQ dry run is not straightforward, then please
open a bug and assign it to the crate owner (asking them to drive the
update). For searchability use a bug title like:
"Rust crate major version update: `some_crate_name`: 123.x => 124.x"
Other notes to help with this part of the rotation:
* Major version updates of sets of interdependent crates may need to be
atomically (i.e. in a single CL). For example the 3 Fontations crates
(`font-types`, `read-fonts`, `skrifa`) need to be updated together.
### Major version update: Workflow A: Single update CL
If updating to a new major version doesn't require lots of Chromium changes,
then it may be possible to land the update in a single CL. This is typically
possible when the APIs affected by the major version's breaking change either
weren't used by Chromium, or were used only in a handful of places.
**Warning**: Sometimes a new major version may be API compatible, but may
introduce breaking changes in the _behavior_ of the existing APIs.
To update:
1. `tools/crates/create_update_cl.py auto -- some_crate_name --breaking`
1. Follow the manual steps from the minor version update rotation for
review, landing, etc.
### Major version update: Workflow B: Incremental transition
When lots of first-party code depends on the old major version, then the
transition to the new major version may need to be done incrementally. In this
case the transition can be split into the following steps:
1. Open a new bug to track the transition
- TODO: Figure out how to tag/format the bug to make it easy to discover
in future rotations
1. Land the new major version, so that the old and the new versions coexist.
To do this follow the process for importing a new crate as described in
[`docs/rust.md`](../../docs/rust.md#importing-a-crate-from-crates_io)
(i.e. edit `Cargo.toml` to add the new version, run `gnrt vendor`, and so
forth).
1. Incrementally transition first-party code to the new major version
1. Remove the old major version. To do this follow a similar process as above
(i.e. edit `Cargo.toml` to remove the old version, run `gnrt vendor`, and so
forth). Any leftover files in `//third_party/rust/<crate>/<old epoch>`
should also be removed.
Note that the following `Cargo.toml` syntax allows two versions of a crate to
coexist:
```toml
[dependencies.serde_json_lenient_old_epoch]
package = "serde_json_lenient"
version = "0.1"
[dependencies.serde_json_lenient]
version = "0.2"
```
## Other ways to use `create_update_cl.py`
### `auto` mode
Extra arguments passed to `create_update_cl.py auto` end up being passed to
`cargo update`. For a complete list of available options, see
[Cargo documentation here](https://doc.rust-lang.org/cargo/commands/cargo-update.html#update-options)),
but the most common scenarios are covered in the sections below.
#### Updating all crates during the weekly rotation
`tools/crates/create_update_cl.py auto` with no extra arguments will attempt to
discover **minor** version updates for **all** crates that Chromium depends on
and for their transitive dependencies.
#### Updating the minor version of a single crate
`tools/crates/create_update_cl.py auto -- some_crate_name` can be used to
trigger a **minor** version update of a single crate.
#### Updating the major version of a single crate
`tools/crates/create_update_cl.py auto -- some_crate_name --breaking` can be
used to trigger a **major** version update of a single crate
### `manual` mode
For maximal control, the script can be used in `manual` mode:
1. Prepare `Cargo.toml` change:
1. `git checkout origin/main`
1. `git checkout -b manual-update-of-foo`
1. Edit `third_party/rust/chromium_crates_io/Cargo.toml` to change the crate
version of the crate (or crates) you want to update.
**Important**: Do not edit `Cargo.lock` (e.g. don't run `gnrt vendor`
etc.).
1. `git add third_party/rust/chromium_crates_io/Cargo.toml`
1. `git commit -m "Manual edit of Cargo.toml"`
1. `git cl upload -m "Manual edit of Cargo.toml" --bypass-hooks --skip-title --force`
1. Run the helper script as follows:
`tools/crates/create_update_cl.py manual
--title "Roll foo crate to new version X"`
- This will run `gnrt vendor` to discover and execute updates that were
requested by the manual edits of `Cargo.toml` in the previous steps.
- This will automatically add more details to the CL description
- To make the review easier, one of the patchsets covers just the path
changes. For example - see [the delta here](https://crrev.com/c/5445719/2..7).
<a id="recovering-from-script-failures"></a>
## Recovering from script failures
Sometimes the `create_update_cl.py` script will fail when dealing with
a specific crate update. The general workflow in this case is to
1) fix the issue in a separate CL, and 2) restart the tool from the middle
by using `--upstream-branch` that points to the last successful update branch
(or to the fix CL) rather than defaulting to `origin/main`.
Examples of a few specific situations that may lead to script failure:
* An update brought in a new crate, but `gnrt` didn't recognize new crate's
license kind or license file. In that case a prerequisite CL needs to be
landed first, teaching `gnrt` about the new license kinds/files
([in readme.rs](https://source.chromium.org/chromium/chromium/src/+/main:tools/crates/gnrt/lib/readme.rs;l=264-290;drc=c838bc6c6317d4c1ead1f7f0c615af353482f2b3)).
You can see [an example CL with such a fix](https://crrev.com/c/6219211).
* Patches from `//third_party/rust/chromium_crates_io/patches/` no longer
apply cleanly to the new version of a crate. In that case the crate update CL
needs to 1) first update the patches, and then 2) update the crate as usual.
This is not very well supported by the script... But something like this
should work:
- Checkout a new branch:
```sh
$ git checkout rust-crates-update--last-successful-update
$ git checkout -b fix-patches-for-foo
$ git branch --set-upstream-to=rust-crates-update--last-successful-update
```
- Fix the patches and upload as a temporary / throw-away CL
(this CL can't be landed on its own - it needs to be combined
with the actual update CL):
```sh
$ # Get the updated crate code
$ ./tools/crates/run_gnrt.py update foo
$ ./tools/crates/run_gnrt.py vendor --no-patches foo
$ git commit -a -m ...
$ # Manually apply patches, creating separate commits for patch file updates
$ git rebase -i # drop everything but the patch file commits
$ git cl upload
```
- Restart the script (the CL created by the script can't be landed
as-is / on its own - it needs to be combined with the fixed patches
in the step below) with `--upstream-branch` parameter:
```sh
$ tools/crates/create_update_cl.py auto \
--upstream-branch=fix-patches-for-foo \
-- name-of-failed-crate
```
- Combine the branches:
```sh
$ git map-branches -v # to orient yourself
$ git checkout rust-crates-update--new-successful-update
$ git branch --set-upstream-to=rust-crates-update--last-successful-update
$ git cl upload -m Rebasing... # --bypass-hooks as needed
```
* `//third_party/rust/chromium_crates_io/gnrt_config.toml` needs to be updated
to work with a new crate version. The same workflow should work as for fixing
`//third_party/rust/chromium_crates_io/patches/` (see the item above).