commit | ffea473b1ceb98f1129d8fd4796ddc67e5a5fc69 | [log] [tgz] |
---|---|---|
author | Lukasz Anforowicz <lukasza@chromium.org> | Wed Dec 13 01:10:07 2023 |
committer | Copybara-Service <copybara-worker@google.com> | Wed Dec 13 01:17:59 2023 |
tree | 7169966a20729ce7a83caba3609a5036badc8297 | |
parent | 4fd87f9256687b539f4a03e5773427dd0b8d22b1 [diff] |
Stop building Crubit when packaging the Rust toolchain. Crubit integration was prototyped in 2022, but it is currently on hold. Before this CL, `package_rust.py` would call `BuildCrubit`, but failures in this step would have no impact on the overall status of the bots. After this CL, we will no longer `BuildCrubit` while packaging the Rust toolchain. Fixed: 1510943 Fixed: 1459587 Change-Id: I74930a7beb14d0fb0910c8332f86abacfe8970b5 Cq-Include-Trybots: luci.chromium.try:linux_upload_clang Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5117219 Reviewed-by: danakj <danakj@chromium.org> Auto-Submit: Łukasz Anforowicz <lukasza@chromium.org> Commit-Queue: Łukasz Anforowicz <lukasza@chromium.org> Cr-Commit-Position: refs/heads/main@{#1236710} NOKEYCHECK=True GitOrigin-RevId: 9d8a38d36369304723a9dcec9c80d17ffd87a0a7
This directory contains scripts for building, packaging, and distributing the Rust toolchain (the Rust compiler, and also C++/Rust FFI tools like Crubit).
Like with Clang, Chromium uses bleeding edge Rust tooling. We track the upstream projects' latest development as closely as possible. However, Chromium cannot use official Rust builds for various reasons which require us to match the Rust LLVM backend version with the Clang we use.
It would not be reasonable to build the tooling for every Chromium build, so we build it centrally (with the scripts here) and distribute it for all to use (also fetched with the scripts here).
Each Rust package is built from an Rust git, usually from HEAD directly, along with the current Clang/LLVM revision in use in Chromium. Hence a new Rust package must be built whenever either Rust or Clang is updated. When building Rust we also build additional tools such as clippy and rustfmt, and interop tools including bindgen and crubit.
The Rust build also includes building LLVM for rustc to use, and Clang for bindgen and crubit to use.
The *_upload_clang
and *_upload_rust
trybots are used to build Clang and Rust respectively from the revisions specified in the Chromium source tree. These are uploaded to a storage bucket when the build succeeds. After being copied from staging to production by a developer (see cs/copy_staging_to_prod_and_goma.sh), they can then be fetched by gclient sync
.
The update_rust.py
script is used by gclient sync
to fetch the Rust toolchain for the revisions specified in the script.
Follow the directions in //docs/updating_clang.md to roll Clang and Rust together. To just roll Rust on its own, use the --skip-clang
argument when running upload_revision.py
.
The upload_revision.py script will update the revision of Rust to be built and used in update_rust.py
and will start the trybots that will build the Rust toolchain.
After the build has succeeded and the new toolchain has been copied to production, the CQ will run trybots to verify that our code still builds and tests pass with the new Rust toolchain.
build_rust.py
will vendor all dependencies before starting the build. To do this it first initializes git submodules. Then it runs cargo vendor
. However some parts of the compiler build are excluded from the top level Cargo.toml workspace. Thus it passes --sync dir
for a number of subdirectories, based on dist.rs, the nightly tarball packager.
If another Cargo.toml is required in the future, and not part of the workspace it would produce missing dependencies, and the set of directories in build_rust.py
would need to be updated.
A build error when building the stdlib in Chromium may look like:
FAILED: local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib ...build command... ERROR: file not in GN sources: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/std/src/../../portable-simd/crates/std_float/src/lib.rs
Or:
FAILED: local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib ...build command... ERROR: file not in GN inputs: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/std/src/../../stdarch/crates/core_arch/src/core_arch_docs.md
When building the stdlib in Chromium, the GN rules must have every rust source or other input file that makes up the crate listed in the sources
and inputs
GN variables. gnrt will walk the directory tree from the root of the crate and put every relevant file into the set. But sometimes a crate includes modules from paths outside the crate root's directory tree, with a path directive such as
#[path = "../../stuff.rs"] mod stuff;
or will include!()
a file from another path, which is common for .md
files:
include!("../../other_place.md")
The first error is saying the source file std_float/src/lib.rs
did not appear in the sources
variable. The ../../
part of the path shows that this is outside the crate root's directory tree. The second error is saying that core_arch/src/core_arch_docs.md
did not appear in the inputs
variable.
To fix the error:
../../portable-simd/crates/std_float/src
. We could also use ../../portable-simd
or anything in between, though that would add a lot more sources to the GN rules than is necessary in this case. It's best to point to the directory of the module root (where the lib.rs
or mod.rs
is located).//build/rust/std/gnrt_config.toml
. The failing crate in the above example is libstd.rlib
, so we want the [crate.std]
section of the config file.sources
or inputs
.sources
, add the path to a extra_src_roots
list in the crate's rules. For the above example, we could add extra_src_roots = ['../../portable-simd/crates/std_float/src']
.inputs
, add the path to a extra_input_roots
list in the crate's rules. For the above example, we could add extra_input_roots = ['../../stdarch/crates/core_arch/src']
.tools/rust/gnrt_stdlib.py
to use gnrt to rebuild the stdlib GN rules using the updated config.BUILD.gn
files for stdlib cratesIf the build structure changes in any way during a roll, the GN files need to be regenerated.
Run tools/rust/gnrt_stdlib.py
.
This requires Rust to be installed and available in your system, typically through https://rustup.rs.
To generate BUILD.gn
files for the crates with the gnrt
tool:
src/
dir of Chromium.gnrt
to run on host machine: cargo build --release --manifest-path tools/crates/gnrt/Cargo.toml --target-dir out/gnrt
.third_party/rust-src
which can be done with tools/rust/build_rust.py --sync-for-gnrt
.gnrt
with the gen
action: out/gnrt/release/gnrt gen --for-std third_party/rust-src
.This will generate the //build/rust/std/rules/BUILD.gn
file, with the changes visible in git status
and can be added with git add
.
To build the Rust toolchain locally, run //tools/rust/build_rust.py
. It has additional flags to skip steps if you're making local changes and want to retry a build. The script will produce its outputs in //third_party/rust-toolchain/
, which is the same place that gclient sync
places them.
Building the rust_build_tests
GN target is a good way to quickly verify the toolchain is working.
Steps to roll the Crubit tools (e.g. rs_bindings_from_cc
tool) to a new version:
Locally, update CRUBIT_REVISION
in update_rust.py
. (Update CRUBIT_SUB_REVISION
when the build or packaging is changed, but the upstream Rust revision we build from is not changed.)
Locally, update crubit_revision
in //DEPS
, so that it matches the revision from the previous bullet item.
Run manual tests locally (see the “Building and testing the tools locally” section below). TODO(https://crbug.com/1329611): These manual steps should be made obsolete once Rust-specific tryjobs cover Crubit tests.
build_crubit.py
depends on Bazel.
To get Bazel, ensure that you have checkout_bazel
set in your .gclient
file and then rerun gclient sync
:
$ cat ../.gclient solutions = [ { "name": "src", "url": "https://chromium.googlesource.com/chromium/src.git", ... "custom_vars": { "checkout_bazel": True, "checkout_crubit": True, }, }, ]
Just run tools/rust/build_crubit.py
. So far build_crubit.py
has only been tested on Linux hosts.
build_crubit.py
will copy files into the directory specified in the (optional) --install-to
cmdline parameter - for example:
$ tools/rust/build_crubit.py --install-to=third_party/rust-toolchain/bin/
Crubit tests are under //build/rust/tests/test_rs_bindings_from_cc
. Until Crubit is built on the bots, the tests are commented out in //build/rust/tests/BUILD.gn
, but they should still be built and run before rolling Crubit. TODO(https://crbug.com/1329611): Rephrase this paragraph after Crubit is built and tested on the bots.