commit	b51dc1d6afb310a2799d2d8ae423143bf02116e3	[log] [tgz]
author	Alan Zhao <ayzhao@google.com>	Thu Mar 28 18:48:07 2024
committer	Copybara-Service <copybara-worker@google.com>	Thu Mar 28 18:56:24 2024
tree	6abf57cdae8d4f863d9bc4440fdd39bb8d5e0222
parent	63acaaf6c41149b18240efeb96d847714e35b7c0 [diff]

Roll clang+rust llvmorg-19-init-2941-ga0b3dbaf-22 : llvmorg-19-init-6501-g5b544b51-1 / 7168c13579a550f2c47f7eea22f5e226a436cd00-2 : b13a71a2e77f4625d1a2b8a5b9488414686ebca9-1

Additionally, regenerate the abseil def files.

https://chromium.googlesource.com/external/github.com/llvm/llvm-project/+log/a0b3dbaf..5b544b51

https://chromium.googlesource.com/external/github.com/rust-lang/rust/+log/7168c13579a5..b13a71a2e77f

Ran: ./tools/clang/scripts/upload_revision.py 5b544b511c7133fcb26a5c563b746a4baefb38d6

Bug: 327446425
Disable-Rts: True
Cq-Include-Trybots: chromium/try:chromeos-amd64-generic-cfi-thin-lto-rel
Cq-Include-Trybots: chromium/try:dawn-win10-x86-deps-rel
Cq-Include-Trybots: chromium/try:lacros-arm64-generic-rel
Cq-Include-Trybots: chromium/try:linux-chromeos-dbg
Cq-Include-Trybots: chromium/try:linux_chromium_cfi_rel_ng
Cq-Include-Trybots: chromium/try:linux_chromium_chromeos_msan_rel_ng
Cq-Include-Trybots: chromium/try:linux_chromium_msan_rel_ng
Cq-Include-Trybots: chromium/try:mac11-arm64-rel,mac_chromium_asan_rel_ng
Cq-Include-Trybots: chromium/try:win-asan,android-official
Cq-Include-Trybots: chromium/try:fuchsia-arm64-cast-receiver-rel
Cq-Include-Trybots: chromium/try:mac-official,linux-official
Cq-Include-Trybots: chromium/try:win-official,win32-official
Cq-Include-Trybots: chromium/try:linux-swangle-try-x64,win-swangle-try-x86
Cq-Include-Trybots: chromium/try:android-cronet-mainline-clang-arm64-dbg
Cq-Include-Trybots: chromium/try:android-cronet-mainline-clang-arm64-rel
Cq-Include-Trybots: chromium/try:android-cronet-mainline-clang-x86-dbg
Cq-Include-Trybots: chromium/try:android-cronet-mainline-clang-x86-rel
Cq-Include-Trybots: chromium/try:android-cronet-riscv64-dbg
Cq-Include-Trybots: chromium/try:android-cronet-riscv64-rel
Cq-Include-Trybots: chrome/try:linux-chromeos-chrome
Cq-Include-Trybots: chrome/try:win-chrome,win64-chrome,linux-chrome,mac-chrome
Cq-Include-Trybots: chrome/try:linux-pgo,mac-pgo,win32-pgo,win64-pgo
Cq-Include-Trybots: chromium/try:android-rust-arm32-rel
Cq-Include-Trybots: chromium/try:android-rust-arm64-dbg
Cq-Include-Trybots: chromium/try:android-rust-arm64-rel
Cq-Include-Trybots: chromium/try:linux-rust-x64-dbg
Cq-Include-Trybots: chromium/try:linux-rust-x64-rel
Cq-Include-Trybots: chromium/try:mac-rust-x64-dbg
Cq-Include-Trybots: chromium/try:win-rust-x64-dbg
Cq-Include-Trybots: chromium/try:win-rust-x64-rel
Change-Id: Ifbb7a18cd24be9c620b958c154b13a2ad1b85b48
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5399878
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1279833}
NOKEYCHECK=True
GitOrigin-RevId: 1e35a7e3cf2c1dcb8be17715db59ed8fbaa85ac2

update_rust.py[diff]

1 file changed

tree: 6abf57cdae8d4f863d9bc4440fdd39bb8d5e0222

README.md

//tools/rust

This directory contains scripts for building, packaging, and distributing the Rust toolchain (the Rust compiler, and also C++/Rust FFI tools like Crubit).

Background

Like with Clang, Chromium uses bleeding edge Rust tooling. We track the upstream projects' latest development as closely as possible. However, Chromium cannot use official Rust builds for various reasons which require us to match the Rust LLVM backend version with the Clang we use.

It would not be reasonable to build the tooling for every Chromium build, so we build it centrally (with the scripts here) and distribute it for all to use (also fetched with the scripts here).

Rust build overview

Each Rust package is built from an Rust git, usually from HEAD directly, along with the current Clang/LLVM revision in use in Chromium. Hence a new Rust package must be built whenever either Rust or Clang is updated. When building Rust we also build additional tools such as clippy and rustfmt, and interop tools including bindgen and crubit.

The Rust build also includes building LLVM for rustc to use, and Clang for bindgen and crubit to use.

The *_upload_clang and *_upload_rust trybots are used to build Clang and Rust respectively from the revisions specified in the Chromium source tree. These are uploaded to a storage bucket when the build succeeds. After being copied from staging to production by a developer (see cs/copy_staging_to_prod_and_goma.sh), they can then be fetched by gclient sync.

The update_rust.py script is used by gclient sync to fetch the Rust toolchain for the revisions specified in the script.

Rolling Rust

Follow the directions in //docs/updating_clang.md to roll Clang and Rust together. To just roll Rust on its own, use the --skip-clang argument when running upload_revision.py.

The upload_revision.py script will update the revision of Rust to be built and used in update_rust.py and will start the trybots that will build the Rust toolchain.

After the build has succeeded and the new toolchain has been copied to production, the CQ will run trybots to verify that our code still builds and tests pass with the new Rust toolchain.

Possible failure: Missing dependencies

build_rust.py will vendor all dependencies before starting the build. To do this it first initializes git submodules. Then it runs cargo vendor. However some parts of the compiler build are excluded from the top level Cargo.toml workspace. Thus it passes --sync dir for a number of subdirectories, based on dist.rs, the nightly tarball packager.

If another Cargo.toml is required in the future, and not part of the workspace it would produce missing dependencies, and the set of directories in build_rust.py would need to be updated.

Possible failure: Missing sources or inputs

A build error when building the stdlib in Chromium may look like:

FAILED: local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib
...build command...
ERROR: file not in GN sources: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/std/src/../../portable-simd/crates/std_float/src/lib.rs

Or:

FAILED: local_rustc_sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd.rlib
...build command...
ERROR: file not in GN inputs: ../../third_party/rust-toolchain/lib/rustlib/src/rust/library/std/src/../../stdarch/crates/core_arch/src/core_arch_docs.md

When building the stdlib in Chromium, the GN rules must have every rust source or other input file that makes up the crate listed in the sources and inputs GN variables. gnrt will walk the directory tree from the root of the crate and put every relevant file into the set. But sometimes a crate includes modules from paths outside the crate root's directory tree, with a path directive such as

#[path = "../../stuff.rs"]
mod stuff;

or will include!() a file from another path, which is common for .md files:

include!("../../other_place.md")

The first error is saying the source file std_float/src/lib.rs did not appear in the sources variable. The ../../ part of the path shows that this is outside the crate root's directory tree. The second error is saying that core_arch/src/core_arch_docs.md did not appear in the inputs variable.

To fix the error:

Determine the path that is missing, relative to the crate root. In the above example this is ../../portable-simd/crates/std_float/src. We could also use ../../portable-simd or anything in between, though that would add a lot more sources to the GN rules than is necessary in this case. It's best to point to the directory of the module root (where the lib.rs or mod.rs is located).
Download the roll CL (on Gerrit, click on the 3 dots in the upper right corner and click on “Download patch”).
Find the failing build target crate's rules in //build/rust/std/gnrt_config.toml. The failing crate in the above example is libstd.rlib, so we want the [crate.std] section of the config file.
Determine if the missing file should go in sources or inputs.
- For sources, add the path to a extra_src_roots list in the crate's rules. For the above example, we could add extra_src_roots = ['../../portable-simd/crates/std_float/src'].
- For inputs, add the path to a extra_input_roots list in the crate's rules. For the above example, we could add extra_input_roots = ['../../stdarch/crates/core_arch/src'].
With the roll CL checked out, run gclient sync.

NOTE: gclient sync will download the version of the rust toolchain from the roll CL. In order for this to work, the upload_rust bots should‘ve completed and `copy_staging_to_prod_and_goma.sh should’ve been run.

Run tools/rust/gnrt_stdlib.py to use gnrt to rebuild the stdlib GN rules using the updated config.

Generating `BUILD.gn` files for stdlib crates

If the build structure changes in any way during a roll, the GN files need to be regenerated.

Simple way:

Run tools/rust/gnrt_stdlib.py.

Longer way

This requires Rust to be installed and available in your system, typically through https://rustup.rs.

To generate BUILD.gn files for the crates with the gnrt tool:

Change directory to the root src/ dir of Chromium.
Build gnrt to run on host machine: cargo build --release --manifest-path tools/crates/gnrt/Cargo.toml --target-dir out/gnrt.
Ensure you have a checkout of the Rust source tree in third_party/rust-src which can be done with tools/rust/build_rust.py --sync-for-gnrt.
Run gnrt with the gen action: out/gnrt/release/gnrt gen --for-std third_party/rust-src.

This will generate the //build/rust/std/rules/BUILD.gn file, with the changes visible in git status and can be added with git add.

Local development

To build the Rust toolchain locally, run //tools/rust/build_rust.py. It has additional flags to skip steps if you're making local changes and want to retry a build. The script will produce its outputs in //third_party/rust-toolchain/, which is the same place that gclient sync places them.

Building the rust_build_tests GN target is a good way to quickly verify the toolchain is working.

Rolling Crubit tools

Steps to roll the Crubit tools (e.g. rs_bindings_from_cc tool) to a new version:

Locally, update CRUBIT_REVISION in update_rust.py. (Update CRUBIT_SUB_REVISION when the build or packaging is changed, but the upstream Rust revision we build from is not changed.)
Locally, update crubit_revision in //DEPS, so that it matches the revision from the previous bullet item.
Run manual tests locally (see the “Building and testing the tools locally” section below). TODO(https://crbug.com/1329611): These manual steps should be made obsolete once Rust-specific tryjobs cover Crubit tests.

Building and testing Crubit locally

Prerequisites

Bazel

build_crubit.py depends on Bazel.

To get Bazel, ensure that you have checkout_bazel set in your .gclient file and then rerun gclient sync:

$ cat ../.gclient
solutions = [
  {
    "name": "src",
    "url": "https://chromium.googlesource.com/chromium/src.git",
    ...
    "custom_vars": {
      "checkout_bazel": True,
      "checkout_crubit": True,
    },
  },
]

Building

Just run tools/rust/build_crubit.py. So far build_crubit.py has only been tested on Linux hosts.

Deploying

build_crubit.py will copy files into the directory specified in the (optional) --install-to cmdline parameter - for example:

$ tools/rust/build_crubit.py --install-to=third_party/rust-toolchain/bin/

Testing

Crubit tests are under //build/rust/tests/test_rs_bindings_from_cc. Until Crubit is built on the bots, the tests are commented out in //build/rust/tests/BUILD.gn, but they should still be built and run before rolling Crubit. TODO(https://crbug.com/1329611): Rephrase this paragraph after Crubit is built and tested on the bots.