Updating clang

We distribute prebuilt packages of LLVM binaries, including clang and lld, that all developers and bots pull at gclient runhooks time. These binaries are just regular LLVM binaries built at a fixed upstream revision. This document describes how to build a package at a newer revision and update Chromium to it. An archive of all packages built so far is at https://is.gd/chromeclang

Check that https://ci.chromium.org/p/chromium/g/chromium.clang/console looks reasonably green.
Sync your Chromium tree to the latest revision to pick up any plugin changes
Run python tools/clang/scripts/upload_revision.py NNNN with the target LLVM SVN revision number. This creates a roll CL on a new branch, uploads it and starts tryjobs that build the compiler binaries into a staging bucket on Google Cloud Storage (GCS).

If the clang upload try bots succeed, copy the binaries from the staging bucket to the production one. For example:

$ export rev=123456-1
$ for x in Linux_x64 Mac Win ; do \
    gsutil.py cp -n -a public-read gs://chromium-browser-clang-staging/$x/clang-$rev.tgz \
        gs://chromium-browser-clang/$x/clang-$rev.tgz ; \
    gsutil.py cp -n -a public-read gs://chromium-browser-clang-staging/$x/llvmobjdump-$rev.tgz \
        gs://chromium-browser-clang/$x/llvmobjdump-$rev.tgz ; \
    gsutil.py cp -n -a public-read gs://chromium-browser-clang-staging/$x/llvmcfiverify-$rev.tgz \
        gs://chromium-browser-clang/$x/llvmcfiverify-$rev.tgz ; \
    gsutil.py cp -n -a public-read gs://chromium-browser-clang-staging/$x/translation_unit-$rev.tgz \
        gs://chromium-browser-clang/$x/translation_unit-$rev.tgz ; \
    gsutil.py cp -n -a public-read gs://chromium-browser-clang-staging/$x/llvm-code-coverage-$rev.tgz \
        gs://chromium-browser-clang/$x/llvm-code-coverage-$rev.tgz ; \
    done
$ gsutil.py cp -n -a public-read gs://chromium-browser-clang-staging/Mac/lld-$rev.tgz \
      gs://chromium-browser-clang/Mac/lld-$rev.tgz

Run the goma package update script to push these packages to goma. If you do not have the necessary credentials to do the upload, ask clang@chromium.org to find someone who does

Run an exhaustive set of try jobs to test the new compiler:

git cl try &&
git cl try -B luci.chromium.try -b ios-device -b mac_chromium_asan_rel_ng \
  -b linux_chromium_cfi_rel_ng \
  -b linux_chromium_chromeos_asan_rel_ng -b linux_chromium_msan_rel_ng \
  -b linux_chromium_chromeos_msan_rel_ng -b linux-chromeos-dbg \
  -b win-asan -b chromeos-amd64-generic-cfi-thin-lto-rel

Optional: Start Pinpoint perf tryjobs. These are generally too noisy to catch minor regressions pre-commit, but make sure there are no large regressions.

a. (Log in to store OAuth2 token in the depot_tools cache. Only needs to be run once:)

$ PYTHONPATH=$(dirname $(which git-cl)) python -c"import auth;auth.OAUTH_CLIENT_ID='62121018386-h08uiaftreu4dr3c4alh3l7mogskvb7i.apps.googleusercontent.com';auth.OAUTH_CLIENT_SECRET='vc1fZfV1cZC6mgDSHV-KSPOz';print auth.get_authenticator_for_host('pinpoint',auth.make_auth_config()).login()"

b. Generate a fresh Oauth2 token:

$ TOKEN=$(PYTHONPATH=$(dirname $(which git-cl)) python -c"import auth;print auth.get_authenticator_for_host('pinpoint',auth.make_auth_config()).get_access_token().token")

c. Launch Pinpoint job:

$ curl -H"Authorization: Bearer $TOKEN" -F configuration=chromium-rel-win7-gpu-nvidia \
    -F target=performance_test_suite -F benchmark=speedometer2 \
    -F patch=https://chromium-review.googlesource.com/c/chromium/src/+/$(git cl issue | cut -d' ' -f3) \
    -F start_git_hash=HEAD -F end_git_hash=HEAD https://pinpoint-dot-chromeperf.appspot.com/api/new

d. Use the URL returned by the command above to see the progress and result of the tryjob, checking that it doesn't regress significantly (> 10%). Post the URL to the codereview.

Commit roll CL from the first step
The bots will now pull the prebuilt binary, and goma will have a matching binary, too.

Adding files to the clang package

The clang package is downloaded unconditionally by all bots and devs. It's called “clang” for historical reasons, but nowadays also contains other mission-critical toolchain pieces besides clang.

We try to limit the contents of the clang package. They should meet these criteria:

things that are used by most developers use most of the time (e.g. a compiler, a linker, sanitizer runtimes)
things needed for doing official builds

If you want to add something to the clang package that doesn‘t (yet?) meet these criteria, you can make package.py upload it to a separate zip file and then download it on an opt-in basis by requiring users to run a script to download the additional zip file. You can structure your script in a way that it downloads your additional zip automatically if the script detects an old version on disk, that way users have to run the download script just once. tools/clang/scripts/download_lld_mac.py is an example for this (It doesn’t do the “only download if old version is on disk or if requested” bit, and hence doesn't run as a default DEPS hook. TODO(thakis): Make coverage stuff a better example and link to that.)

If you‘re adding a new feature that you expect will meet the inclusion criteria eventually but doesn’t yet, start by having your things in a separate zip and move it to the main zip once the criteria are met.