Import changes for goma client

  - 21f14a33dde4673e2f62bbdb8bbef9712d8b1ba2 [rustc] Add rustc libs to compiler_info resources.
  - 37d16ad77d8bd0c2d1fe3c8a4e951150d178350d base: use C++11 delete functions instead of DISABLE_COPY_...
  - 3ed23bc150903060a915c68df33bc0ec2e2b8ce2 introduce more logs for busyloop debug
  - 6c015e0c3e6d2143494628a4397ae11d95f9183e lib: use C++11 delete functions instead of DISABLE_COPY_A...
  - 47a01f5c04d02d8bb7c8780ad93801c44c5b3d11 Roll client/third_party/boringssl/src 78c88c999..c37e64cba
  - 45c9fb951f3c8cea4c6411f432548532242be7ac Support not only x86_64-cros but also other clang.
  - 6694153f78fbd02b8a359748f288fa659012ec39 Hack to create directory for EXEC_PREFIX detection.
  - e71870bf67c0e9bfb42c7243b59e4cc71410deb9 update broken bots.proto link
  - 2cb1819cf28fce6879137730acb11d1ed689eeec update protobuf to 3.8.0
  - 92829246bcb02885ce0e38d136e5424f7e754147 Also add python2 as ChromeOS clang's dependency.
  - d607dc24455ffcc2d9aedd7b45736c5903a445d0 [rustc] Use rustc's sysroot/bin/rustc as its real path in...
  - 544379c9deb20b675b75562f363fa9615bfd01ce Roll client/third_party/boringssl/src 79ab5e8fa..78c88c999

GitOrigin-RevId: 21f14a33dde4673e2f62bbdb8bbef9712d8b1ba2
Change-Id: Ie88b870ff08a9d919d59b0ee4e55a9e28d009559
33 files changed
tree: ec442722c60a81c4ccebc95bd99e49b53e56860b
  1. base/
  2. benchmark/
  3. build/
  4. client/
  5. doc/
  6. infra/
  7. lib/
  8. test/
  9. testing/
  10. third_party/
  11. tools/
  12. .clang-format
  13. .gitignore
  14. .gn
  15. BUILD.gn
  16. codereview.settings
  17. CONTRIBUTING.md
  18. DEPS
  19. LICENSE
  20. OWNERS
  21. PRESUBMIT.py
  22. README.md
README.md

Goma client

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the Goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How Goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the Goma backend caches the compile result. If the same compile request comes, the cached result is returned from the Goma cache server.

How to build

Goma client can be built on Linux, Mac, and Win.

Prerequisite

  1. Use 64bit platform (Linux, Mac or Win).
  2. Install depot_tools.
  3. Install dependencies.
    • On Mac, install Xcode.
    • On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume the Goma client code is checked out to ${GOMA_SRC}. You can set this in your environment, but do not export it as it will make gomacc complain.

If you want to develop goma client, make goma client source unmanaged by gclient. Open .gclient file, and check "managed" value. If it's True, changed it to False. Otherwise, gclient will manage your repository, so your checkout can be unintentionally changed with gclient sync.

Move to client directory (which is under git repo), and configure git repository with your username and emails.

$ cd client
$ git config user.email 'foo@example.com'
$ git config user.name 'Your Name'

Build

$ cd "${GOMA_SRC}/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "${GOMA_SRC}/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

Coding style

Follow Google code style.

For C++11 features, we prefer to follow chromium guideline: http://chromium-cpp.appspot.com/

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "${GOMA_SRC}/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "${GOMA_SRC}/client/out/Release"  (Replace ${GOMA_SRC} to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

More details are avairable in chromium's build instructions.

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR="${GOMA_SRC}/client/out/Release" USE_GOMA=true make -j4

Here, -j4 is not related to Goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for Goma.

For general development

  1. Build Goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.
  • You can use autoninja in depot_tools instead of specifying gomacc manually.