Google Git
Sign in
chromium / infra / goma / client / 0bf6949d1e7e5cb984b900a47a084b5796817f3b
commit0bf6949d1e7e5cb984b900a47a084b5796817f3b[log] [tgz]
authorGoma Authors <goma-dev@google.com>Thu Jul 26 05:39:32 2018
committerYoshisato Yanagisawa <yyanagisawa@chromium.org>Thu Jul 26 06:45:22 2018
tree6ad8124813a25968b76d6a33c1c9f3f77d82fe25
parent6cd37a6e2f9249d3ffc54d6336605e9d6288a864 [diff]
Import changes for goma client

  - 7f9a65fd32a42bea7c273a51db676700002012ae Convert compiler info classes to use absl/time
  - 2870bddd0e25008749b4490c22cebde73da6d543 Convert CompilerProxy from time_t to absl/time
  - f8af25f3499207185cdc0d549910c6a570fba05f SocketPool: replace int durations with absl::Duration
  - 26c220a126675cf0fe7c2052938bd055eb610f52 Introduce CompilerTypeSpecific
  - 495a7cd20c89fbbd342396c3388d84d18556d6c1 add more logging for http://b/111816632
  - 6a29d6c5e7b0816c4384f5795249d117ffafc753 ioutil: remove StringStrip and StringRstrip
  - 79bc0dadbe637a1e1264cc43c557fd36382427ef http: fix Header() contains trailing CRLFCRLF
  - 31fe88583e5670c263708650de2cf0b53eb780c7 Split list_dir_cache_unittest into several test cases
  - eb8ef312ec2b991d718c3d27bae4adedb32f5371 Remove gettimeofday_helper_win
  - e30d8eb5190b0af3243bc0433be690101900b825 Remove timestamp.h, replace with absl/time
  - 94fef1f431641cdb977c87a37a7ed6de6cb10df8 goma_auth: use goma_fetch instead of curl
  - 71670a0e50eacffbb99f7a45e2d1c94e997c636d jarfile_reader_unittest: fix #if guard
  - 0b2ad245aae63c3043e6fe0a4ff5f5f93c366c59 use ASSERT instead of PCHECK for OpenSocketPairForTest
  - 1e0a435c387bf8ac0d09ea2b52c7eccaa68c6ad7 Disable subprocess killing again
  - e055fdc60a8c314983c7987ada3a63b6f8dbdae1 ioutil: use abseil's strip for StringRstrip and StringStrip
  - 0ab22ea666f3a35469b888d90a4169bd3196116e http: use ExtractHeaderField to check message header
  - 514dd420eb0339e6a41a387473a63bcf4f73d406 Make copybara up-to-date
  - 256e6386ae3929e7a891dbf8fda54bdb571bd581 Attach info on crash.
  - f41ee77e3456dc8d295662db271ea7e61acf05e7 Convert CompileService to use absl/time
  - 91cd0ef1bb9a7659d893e0b31a06a35ab4075426 make level of msan origin tracking configuarable.

GitOrigin-RevId: 7f9a65fd32a42bea7c273a51db676700002012ae
Change-Id: I3efe5158ceb3c80a140d61bc9b75364d55f88af9
77 files changed
tree: 6ad8124813a25968b76d6a33c1c9f3f77d82fe25
  1. base/
  2. benchmark/
  3. build/
  4. buildtools/
  5. client/
  6. infra/
  7. lib/
  8. test/
  9. testing/
  10. third_party/
  11. tools/
  12. .clang-format
  13. .gitignore
  14. .gn
  15. BUILD.gn
  16. codereview.settings
  17. CONTRIBUTING.md
  18. DEPS
  19. LICENSE
  20. OWNERS
  21. PRESUBMIT.py
  22. README.md
README.md

Goma

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the goma backend caches the compile result. If the same compile request comes, the cached result is returned from the goma cache server.

How to build

goma client can be built on Linux, Mac, and Win.

Install dependencies

  1. Install depot_tools.
  2. Install dependencies.

On debian or ubuntu,

$ sudo apt-get install libssl-dev libc6-dev-i386

On Mac, install Xcode.

On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume goma is checked out to $GOMA_SRC.

Build

$ cd "$GOMA_SRC/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "$GOMA_SRC/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "$GOMA_SRC/client/out/Release"  (Replace $GOMA_SRC to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=$GOMA_SRC/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for goma.

For general development

  1. Build goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.

Tips

  • You can use autoninja in depot_tools instead of specifying gomacc manually.