Google Git
Sign in
chromium / infra / goma / client / 70685d6cbb19c108d8abf2235edd2d02bed8dded
commit70685d6cbb19c108d8abf2235edd2d02bed8dded[log] [tgz]
authorGoma Authors <goma-dev@google.com>Fri Feb 22 03:23:01 2019
committerTakuto Ikuta <tikuta@chromium.org>Mon Feb 25 01:27:44 2019
tree241e25bc4be90591a504e550d1b65ca65c4d2b60
parent04504b62d2d073b9bf5038f2ae1f47794a4c967a [diff]
Import changes for goma client

  - 6ba7ba97b74f27fe8971a636776fd8e4866f17e6 bug fix: opposite condition was given for gcc flags.
  - 3e1e782d86d8ee82914a45f2e7f0143ab0542157 [cleanup] moved where to enable clang modules support.
  - 7fd5eba6cbe8e92b8ff20a22b7c49927df949d37 Move compiler specific ShouldFallback code to compiler's ...
  - 80a1ddc29616db0fccb66388644210290043c32a include processor shouldn't fail if stdc-predef.h is miss...
  - e397133b6e45a924711ad3f629fe92517c3ff982 goma archive should contain LICENSE file
  - 51eed495ba6c129cdcb98475aab4b8b56c14a1b9 Revert "Reland "update win_toolchain""
  - 505b2d538d1c20de1311497528410c06c6600f77 Fix to collect chromeos clang resources
  - 7a5ca494e41e917483d901b69239bdad151d3ab7 Reland "update win_toolchain"
  - f9e64e28019f21c122b6af53ce6f5d42b872da9f Make runhooks for windows toolchain run before clang update
  - b5bf17db9114b1c00d25656aee6d6339f0a670e5 Roll client/third_party/nasm/ 313e2e8b9..ae8e4ca1c (1 com...
  - 9cb38d6d7992d3c1e219e7b80f85f10c7789f380 Move link and precompiled header to CompileFlags
  - 651c014f627ef9e7c3f429522b93ee70c03f7c01 suppress tsan failure in abseil
  - ee26c5d47d574b3c9e0637d588349cd2cc5a0de1 Roll client/third_party/breakpad/breakpad/ 54fa71efb..a0e...
  - 144e5a641538126436d9c640a0dabf5824cf1891 Record failure of updating required files.
  - cdd826eb6b7a3a9f4a6dfb6c8e4f5d53b1c2da05 Revert "update win_toolchain"
  - f6200544dfc76dc4dc026c5e056bd4846eb7f136 client: Fix HTTP status lookup in status page script
  - ec143070d399dc54a9d13a93deca84df12df350b Roll client/third_party/zlib/ 6c1421668..1337da531 (116 c...
  - 30502b42b04c153440aa217e831a1384603717c0 update buildtools
  - 76a498920050869f71c0c34b608912ba3468417e Add document for include processor
  - 618fd3e3f31f1a0c020be505a4f138e829f6f8e3 Support clang in chromeos chroot for ATS
  - 03690944ee3620d9c0d118df34b8655ff8ef1572 update win_toolchain
  - a81da25f5f772ee75769b33c37c05f6b9f25505f update excluded folder for copybara
  - ce5d54e0fb11bbfba85c95eb7bb1b9230469fa43 Use Chromium's zlib repository instead.
  - afbb53cf251c16ef3445533fb26103eedca700fc fix typo in licenses.py
  - b9a143716e9e2946f4b75c411b32f37614d76f18 remove yasm directory
  - b02c2bfd358559876348c3dc5cf96a54d8fa7bb0 Use nasm instead of yasm to build BoringSSL assembler on ...
  - b3564817cf24472a419fbd4a09de4061beb3a7f9 client: Use constexpr defs for upload file size limits
  - 1b41705b9c8005890a01547aab25ba8c39265814 http: fix IsHealthyRecently

GitOrigin-RevId: 6ba7ba97b74f27fe8971a636776fd8e4866f17e6
Change-Id: I42b30ec3f86a3bf2053dd92b71597353a86db199
55 files changed
tree: 241e25bc4be90591a504e550d1b65ca65c4d2b60
  1. base/
  2. benchmark/
  3. build/
  4. client/
  5. doc/
  6. infra/
  7. lib/
  8. test/
  9. testing/
  10. third_party/
  11. tools/
  12. .clang-format
  13. .gitignore
  14. .gn
  15. BUILD.gn
  16. codereview.settings
  17. CONTRIBUTING.md
  18. DEPS
  19. LICENSE
  20. OWNERS
  21. PRESUBMIT.py
  22. README.md
README.md

Goma client

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the Goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How Goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the Goma backend caches the compile result. If the same compile request comes, the cached result is returned from the Goma cache server.

How to build

Goma client can be built on Linux, Mac, and Win.

Prerequisite

  1. Use 64bit platform (Linux, Mac or Win).
  2. Install depot_tools.
  3. Install dependencies.
    • On Mac, install Xcode.
    • On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume the Goma client code is checked out to ${GOMA_SRC}.

If you want to develop goma client, make goma client source unmanaged by gclient. Open .gclient file, and check "managed" value. If it's True, changed it to False. Otherwise, gclient will manage your repository, so your checkout can be unintentionally changed with gclient sync.

Move to client directory (which is under git repo), and configure git repository with your username and emails.

$ cd client
$ git config user.email 'foo@example.com'
$ git config user.name 'Your Name'

Build

$ cd "${GOMA_SRC}/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "${GOMA_SRC}/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

Coding style

Follow Google code style.

For C++11 features, we prefer to follow chromium guideline: http://chromium-cpp.appspot.com/

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "${GOMA_SRC}/client/out/Release"  (Replace ${GOMA_SRC} to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

More details are avairable in chromium's build instructions.

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=${GOMA_SRC}/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to Goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for Goma.

For general development

  1. Build Goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.
  • You can use autoninja in depot_tools instead of specifying gomacc manually.