Google Git
Sign in
chromium / infra / goma / client / af2d9997a6faedfa3599c0f1686fac3d4e488bb8
commitaf2d9997a6faedfa3599c0f1686fac3d4e488bb8[log] [tgz]
authorGoma Authors <goma-dev@google.com>Thu Mar 07 07:21:52 2019
committerSimon Que <sque@google.com>Tue Mar 12 21:07:49 2019
tree2bd205e60fef0dceed5bac3c82d5289c28e00abd
parent904032d18896ff745417e5941bf097c9a5a010ff [diff]
Import changes for goma client

  - 34ed5bb855f067e202b9536a3dba2e0867de92dc add stats for dropped content
  - 9971bfa8af8d6243003873d265c9bcafeb1c0ed5 change num_uploading_input_file when ShringExecReq
  - 80282d55ad42ef30ffa88123ba45bd4b872e9142 use info log
  - 5a59b9a43b09d39928a91b83d2b4c711858df6ff flag: deprecate STUBBY_PROXY
  - 221fea17dd80ea48c917bfc99cae4181ec9880fe Shrink size of ExecReq before sending
  - b51b0afd11c4dd67fad8b8e529cee9c1096ed172 Roll client/third_party/boringssl/src 98ad4d77e..ae1e08709
  - 67649bdc0090c08faf2a078f516ff1820764dddb Revert "client: Add func to create FileDataOutput from Ou...
  - 010946b6e8acad846e82919d0ab89bf0ea3b5a9b Fix on finding simple Chrome deps.
  - 96a6a0f7bb6cc4fec680a1a0fe7b80b58081a703 client: Add func to create FileDataOutput from OutputFile...
  - 3765767fcb2a588194d100c5ae47b7c771842b43 Introduce FileBlobClient and FileBlobDownloader
  - 5bb84798af45f64c823ad6abd760a1c8567d947e client: Move FileServiceBlobClient classes to client/blob/
  - 4891b27ce90289176def22ec9eb2a925964ee1f7 cleanup: Delete extra lines in compiler_proxy_http_handler
  - b5e52eb8d471b47241a732c7101bddfb7eb30640 Revert "lib: Delete unused functions in FileServiceClient"
  - 351eb89f1687aed578091ffc4cd0993b853c16a8 workaround: non existence of *.so should not cause local ...
  - 3050624060376add0d3c7d2e06e0dece8d381b97 lib: Move FileBlob helper funcs to goma_data_util
  - 346475da322408a22e32e1c6754126982da32015 lib: Delete unused functions in FileServiceClient
  - 270b3876b35a10d9a4e136bd683a83bfde0e68f1 client: Move OutputFileInfo to goma_blob
  - cf3a20d9f59a9c0abac81f2e4dc25d0bee631596 lib: Move OutputFile to its own module
  - 50439e681793587b5ca8f68c6fbca4d4b10bc649 cleanup: flip ATS flag in one place instead of scattering.
  - d832c5482074b08aea29721d28cae270ebc86fb1 openssl_engine: follow Chromium's default cipher config.
  - e13fde91029d1b20a831daccbd204ec64136865d openssl_engine: use SSL_set0_{r,w}bio instead.
  - 57247212d8ecf54d0aeb834e17b873e3bc46ab4f clarify C/C++ IncludeProcessor doc
  - 842de03a9c13da6e71edbbb11eeac868a46dc408 openssl_engine: use SSL_CTX_sess_set_new_cb instead.
  - fc3b67310c16e2f9a85641b3b8d33d15abd822c9 Reland "update win_toolchain"
  - b549007547788e5699bcb773a2ad2930d4d1667c openssl_engine: shows TLSv1.2 instead of TLSv1/SSLv3.
  - 12375b4122224e892d3351af37cc055f64113c25 Roll client/third_party/boringssl/src 104306f58..98ad4d77e
  - 9896dd98f5d2b0005684545bfd8edad5d88b914f cleanup: use mode_t instead of int
  - e1a1915dc5f75afbf9c589875a0c9e9ceb52871d extract copying msdia140 code from vs_toolchain.py
  - 4c2487daf5b0b7590c64925fdc35d9314925d09b show all command line flags if hovered
  - 92addd8d44a3ce1986750f58206c67748580904e fallback if implicit clang module maps is used.
  - db40aff6263f1806a6f933b8b91832224bfa8ed8 cleanup: move spawner.h to spawner_lib
  - a9f0b1880ad9ae14e53113d9200ee33c58076119 cleanup: run gn format

GitOrigin-RevId: 34ed5bb855f067e202b9536a3dba2e0867de92dc
Change-Id: Ida0ce95dc9a8f603156aba5b70fff8c0d10454b2
84 files changed
tree: 2bd205e60fef0dceed5bac3c82d5289c28e00abd
  1. base/
  2. benchmark/
  3. build/
  4. client/
  5. doc/
  6. infra/
  7. lib/
  8. test/
  9. testing/
  10. third_party/
  11. tools/
  12. .clang-format
  13. .gitignore
  14. .gn
  15. BUILD.gn
  16. codereview.settings
  17. CONTRIBUTING.md
  18. DEPS
  19. LICENSE
  20. OWNERS
  21. PRESUBMIT.py
  22. README.md
README.md

Goma client

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the Goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How Goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the Goma backend caches the compile result. If the same compile request comes, the cached result is returned from the Goma cache server.

How to build

Goma client can be built on Linux, Mac, and Win.

Prerequisite

  1. Use 64bit platform (Linux, Mac or Win).
  2. Install depot_tools.
  3. Install dependencies.
    • On Mac, install Xcode.
    • On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume the Goma client code is checked out to ${GOMA_SRC}.

If you want to develop goma client, make goma client source unmanaged by gclient. Open .gclient file, and check "managed" value. If it's True, changed it to False. Otherwise, gclient will manage your repository, so your checkout can be unintentionally changed with gclient sync.

Move to client directory (which is under git repo), and configure git repository with your username and emails.

$ cd client
$ git config user.email 'foo@example.com'
$ git config user.name 'Your Name'

Build

$ cd "${GOMA_SRC}/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "${GOMA_SRC}/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

Coding style

Follow Google code style.

For C++11 features, we prefer to follow chromium guideline: http://chromium-cpp.appspot.com/

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "${GOMA_SRC}/client/out/Release"  (Replace ${GOMA_SRC} to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

More details are avairable in chromium's build instructions.

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=${GOMA_SRC}/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to Goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for Goma.

For general development

  1. Build Goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.
  • You can use autoninja in depot_tools instead of specifying gomacc manually.