Google Git
Sign in
chromium / infra / goma / client / f5efece09286e7fab5fe52a77a6f3bd515718ae6
commitf5efece09286e7fab5fe52a77a6f3bd515718ae6[log] [tgz]
authorGoma Authors <goma-dev@google.com>Mon Mar 08 23:41:59 2021
committerYoshisato Yanagisawa <yyanagisawa@chromium.org>Thu Mar 11 04:53:10 2021
tree03bba10b0b56c5a0465ca52e5f4cda2e10289d8c
parentbe1b7f4547023e809b636dc9572c976ccb33dcac [diff]
Import changes for goma client

  - 318cf825c3f1d1f8c79699502c9dd1422635f47b [flags] Add support for '-fprofile-list=' flag
  - 68960af0a6dc9382a86375d458012fbc1016719f Roll client/third_party/boringssl/src 0a86cbff4..04b3213d4
  - ba8c81659d8475c709da44abdc09d79bb049adbd Roll clang from 00c3f975ba30 to 630ab8a35c3e (1 revision)
  - 36d95b2b7f20f2a54037aa0e43e895022f58ef04 Roll clang from 791d56c8f00d to 00c3f975ba30 (1 revision)
  - 9567330918aa18b75696aa150b929ff7b5db740c Roll clang from 1ab4fae32e01 to 791d56c8f00d (1 revision)
  - 1452c56304c971facfe75369dd1a85a34d335d1c Roll clang from 77fd7947dd83 to 1ab4fae32e01 (1 revision)
  - de67c57231910a43efd2746bb29e832cefc15c81 Roll clang from 6a9fd91d34f5 to 77fd7947dd83 (1 revision)
  - 7bf012eb7780badd8c4e2aba342bcd29b7f4c399 Make conflicting -f*-compilation-dir flags not relocatable.
  - c079e003b8dea5671ba7fad04a6c390673eb9124 Fix -ffile-compilation-dir handling in GCCFlags.
  - 4e958cfbec6803adf7f8119c41ca04ecac5b4c81 Allow normalization of cwd if -f{coverage,file}-compilati...
  - c5d853b45e228f92a52766ff276adea54c1d1e7a goma_ctl: fix print format
  - bd10516034d4a28698ad6234fb146055ec0fa28a support clang-cl for linux ATS
  - cb37b6ef00face1bf500b98e1b8d85706fdcb498 client: normalize PWD env var in addition to cwd
  - 7cdbdeec03aea9dc55d246db18aaa8c0c40bf30f Roll client/third_party/boringssl/src 1a7456ac1..0a86cbff4
  - 21f438b7763607c154124e8873e3d0d71dab06b3 lib: support -f{coverage,file}-compilation-dir flags.
  - 54a632f1681f13de2c9e7776d04a9fddefb255fd won't normalize of -fcoverage-mapping flag is set.
  - 7c3cc4981d6e2bd3605ae666510b8c7685d1073f Roll clang from 823927c5fe09 to 6a9fd91d34f5 (1 revision)
  - 04bece66c5f912117d92dea30c91797624c50f20 Roll clang from 79ca91dbe12a to 823927c5fe09 (1 revision)
  - 2513fe2288b657334953a12fded34704f55cb2dc Roll clang from 763a0667a065 to 79ca91dbe12a (1 revision)
  - d3eb5fcb49f47a3a1b3ab3b745766d3c13cfa077 Roll clang from d15cefb01f58 to 763a0667a065 (1 revision)
  - e66952d13c02b429fbc78950e4981e01d20c14db Roll clang from cfd0f628093b to d15cefb01f58 (1 revision)
  - efa0758961c7293abb6e0d02a5bd4450910f2506 fix raw string literal handling
  - f06199e6695f9064e26afa10021ef2d5956cfd7c Roll clang from bf8c8f36a153 to cfd0f628093b (1 revision)
  - 9a4442c57268f3e18c803d683f12b25791e7e39f Roll clang from 6cf138ce6f58 to bf8c8f36a153 (1 revision)
  - 78ebdb587533b3f7518b6c0a197b0767ca0f51af handle raw string literal
  - f15af0ee50555431ba953f32ca917e5459a3d40c Roll client/third_party/boringssl/src 238a25831..1a7456ac1
  - 722a4f64c77ae31579ba55001ae8fcd9e22d8d1f Update go to 1.16
  - 24d7a4963598c429f1f43a3d9ad606d3e79cda02 Roll clang from 92688e86c2b1 to 6cf138ce6f58 (1 revision)
  - 97fdeedcca22fa7594023105bbb29a421545ece5 Roll clang from 99902658e154 to 92688e86c2b1 (1 revision)
  - 15905577c2ba9d5c6e263323942293810880b475 fix log: space after task id
  - fbc67cd52b6a4cb09704f16321c9d47c2483a907 Roll clang from ce95c4ce3863 to 99902658e154 (1 revision)
  - 0d467cbfa863134d883b39818b6596af5a3e36e6 Roll clang from 6ec6688f3b16 to ce95c4ce3863 (1 revision)
  - e3251977b9dda1883a07cb8ba82f7564889015f1 Roll clang from 823dc87476ee to 6ec6688f3b16 (1 revision)
  - 031c5040105cbecce9ba7833209b6100d2eda522 Roll clang from 953a8eb380b6 to 823dc87476ee (1 revision)
  - 65f4a4f8fe47986172243ce6a55e37d73bb714ac Roll clang from 7d3fc7337eed to 953a8eb380b6 (1 revision)
  - d0be68e487b3db11c7afb363aa631d55bf9199c4 Roll clang from 3238c48cdfb1 to 7d3fc7337eed (1 revision)
  - 34b99a9ef62ddc254c1b951a7de7b3574731e006 Improve python3 support on Windows.
  - b3ef85bb978b71401645d6c718c21b55e0e6b91b Roll clang from dbe0d801bbf8 to 3238c48cdfb1 (1 revision)
  - f908f9d964908caf1914a9cf3bf006965e28b996 Roll client/third_party/boringssl/src a2278d4d2..238a25831
  - fefcc76bd39b439d3252553c0fc532b73a7e7974 Roll clang from a6f0ae6b09b7 to dbe0d801bbf8 (1 revision)
  - c2134aac77d553964ad3dc29dd3656c99ce223cd Roll clang from fea424690750 to a6f0ae6b09b7 (1 revision)
  - bc1bf1475e5f6c9761afd6311bbd2ec11ead89fe Roll clang from 1d684487dbc0 to fea424690750 (1 revision)
  - f7fd7bca34ae910b60a133a4e15ea7d51307204e Roll clang from 35e786838362 to 1d684487dbc0 (1 revision)
  - 4a809d584bfa72e2cc3002f92cb7d7fe8e9a4ed9 modulemap lexer: change recursive call to loop
  - a7d58eda07518d30cc95999cfb4223e63c3bfbc3 Roll clang from f33f4c53ae51 to 35e786838362 (1 revision)
  - 2d332250b04e3d1ccfac2b6cc97326190b10c88c Roll clang from 32c60639df25 to f33f4c53ae51 (1 revision)
  - 112cf248cc06c97f435edb94c7ad81c518d2a6ba Roll clang from c81f3d3a3e0a to 32c60639df25 (1 revision)
  - eeba72c031ca8c5353476d9a77b0f7f04e1b24bc Roll clang from ca21e9b0ac12 to c81f3d3a3e0a (1 revision)
  - a6a50c63744cac963886e12ff441cf64bad6cec0 Roll clang from 8f265d6ddfe9 to ca21e9b0ac12 (1 revision)
  - e73ba9802ca53c366b209651480cb385416f08ec client: use original_cwd in local output cache if it is set
  - 4a98d2a11f2077d7f5e179f376fb0d84d757f2f4 Roll clang from fae3adcf1661 to 8f265d6ddfe9 (2 revisions)
  - c3167a8f4d258190bbf17dfaec57e0451dcb0377 Roll clang from f18ba70b3aa5 to fae3adcf1661 (2 revisions)
  - d7c1d15325a1d0e7a592e67a4567f91442f0ed33 Roll client/third_party/nasm/ ae8e4ca1c..19f3fad68 (637 c...
  - 96a24ac2e3fdb361b08d17d0f3018902d5714b8c Workaround for the semaphore timeout period has expired o...
  - 1103b7a0d819890b2b120befa009291733317733 Roll client/third_party/boringssl/src 4251d0d3f..a2278d4d2
  - f067e03d556fe5d438c1854b73674c26fba2ff40 Update go to 1.15.8

GitOrigin-RevId: 318cf825c3f1d1f8c79699502c9dd1422635f47b
Change-Id: I31778074d5317cbe10d9ffcafbef19839c9bb240
40 files changed
tree: 03bba10b0b56c5a0465ca52e5f4cda2e10289d8c
  1. base/
  2. benchmark/
  3. build/
  4. client/
  5. doc/
  6. go/
  7. infra/
  8. lib/
  9. test/
  10. testing/
  11. third_party/
  12. tools/
  13. .clang-format
  14. .gitignore
  15. .gn
  16. .style.yapf
  17. .vpython
  18. .vpython3
  19. BUILD.gn
  20. codereview.settings
  21. CONTRIBUTING.md
  22. DEPS
  23. LICENSE
  24. OWNERS
  25. PRESUBMIT.py
  26. README.md
README.md

Goma client

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: For non-Googler usage, please see Goma for Chromium Contributors.

How Goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the Goma backend caches the compile result. If the same compile request comes, the cached result is returned from the Goma cache server.

How to build

Goma client can be built on Linux, Mac, and Win.

Prerequisite

  1. Use 64bit platform (Linux, Mac or Win).
  2. Install depot_tools.
  3. Install dependencies.
    • On Mac, install Xcode.
    • On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume the Goma client code is checked out to ${GOMA_SRC}. You can set this in your environment, but do not export it as it will make gomacc complain.

If you want to develop goma client, make goma client source unmanaged by gclient. Open .gclient file, and check "managed" value. If it's True, changed it to False. Otherwise, gclient will manage your repository, so your checkout can be unintentionally changed with gclient sync.

Move to client directory (which is under git repo), and configure git repository with your username and emails.

$ cd client
$ git config user.email 'foo@example.com'
$ git config user.name 'Your Name'

Build

$ cd "${GOMA_SRC}/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "${GOMA_SRC}/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

Coding style

Follow Google code style.

For C++11 features, we prefer to follow chromium guideline: http://chromium-cpp.appspot.com/

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "${GOMA_SRC}/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "${GOMA_SRC}/client/out/Release"  (Replace ${GOMA_SRC} to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

More details are avairable in chromium's build instructions.

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR="${GOMA_SRC}/client/out/Release" USE_GOMA=true make -j4

Here, -j4 is not related to Goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for Goma.

For general development

  1. Build Goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8088 to see compiler_proxy is actually working.
  • You can use autoninja in depot_tools instead of specifying gomacc manually.