Import changes for goma client

  - 4f4a81fefffa9d02611dfe2076abd424f1fc2d6e Use filestat taking time to know whether filestat is stal...
  - 611102dcbecf62dc1d504d596d2eb4c9228a0143 Try to use sub-second resolution for file stat
  - 45f9c7be7ae5b8e08642faabd69f1c2dd3b5d883 Roll client/third_party/boringssl/src 0f5ecd3a8..bf5021a6b
  - 42d66d9a1857da2034ba5c02573a9bd836092736 goma_fetch --noauth by default.
  - 8c6e396f9afbdbb2b14e2860499c9744f91eeafa oepnssl_engine: clarify which OpenSSLContext was used.
  - 83fd9472fcf3b3c3fcd2c93afddf250c85f10410 Enable subprocess killing on windows
  - 3f81a2d603830c8ecd1c7028305e174ca4c17ed4 Allow empty hash_key if not using remote.
  - 21e611e515aaa3e1c2872cf9050b55f24afef02b http: mark "error:" only if GOMA_FAIL_FAST is enabled.
  - 91afaf1efd549353efefd80253e3e4967b78f8f0 Omit elapsed time from goma_ctl.py stat error field.
  - e62af20e2093a5f84152e29908f3780785290b44 http: clean up code to use proxy.
  - fdf906b599a96517d3fcc0a3d4e3fcc39938c1b7 DepsCache doesn't support link
  - f4cb027ffb8586c3678da68ef07efc9b082707fa Simplify condition
  - 46c87e22ec1f69e44ae40b40aee1f6a2a5feb3ce http: Also pass the destination host in the Host field fo...
  - 75b0b5a0b5ec2d6df197877117ef0403b407a1d9 CompilerInfoBuilder should work correctly with clang modu...
  - 2dc4f6fc403ba59f8cd8b6e0adce1f74e0744f7a roll_boringssl.py: suggest how boringssl was rolled.
  - 40a6d9fb206b0f732fc08dc04ce72772feb04115 Support pnacl-clang, nacl-gcc, and nacl-clang for arbitra...
  - 9ef6060150a60bc441af6219717cb29c56ffa09d Introduce ToolchainSpec
  - fdc4e10a60147b8057e9fc864c68077e572b3823 gcc_flags: clean up important environments.
  - f32535c85dad986645e4418d0c1fc52ca82ef591 goma_ctl.py: Not to use proxy to connect to compiler_proxy.
  - b83eb729a0d3c799b69212d6339e29ca0c2ed1ce Extract nacl-gcc code from clang_compiler_info_builder
  - ab089abda5a74339d42b9559bf47b16b0c1e7f9f Add GCCFlags::IsNaClClangCommand
  - 57c05f37771abbdcea2173f2d236aa474260b376 Roll client/third_party/boringssl/src 46e12b03f..0f5ecd3a8
  - 2586d90c993b5bf54b89ec6addc471abdfcf833d roll_boringssl.py: remove removed asm files.
  - 14c5d46723647b9ccba83c6871304ba491e9f874 Show dump request message on web browser
  - 8414ba462d435d6634a63a312e1fa4134258760f .gitignore architectures Goma client do not use.
  - e0b510cc4301c7452c7837c1fadcd58d2a60a3a4 Import roll_boringssl.py and modified for Goma client.
  - bbf67b466f5c675e72434dfd14661d9f29b29084 Update PRESUBMIT.py for buildtools.
  - fec6c326835d374477ed5e90448d08ea6872bf18 Use subprocess32 to set timeout in test execution
  - fdde074b6a36305311989b805afd1d244c11385c goma_ctl: crash dump code visibility change.
  - 82280082c6100eac5f58b5ac00a2f27ac11c9da6 update early-access-guide.md
  - 3a17e8ef44b7b444f1448c55148818a3ccee7459 Remove HttpClient::Options::http_host_name
  - ff1e43cc6dd24f7ee943bb7f2cc82bf892d210da goma_ctl: Remove unused proxy parsing code
  - 49041e914dc2d98c99ca77fb85563a832d47fb46 Cleanup build config
  - c3be787218de4b727c15ec88d637bb5f93e3c061 Remove libc++/libc++abi from DEPS
  - 08135f74a24a9fb4537de0b7a2547eecad927e97 Use libc++/libc++abi in buildtools
  - 3e2743863ba03e2756a1ae679a6913516bfb5072 Send subprograms as input
  - 4b5f7d266b30e7cf7522d7da7f1a39365c05cf7c Replace buildtools directory to chromium's buildtools

Bug: 912917
GitOrigin-RevId: 4f4a81fefffa9d02611dfe2076abd424f1fc2d6e
Change-Id: I8ff3b5eb7c74f91d9877fbaeb530d8fdeb014e13
152 files changed
tree: edbca6f583fe891247842c858dee73af0091f112
  1. .clang-format
  2. .gitignore
  3. .gn
  4. BUILD.gn
  5. CONTRIBUTING.md
  6. DEPS
  7. LICENSE
  8. OWNERS
  9. PRESUBMIT.py
  10. README.md
  11. base/
  12. benchmark/
  13. build/
  14. client/
  15. codereview.settings
  16. doc/
  17. infra/
  18. lib/
  19. test/
  20. testing/
  21. third_party/
  22. tools/
README.md

Goma client

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the Goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How Goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the Goma backend caches the compile result. If the same compile request comes, the cached result is returned from the Goma cache server.

How to build

Goma client can be built on Linux, Mac, and Win.

Prerequisite

  1. Use 64bit platform (Linux, Mac or Win).
  2. Install depot_tools.
  3. Install dependencies.
    • On Mac, install Xcode.
    • On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume the Goma client code is checked out to ${GOMA_SRC}.

If you want to develop goma client, make goma client source unmanaged by gclient. Open .gclient file, and check "managed" value. If it's True, changed it to False. Otherwise, gclient will manage your repository, so your checkout can be unintentionally changed with gclient sync.

Move to client directory (which is under git repo), and configure git repository with your username and emails.

$ cd client
$ git config user.email 'foo@example.com'
$ git config user.name 'Your Name'

Build

$ cd "${GOMA_SRC}/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "${GOMA_SRC}/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

Coding style

Follow Google code style.

For C++11 features, we prefer to follow chromium guideline: http://chromium-cpp.appspot.com/

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "${GOMA_SRC}/client/out/Release"  (Replace ${GOMA_SRC} to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

More details are avairable in chromium's build instructions.

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=${GOMA_SRC}/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to Goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for Goma.

For general development

  1. Build Goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.
  • You can use autoninja in depot_tools instead of specifying gomacc manually.