Import changes for goma client

  - 887e584093bed123f2801b32213826584c5f9c1e http: fix http response streaming
  - aa65f0ea6d28f3bfd718d23369bae2d2e2a19c9b Simplify SimpleTimer API in preparation for absl/time
  - 4a64def045d69cca7d5a26936b7d2203004ce27d http: add HttpClientTest like HttpRpcTest
  - 5c4af20dcdad4e3db69b03950f1c6bd4611650b1 Convert filetime_win to absl::Time
  - 73a8d509cd89bf999f5ac40972fab503b15e682c Add unit test for filetime_win
  - 137a84c794e0b18ef8a78b41545cecb19397280f http: fix oauth token response handling
  - 5c71f6793e6ad5bba859c3794b572bb124a5875c http_util_unittest: remove ParseChunkedBody
  - bd0903402c7ed20870e94a4647587efc2272c567 Use absl/time in local_output_cache
  - 07b102f129eeb66bfb52dfc58f878dd8efae2bcf Make subprocs write parent stderr if sanitizer option is ...
  - fe066c0c8b2be8e41e297d69a4557b149cc43a38 workaround: make msan understand fd_set is initialized.
  - 671cefa2815ae81be1cc374b2ba32a294aff25c2 remove ParseChunkedBody and CombinedChunks
  - 6121270fa28c29a6c1124892381c9e19511683a9 Migrate JsonWebToken to absl/time
  - 1120f7adfa838559906c84c30dae75d101f00863 warns performance regression due to sanitizer.
  - 47936ef28eaebb3350c941ba46e2a82b7d7d0929 goma_fetch: add -post <url> [--data <body>]
  - 92b6b3c4ac5f7bbaa829246ccb17155aa71f01e9 Workaround: not inline CppToken::IsPuncChar under MSan.
  - 9e5ef98aa01660ddcec884537b3fc7d0d6e49261 Update boringssl to 20b6a4e2
  - b46e5c9715f8ecbddda056bf4220590b01e25ae0 msan: set memory track origin.
  - 3a7ab97e3d4ec21ac44b2f19839c1dada4ec17dc fix msan failure in spawner_posix.
  - b912af9283570a3440ea07359913771fef69aa88 Convert SocketPool to use absl/time
  - 76acdfa0e5745ccc1415d8387e9dd1f546c0dfab Use non-asm boringssl code for msan.
  - 01848d1ff4726698a051dae30fdf696d2d42d50d Convert CppParser to use absl/time
  - d3969d12144f2dfd6578b0867d263048264e00b1 Convert openssl_engine to use absl/time
  - b610dfe05e613789f3066219aad00abd141c9e97 Add unit test for Date/Time tokens from CppParser
  - d834a52f02325a8b826a14b00c5fe20253411f61 http: use HttpChunkParser in HttpResponse::Body
  - 3950032af92b9bda14c2cb1141491d9527beb5df suppress use-of-uninitialized-value msan error in BoringS...
  - 26760ebb4281a9f2d6761989b029085a607aabbb cleanup: Split cache related libs from compiler_proxy_lib
  - dcebcf9af1ef32993e6a81166a5285ff1cbd6afe Replace integer time values with absl/time in oauth2_token
  - 08399dfc3dc603f5326a82e5a823959869227226 msan: suppress use-of-uninitialized-value in logging.cc.
  - 3e447b82086467829c239575d84e1ca5556331a1 Don't log consumed memory if the amount isn't changed
  - 94a5593496990ea820141908dfdbb40d1ced6db4 Use absl::Time in client/cros_util.cc
  - 3a2da27716e53d9f19a01526698037710d2b1c4c fix HttpChunkParser ConsumeSize
  - 6ca3f0d294571203e66d727496006741be40364b Use absl::Time in client/log_cleaner
  - 39498518bf0851b4f4b7a05e52b252447df05ae4 http: refactor response body
  - 702b258adc414ba9461b013bcf4aa94029fa1a0c Add sque to OWNERS
  - a904fea69e653240b667837be888812ab3826309 Replace PlatformThread::Sleep with absl::SleepFor
  - ac1b39fb4d689404bc139c51bf63c56ff0d009a8 Prefer to do remote compile if http status is temporary u...
  - 71bdd8cb91b97fe529163cad327d646603391166 Support __VA_OPT__
  - 989a5d27a53ab621e0d7cb7b1b0b64b5b734bd2e enable HttpRPCTest.TLSEngineServerTimeoutSendingHeaderSho...
  - b6529bb5dc0add9151a7866e066254cfbee00c29 cleanup: convert File::Copy to file::Copy
  - 6c80b70adc37228f02762462b087fbf0943a45a7 Revert "Revert "http: streaming request support""
  - 2c7bea40e2b6f5c470e14e93451b0985ee658e78 Update boringssl to 861abccb
  - a394194a15ae8f800cb61ca7c81024ff945d70c3 Do retry when rpc failed during receiving response
  - 103f25caed3f9beced6f514742daa4ecc58ecf4e update message.
  - bce479902c02d6f3d867c046253991f73f0fbb55 Move LinkerInputProcessor to linker/linker_input_processor
  - c3f46d157a0cec08fcf1ad5ebe319e534557f6e3 Roll client/third_party/abseil/src/ f88b4e9cd..8f612ebb1 ...

GitOrigin-RevId: 887e584093bed123f2801b32213826584c5f9c1e
Change-Id: I8bd10495569aac1eab0bea9923b7bb0969008548
152 files changed
tree: c9d7dd14cba7e197929dd03ba375000802cebddd
  1. .clang-format
  2. .gitignore
  3. .gn
  4. BUILD.gn
  5. CONTRIBUTING.md
  6. DEPS
  7. LICENSE
  8. OWNERS
  9. PRESUBMIT.py
  10. README.md
  11. base/
  12. benchmark/
  13. build/
  14. buildtools/
  15. client/
  16. codereview.settings
  17. infra/
  18. lib/
  19. test/
  20. testing/
  21. third_party/
  22. tools/
README.md

Goma

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the goma backend caches the compile result. If the same compile request comes, the cached result is returned from the goma cache server.

How to build

goma client can be built on Linux, Mac, and Win.

Install dependencies

  1. Install depot_tools.
  2. Install dependencies.

On debian or ubuntu,

$ sudo apt-get install libssl-dev libc6-dev-i386

On Mac, install Xcode.

On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume goma is checked out to $GOMA_SRC.

Build

$ cd "$GOMA_SRC/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "$GOMA_SRC/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "$GOMA_SRC/client/out/Release"  (Replace $GOMA_SRC to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=$GOMA_SRC/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for goma.

For general development

  1. Build goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.

Tips

  • You can use autoninja in depot_tools instead of specifying gomacc manually.