Import changes for goma client

  - c1ddb57837507a7347f10210efd428bbe165b8fc Add "depot_tools" to DEPS.
  - 880a1f60fc2093eb4f7f57375b7740af11c76776 run_unittest.py: change default --build-dir.
  - bb1e51dd7c5fbaabf9726e345fb3b4dd822630e7 Update message.
  - db65fb3cbe6c7edc80deaf3d8c947e272d911c63 run_unittest: use argparse instead of getopt.
  - 820542bd8f38fd7fd3347f3f1b317b0127f02560 Fix: typo
  - 4656343182d83f6185809a7f33a3a9c237cec0c8 Make worker thread die quickly
  - 00f080190ec83ae97a5c4e476281f9946e54aadc refactoring: Rename CompilerType to CompilerFlagType
  - 996caf60d40e761ef34975506d6a38e142b73a34 exclude OWNERS.
  - fa950ae384193564634c7ae3a2dfa00993266f9d Move cxx include processors to cxx/include_processor
  - cc214fd2bd398868a993ab67f6f45b20cf1755cc Make LZMAOutputStream less chatty.
  - 6088284b16782627be0d18d4dfb0b1ccb5102745 Remove deprecated flags
  - f8fdc6f28859e3fcde2c8210772bd61302a0b5bd Make compiler_info_builder_unittest faster
  - d0b1c32568dc310aa916c0d6323fceb04091f8d1 Use smaller kIdleIntervalMilliSec in tsan build
  - 85be5e6e6c0975bcc763d4ba86ec126154b3bff1 Move C/C++ data to CxxCompilerInfo from CompilerInfo
  - a5b2d257ed5286e476b704ed967b20ba20c14e9d make msan available.
  - f8b87ba4e7a1512920f6388e89aa149bb7dee1b9 Fix msan failure in LZMAOutputStream.
  - b96b5bd683a2f5cb42339ebc201f5c8de7f7c389 Ensure CompilerInfo is CxxCompilerInfo in DepsCache
  - 632811680d393afdfd14409569d207f66e7a9459 Use CxxCompilerInfo in cpp include processor
  - d21c73f1ec0e8734d380b853588ffa5d4cef49cf fix: LZMAOutputStream memory leak.
  - 84bae7464608f1f39bb2957e14131c33526a1cbf Restrict subprocess delay time
  - f1233d443fbf2a2066957be56d119a6c2abd353e Revert "http: streaming request support"
  - cbf89e0a5faa76811adf06042944c1cd30c408e5 fix sync again X(.
  - 90da4cbe0702c69d390bf780a667fc30d4485bdd Use dedicated lock for refcnt_ in CompileTask
  - 1f75ed037997fd41586f5d212c052dd454db305c allow to access error code in LZMAOutputStream.
  - f0fccd1ebe9cf6f1deba125e0ddba6f85d1ce23a fix sync failure.
  - 60912f103caf206dac8c76f973392ffae4e62867 Update boringssl to 82639e6f
  - c3ef5de353292c6b827fecdb3e26ab3a05a8b9b2 Reenable subprocess kill.
  - c74bc55d1c9f32cabb6e3d53553c1e9c9a63ae38 Introduce language specific CompilerInfo
  - 2bbe3f65954899310e06959c81179a3c6df7b099 Implement LZMAOutputStream and don't provide ReadAllLZMAS...
  - db0196cb8f55ddb572cab14811497452c6ac78b8 Fix typo begin -> end
  - 2cfbb23446eb8c3affc6d2d87900c169fa40e8c1 archive.py supports name in api.platform.name.
  - 2b58ff8c0e3b910028900764a885b9fd23b9f18c http: streaming request support
  - fd442b51df52760209b645b0cd7e73463d6ec1d8 remove GetBaseDir.
  - 58d736062c5a34567e4bff2316524aa890b4c5b1 Add GetClangPath function to unittest_utils
  - 2621fb73009e1aa968cd615a141e5fb44a376cbe Add smoke test to get compiler_info from clang
  - 0b1a6b2d90a03725a1ad61eba1a2247e6f5871dc clean up: rename to InflateInputStream.
  - 44294f29ab62f77b7f09e4b64aa22cd95873b19e Don't mix pipe to send pid and SubProcExit
  - 3ae8db2e9df230574cd80dabaca6cbb3985e7ee4 clean up: make compress_util have a code to handle deflat...
  - cba34486969d19c8ce8cbf3837d1f19967dd7f98 Remove CompilerInfo::SubprogramInfo::FromPath
  - 1857bf2080ce93b9591263fad59ed4c59e0d59b8 move GetCurrentDirNameOrDie from ioutil to mypath
  - da380f6580fcb757308be701976396c2cfe8d9ed Make gn check run when we change source files in PRESUBMIT
  - b6d2888f2581f769e9e7172abc0de84636de7e67 Fix subprocess_task_unittest for chromeos bot
  - b00b7002037a83ab98b278572c080e4d503858e0 Make the name of scoped tmp file random.
  - 4395d3763864b0fae704553ad5202068be3f3ea8 PRESUBMIT: use cpplint in input_api.
  - 0fcd110503d1eef9e628354c88d77e594d22617e http: use LZMAInputStream instead.
  - c39b0a554ccbafab998614185c9d0ad3e8e9fe08 Add test to verify that Clang subprocess can be killed co...
  - df448822d3b3cd0920f1635a396572518dd2aa8a remove DeleteRecursivelyOrDie
  - d37c7d0ce842ea9a8eb973235a7bfdec4092fbc0 Resolve path in GetMyDirectory
  - 6cb6a4938a3abe5592e852f4a7e63b309fe1e670 http: refactor GetContentEncoding
  - e73b1fa08aae46ca646836baa17975826831c8f3 refactoring: split CompilerInfo and CompilerInfoState
  - cb7a76c667d44418ceb562bc83d24cb90416b323 move http related code from ioutil to http_util
  - 50c4ee02728342b0ebc945eb1eec3756d99f6cce Ensure one CompilerInfoData language extension exists
  - f06656ef2be823a7c6c360db9a95dda1965910fe Create cxx/ directory to contain C/C++ related code
  - 9b549a2d6bba6a6ec55fcd12c8a33931c27b1c17 Gather clang_tidy specific code to clang_tidy/
  - a7a3e77d79da347e336f5be7c721c3977af6c131 Revive subprocess_task_unittest for Mac
  - e8a6e18a8bbdeecd7cf6129304d3e65cc4b59dfd http_util: add HttpChunkParser
  - 2efe14183752499ba66bdb55f3f5473793c3d8be Split CompilerInfoData to common part and C/C++ part
  - 4e62802659f5489fd2eb2dafa1d36ac55a72b2f8 Remove subprocess_task_unittest from mac temporarily
  - 40552a70972d4c9502b791b4d6dfd71c755fe29c http_rpc_unittest: fix TLSEngineFailWithTLSErrorAtRead
  - 62bc17d218d9a4c0380f65ef4424d33161deaf8c run_unittest: detect unittests by gn
  - d6b904b01b055ad59718014f74a7502c3e42890b fix posix_helper_win
  - 19ed1d82ffaa2d7d85a45497c451f4d65f2d6c0a Wrap CompilerInfo with unique_ptr
  - c117f943efdd3dc3fe94dae28c7dfe946acb7865 Use valid cpp testcase for IncludeCacheTest
  - 851e1e20f30030a3255ba8f93f37215e7d5a1335 remove http_util_unittest from run_unittest.py
  - f3c34db95e15373a04f5858d1fb27f9de99ee940 Use absolute paths to check executable existence.
  - 2ba07f01f8e3c83b93036e64c740e5c06edf65ea Update boringssl to 791f2822
  - dd008ce522d5d1352aa360b0854b00035a066308 Use absolute path to check nacl-gcc existence on Win.
  - 3e564e0ed4d768b5474ef1fb0e3a600ef025ef34 ioutil_lib: base should be public_deps
  - 53d6786fef674ff72928149678df1102b24d6884 goma_fetch: add --head option
  - 4c7a094ede3fdba87a46e7eb1ebeaadc608547de Do not set abspath to Input
  - 5ff8711f0c5640988701f8717c21b6c65593b3be Fix usage of ReportDifferencesToString
  - d3372a0c77380449c9019f95873c06893ebc601f Add presubmit check for gn check default target.
  - 9ea770d8737df8c46f8ef89f979c59b6778fe80c Add ClangFlagsHelper for clang specific flag
  - 336c3463f6987537ed8cb820d6a37a591327b181 Introduce CxxCompilerInfoBuilder
  - ffc29a58083e053b2af9461f36b5a7bec5e08a97 Support fdebug-compilation-dir for clang-cl
  - 74ece4e79b525cc5bdcf636ceb6ae2d495af469c Ensure CppInputStream has filename
  - b189d490066c68b3cd8766accd62c2d7a88b79c8 add goma_blob: refactor file uploading.
  - bc09ba0045c3e795991b441d7147f24cf1e7e6dd goma_fetch: fix retry without authorization
  - 7d7250031161c7ab38e0b796e954a2ed8d59e9be Generate html reports for files and directories.
  - 59af816bd7d9ed5acdc4dd49864c02b10450e430 Fix vc execreq normalizer
  - b73825740c0b9aceca9e75cc0156e18eef17b121 Ingore #warning
  - 25ef34fa6dcf59fd9d859de27d2e139e82e86ff5 Log descriptor poller type only once
  - 98ffeb9d4e5974a19ceda69492ec42e099a7f5c5 gomacc: logs if connection to compiler_proxy fails.
  - 698d5c7ce3280ca3d683b558cd55639384d3d34d Log poll mechanism in poller ctor
  - 098aee529226b9642ddd8753eea777b91c02a63d Calculate per-file and per-directory code coverage summary.
  - 862243a704627ed3f71b2d7720895cb0148a9530 Use appropriate descriptor poller
  - 043fdc3df2e83b76ea2a1015606a1137fc47b18f Print message before login.
  - 2c6e01cedb4d667a441ae83b22700701364baae4 Clean up clang_tidy_execreq_normalizer headers
  - b9375a67c0d860d6d8028981c2046fb6db5a6aec Fix all gn dependency on Windows
  - 1645010e2fb58ac90b616546def9f2d60e8da818 Fix all gn dependency on Mac
  - 22066672fa8bc64ce368ac32b876f3873b2361a4 Use local compiler path for compiler invocation in windows
  - 12cc2da421c9ce9ea25b1460483926327aaf80d1 Fix all gn dependency on Linux
  - 9d31bd6221bf23b76788595eb4196ef0e95de787 Generate per file html report.
  - be55c48f02e738ed9d4ada4a79d01fd4667f7fb4 goma_auth: make login command similar to cipd and depot-t...
  - 220b92c0378db84edc4d03227dbdc4a08712414f Split util
  - b52a30ca803bc8e2238133413eb314512ff33fc2 goma_auth: fix pylint
  - ea6b5a67db853e06ab811771bea7a30cefaed995 cleanup: Don't call CppParser::EnsureInitialize from Incl...
  - 2f21595b1ca02f57abffc3d3903975b13ff687c9 remove static_darray_unittest from run_unittest.py
  - 3e5e9e5073ee7a61659b939ef86d54c22ad796bd Create indexed profdata file.
  - 8677cfc8544009aaf5b469a3de3ece47dbc56378 Update boringssl to 954eefae
  (And 81 more changes)

GitOrigin-RevId: c1ddb57837507a7347f10210efd428bbe165b8fc
Change-Id: I40e9135b2c7337be1e20b2be000a10b58863356c
276 files changed
tree: 288be17c25f14ef027d4eae621fd5fd20737e6d9
  1. .clang-format
  2. .gitignore
  3. .gn
  4. BUILD.gn
  5. CONTRIBUTING.md
  6. DEPS
  7. LICENSE
  8. OWNERS
  9. PRESUBMIT.py
  10. README.md
  11. base/
  12. benchmark/
  13. build/
  14. buildtools/
  15. client/
  16. codereview.settings
  17. infra/
  18. lib/
  19. test/
  20. testing/
  21. third_party/
  22. tools/
README.md

Goma

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the goma backend caches the compile result. If the same compile request comes, the cached result is returned from the goma cache server.

How to build

goma client can be built on Linux, Mac, and Win.

Install dependencies

  1. Install depot_tools.
  2. Install dependencies.

On debian or ubuntu,

$ sudo apt-get install libssl-dev libc6-dev-i386

On Mac, install Xcode.

On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume goma is checked out to $GOMA_SRC.

Build

$ cd "$GOMA_SRC/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "$GOMA_SRC/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "$GOMA_SRC/client/out/Release"  (Replace $GOMA_SRC to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=$GOMA_SRC/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for goma.

For general development

  1. Build goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.

Tips

  • You can use autoninja in depot_tools instead of specifying gomacc manually.