Import changes for goma client

  - df5af6208f6dcc90112ab722da8ebd0bfc9a5db7 Update clang_version
  - 0a87b97db08ce682ed142f62ea3e761bb4157097 Set infra/config/** as common copybara exclude
  - 3686c3a9be6c23081be6e702b4b2aba871a72e3c Copy MSDIA140.dll to target directory
  - d9f7c9e269c5ec69dc16ba48401ae7f77af100e1 Update boringssl to f8058d41
  - 4ac22b33139a0e88f2333c3686be3681beb7ae5c cpp_tokenizer: support \<octal> or \<hex> in char literal
  - 3e2e50eb7406a5e42990c8f8479b43b8709dbc30 Use GUARDED_BY for named_pipe_server_win
  - ddf8b9a45abf4bd001fd390faa636e3d6446c816 Scrub NOTE, HACK, and FIXME too
  - 866c005eab5542c0f8d051c4623a829b0ea22300 Refactoring: Don't use string as buffer
  - 80401c3e7e2944f03492344448df2e8ada5adeda Move third_party/goma to third_party/goma/client
  - 79fe2179604ce2e14f5cbddf3f94c9d042e41664 Refactoring: Mark lock mutable
  - a56a6696488b881b891d399c7b0ca942dbf83125 User-friendly error message of gomacc
  - fa18a560e98ae59a00454bfb2981bf1851a292b5 Refactoring: ConditionVariable ctor/Wait()
  - a63fa6bc244f79ae10129889deb32e20fbef7b7f Check auto_updater_ in CompileService
  - 324a3f6f4f44ea5ee1a200d3db9cd3ebff2dbe46 Disable auto updater by default on Windows
  - 71efb733c2b2a428e6abe8c763f0a9ced4277d2e Dump counterz in quitquitquit handler
  - 393f281b73f2dfd6c784dce7adf0fe9f276eeb10 Support -fthinlto-index=.
  - ff18d87de112c1b7af500852c7cffe3223a406ea Increase default NAMEDPIPE_WAIT_TIMEOUT_MS from 13s to 30s
  - 48f0c4dedef53ee4f3b75ab475e1f37e19e8031c Add absl::SkipEmpty for StrSplit
  - 14683be1df5b8c0df9c5ee6d6889e0c87598c4e7 Refactoring: Remove third_party/chromium_base/string_util.h
  - f6478eb78786b87e3649586c12d0afa2232bb259 Add absl::SkipEmpty to parse version string
  - 362ab8e90ab066ff90f334259eff653ccb8f374a Replace base/strutil.h with abseil
  - 10c3f134e3fb31bd652cfcf0dbd9f644ba45d6bf buildfix: Remove split_unittest from run_unittest.py
  - bb61eb5d6439ef1625fc19cc0ad9427b539b44ff Remove MAX_ENV_BLOCK
  - 210222c7cd63315a8f584e0545c7231a90bb46de Refactoring: replace strings::Split with absl::StrSplit
  - 8965c3abf07aecd4ab114089adc5325778a31283 Organize cctz BUILD.gn correctly
  - d7eb2ff25c452b5f73afad1ef1ff5c448a12028f Update boringssl to 085955c5
  - 3265dfeece57c94f3ba2dce4493b6722a4a93ec6 Remove non ascii related functions from string_util.h
  - 7040d630e9ba41e728dd9f67be73de02fffcbcf2 Make GetUsername() in goma_ctl.py the same behavior with ...
  - 7e15a416eefdc47a4aa17ae824660b9328d7f683 Replace StartsWith, EndsWith, StrCat with abseil
  - fc416e3aef66f5c74b7dd63e2960bf700a96f9a7 buildfix: don't assign StrSplit result to std::vector
  - a09c2238a5bad0e9a0208e347d9bc38e962711b6 buildfix: Remove absl/debugging/stack_consumption.{cc,h}
  - b6f1b0ef59d970a894870d67f350d54c6c7a5d1d Reimplement strings_piece_utils with abseil
  - e0850e543e8c1273d460cede30e34bfe7d9e6aab Remove StringPiece, use absl::string_view
  - 10d6afced424a9712b9b02247a463befd29a2a76 Refactoring: Use absl::StrJoin instead of strings::Join
  - 431965da988cdadc5c76b2702f649ad9b5961643 Remove string_piece_unittest from run_unittest.py
  - f92c0f723dd855da5c71bd937d322f0b87ed8098 Introduce abseil
  - 48d777018f6ad70bcbabdfc4e214fa50819be289 Remove gomacc_ipc.{cc,h}
  - 098334095afb0f575cd0913ac390b3c6a6f91fcf Update boringssl to 638a408c
  - 74befa1ff9651572d6661b52e7150173e1ceec9e Increase DEFAULT_MAX_ACTIVE_TASKS to 2048 from 1024
  - 704daa3d7bb91eb738013ce29c10caa53fb633a5 Use std::unordered_set if a key is a pointer.
  - 9a1b6c4636428e3093847cea5a27d2a17eb9fcdb Remove unordered.h, use std::unordered_{map,set}
  - dd9639aca063e7f28a8329e4249561bdedf808a8 Don't use std::string in lib
  - b91c7ebd62998c3c667ee0f45a9ab1c8fe2bb590 Won't rewrite COFF if /Brepro or something equivalent is ...
  - b494e3ef68157e5056adb108a60d6d2814c9762d Add "fallthrough" in known warnings.
  - 845cf8e6797e2f42710752f943b439fcb5922d1e Use emplace instead of find + insert in IncludeFileFinder
  - 79ba638d1197db35d519f1e94cc61add1cbe380a Update boringssl to 2044181e
  - c50198debac0a10a0bed9aa6bc3c500f4e974c58 Use absl (backport of cl/184474102)
  - e7bb70e458255cee0157846eb47316fb74c17c9e Fix data race of gSubProcessController
  - 5b46849de7e1d47f77d7680894e8a11f9326ee7b Initialize INIT_ONCE with INIT_ONCE_STATIC_INIT
  - 1edf008eae5bec418434c061bba982b6fe8ee1ee Fix data race of subprocess controller client
  - fe26e631fdb50d1e5b5595107552e0965208a095 Fix data race of HttpClient::Status::finished
  - da19513e597a5145c79942f591388109acdfba56 Add /Brepro- flag for clexe
  - b25c7681323e171ff2dfae534f5bb24a8099c3f1 Add /Brepro flag for cl.exe
  - 28dcc1d3c50c59d3788c77575caa62f81b3d6539 Escape double quotation in double quotation
  - cd50b65dd23dd45e403f1f0945a37cb6ea275bb3 Support -std for clang-cl
  - d4dc96a84c142bc629e9ae88f90972ca75a51f4d Update boringssl to 5301c10c
  - af3f875a6e0b288dc01ddeea72ff7e1e086aa63a Implement synchronous stop of compiler_proxy.
  - 22a17ce30e9e0ccc960136d3725b6f861abed1de Make more javac flags known
  - 4556b622bc2b44f2d2aef12fcd802588e5f3d775 Add absl::StrSplit
  - be17a3ab827e06055a14e7bd83c25c35eb132b8b Understand --system-header-prefix flag
  - 3a0f899a870e1906e63560eece4d826a09de4a11 Considier attributes having :: in C-mode
  - 0522c95c4e45b5ce548617bfd1ccd359f8bdff8c Update boringssl to 5ab54840
  - 7f08409805d825fad4551ff03fa6efdff68d010e Add flag to enable counterz
  - 66e02813c20fce68751083a039c95a458f1bdae8 enable counterz by default
  - 2b01af9714a60ca43066b007159f239c9fac5853 Allow clang warnings
  - bb760dd3c5903ac5d2e80470ec34271baa0801a1 cpp_parser: handle char as int in evaluator
  - c057ebe44773593f7412b408a49bcc379bed0d50 Fix: file contents should be got with absolute path.
  - 323d2a7b0f00382e7acd58f7e82eecb9d1bddfae Hard compile fail for unsupported compiler flag
  - fb07aff96f2ec93653df7613c0bbc48bfa0e8d70 Don't assume __GNUC__ is the same in compile and test
  - c370ef8fba8642e48329698ffa75936c0da48200 Dump counterz using protobuf
  - 6601cd8c0e2a12bc6524169dfb35850b934e9293 Understand __has_cpp_attribute even in gcc C-mode
  - 63bd66f848307d28c162c59f2f9ae8c2c384ca8e Update gn to revision 523254
  - b91440a99a4d585a20cae30fb9ce9fef5b034371 Update boringssl to 94cd196a
  - f8a1801ff11c418ccdfd9cbf1d9167fe1a53efa7 Add GOMA_COUNTERZ for each preprocessor directive
  - 7be6298661928e63d4a5142014735c8f90a472f5 Make more warning flags known
  - 01085727194cf1e25b169b39633542b837607722 Fix crash by non-quoted arg0
  - 5d92f99e8998d5e4235d31825b02666a2ed51a0e Support more warnings
  - 8293c9df89326291d27c2e810be0e50934d57309 Fix gomacc crash for counterz
  - e478ed30d2d1bea43c5b456b256fcb8d21e941c2 Insert GOMA_COUNTERZ to some time consuming places
  - 8caa044a87550aa9f8c77867cb244f6e4373bc74 fix clang-cl warning
  - cf804d5dde1fa273d328515f7a4b740b1af5c4b1 Make CompilerFlags understand -fplugin=
  - 9f2883846c87e2a02b6204224cabb79587cd79d6 Support -fms-compatibility-version=
  - bd68033dcc39d4a8fe38ac2077def39f4b1ea6ee compiler_flags: support -fprofile-use for clang
  - 5be86594ec361cc546faf9e1980aded1c8e1b414 Update boringssl to 0c9b7b5d
  - 2609eaf3c54c5091bee99e05a290e49d8219ad86 Add public version repository URL
  - 503f8179c682a4b4f696491c649a5c0a10374f64 autlock_timer: sort by total wait
  - 12086e314e628da174fe7c04c9c9f7eaf1033d4e Add publish_to_external_repo task in copybara
  - d06fabe3f93e5976259d9c344df284396443cf51 disable warning as error for clang-cl
  - f76ff19559906436075ef76f7c75bc1d919627b8 Make namedpipe timeout configurable

GitOrigin-RevId: df5af6208f6dcc90112ab722da8ebd0bfc9a5db7
Change-Id: I903dfc41d7117efe16cb498a55d64138c04c0ca1
228 files changed
tree: 79331cd477f191ad9e1659c52460cc981049e09a
  1. .gitignore
  2. .gn
  3. BUILD.gn
  4. CONTRIBUTING.md
  5. DEPS
  6. LICENSE
  7. PRESUBMIT.py
  8. README.md
  9. base/
  10. build/
  11. buildtools/
  12. client/
  13. codereview.settings
  14. infra/
  15. lib/
  16. test/
  17. testing/
  18. third_party/
  19. tools/
README.md

Goma

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the goma backend caches the compile result. If the same compile request comes, the cached result is returned from the goma cache server.

How to build

goma client can be built on Linux, Mac, and Win.

Install dependencies

  1. Install depot_tools.
  2. Install dependencies.

On debian or ubuntu,

$ sudo apt-get install libssl-dev libc6-dev-i386

On Mac, install Xcode.

On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume goma is checked out to $GOMA_SRC.

Build

$ cd "$GOMA_SRC/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "$GOMA_SRC/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "$GOMA_SRC/client/out/Release"  (Replace $GOMA_SRC to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=$GOMA_SRC/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for goma.

For general development

  1. Build goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.

Tips

  • You can use autoninja in depot_tools instead of specifying gomacc manually.