Google Git
Sign in
chromium / infra / goma / client / 04504b62d2d073b9bf5038f2ae1f47794a4c967a
commit04504b62d2d073b9bf5038f2ae1f47794a4c967a[log] [tgz]
authorGoma Authors <goma-dev@google.com>Mon Feb 18 04:58:46 2019
committerYoshisato Yanagisawa <yyanagisawa@chromium.org>Tue Feb 19 02:48:51 2019
tree7b444216dcd0665e294ebe286238fdbd038d197c
parent01dbe2875fb5aaa3a3bd125b40afca28ce2faa26 [diff]
Import changes for goma client

  - 5047ff0e6b3eeecac36906d5c47ee2b43b70c6eb Roll client/third_party/boringssl/src 20a9b409b..104306f58
  - eeec7effcec655c0861afc4488c2217fe7946a65 don't send a request when a header file is missing
  - de1a61cba4d499dac8ea2545fbb3a595d7e52e86 Update gtest to 1.8.1
  - 40e68c5d85cc58b8e6cb7092e368ed367072f1f4 don't consider 32bit user
  - 2031f083293f92d0765adb51efb7b94e0e253058 Also log virtual memory size.
  - 73a87f973e6b76ca019c4d6b73c4033455801709 Reland "change oauth2 token URL"
  - bbe7998cb9fa56b117f8ad706f02bdabefc78116 Use LOG_SYSRESULT instead of PLOG in machine_info.
  - cc9e3f4310d630c77eeb8be812237d76282ba955 Add GetVirtualMemoryOfCurrentProcess.
  - edc5233949845f38c582ed21bf7b89ae2821382d improve log to see the reason of blob download failure
  - 631943657ad734de38cee03610a580a40af0fb14 Obsolete the way to get temp directory from environment v...
  - 5c782029c3493c299f312f3f107c90438b91f3ea client: Initialize CompilerFlags in compile_service.cc
  - 5fd5e4d9e51181dcaf1e8df21dcefb88b8943d33 lib: Simplify CompilerFlags failure handling
  - 92582169aa7be95f3e66ec664aad729eacaf1f23 client: Improve labels in compiler_proxy status page
  - 65b222dc9dcaa824515aac018cd03d9444c8f709 Check FileStat proto value really exists before copying
  - 0b9a8f43e1a7e5cefe3f961657ba6d2e0a693d30 client: Default-initialize variables in CompileService
  - 5c8c00aa2448cdd329fb2a55b6bd60f2333f0027 local error should not be shown as remote error
  - bdfe2d9a89b2c9c29c65c334ab59827748d77c57 client: Move RpcController/MultiRpcController to own file
  - b6133b62bb9dcf167e4da29d4fc9b41bfe020bab Check patch formatted or not in PRESUBMIT.py
  - 8db287703e187485d0a38ee89afe674c1f1d14fb cleanup: replace basictypes.h defs with C++11/absl equiva...
  - a7ea3d5da10e4ae39fa5b13af32ec55e876db5fa client: Use default initializers in CompileTask
  - 748d530f74fd10fab6bc26331af20bf443c9b354 client: Add func CompileStats::AddStatsFromOutputFileTask()
  - 7d64ce0761aa90e4fc25275945a322cf4431b845 client: Helper funcs to get NetworkFailureType and CacheS...
  - 92b71172142a7a863c7ef66d58b4822dbe567a46 client: Move LocalOutputFileTask to its own file
  - a9dce4ee4dadd67573769f8cafe4111ab3bf6ee6 Revert "change oauth2 token URL"
  - ee2b803b8ad77c69ac80407a118dd9513e658460 client: Move OutputFileTask/OutputFileInfo to their own f...
  - b59e405562b906c46b3408ac5311dbc2559923d4 client: Initialize CompileStats in CompileService
  - 410ec328855e40f864dd076fbfe7a8e372eef4c0 change oauth2 token URL
  - 3676e896403d76740c9a9429b6e9fa23422fafe4 client: Move CompileTask::InputFileTask to its own file
  - 1355c5229b63afa07f8a4904d1476bb1e192b11e client: Set default OS dimensions and path style in Compi...
  - c4772743e3fb1d5677518cf9fd6923c293677530 add trace id to log
  - edfe4919685647c433580a42925c50d3cf038c47 compile_task: more logs to see file missing case.
  - af363f32d641ffdf9e8ccc98465c6b2acc775005 compile_task: revise to use absl::flat_hash_set.contains().
  - 36da9cdd7ad3dca7efe65f8cc2516a742917017c lastchange.py: import from upstream to fix pylint warnings.
  - bd626fb6258d7ef8a7a31219526a56b89cb85c2a Migrate client/ directory from python2 to python3.
  - a56178f15d90a0ab02f9e22cf46c38697b40e4d0 Migrate python2 to python3 in test/.
  - d61ce838641f2212f5aeeb90bed6540b91c3c35a Include third_party licenses into LICENSE
  - 9032de8eac90bfdfb1106c21b8f8095bd89886bf Roll client/third_party/boringssl/src 8cbb5f8f2..20a9b409b
  - 3fb17f3f6909c156b9d3d7636466c0dfcb68f4a7 retry ping when connect timed out
  - 846675d72e7a3adc36ca15a7feedd00353defa1b use goma.chromium.org for default API endpoint.
  - 2b8a49db7c78074c2eb6f539f37f9e063ad77389 Update jquery to v3.3.1.
  - fd939f20603d1d628bc118b6bf56b66f9fa5a431 Replace chart with chart.js
  - 315f333c61f964b30b61c5b24f2ca31e24627d87 client: Set mutable_requester_info fields in CompileService
  - e551e805b67cc9b1013a2c2d3a3cc72499779895 client: Create CompileTask::InitCompilerStats()

GitOrigin-RevId: 5047ff0e6b3eeecac36906d5c47ee2b43b70c6eb
Change-Id: Ibac7b0d0e5072fe3b456a3dabfdbda7dac7dfdb2
115 files changed
tree: 7b444216dcd0665e294ebe286238fdbd038d197c
  1. base/
  2. benchmark/
  3. build/
  4. client/
  5. doc/
  6. infra/
  7. lib/
  8. test/
  9. testing/
  10. third_party/
  11. tools/
  12. .clang-format
  13. .gitignore
  14. .gn
  15. BUILD.gn
  16. codereview.settings
  17. CONTRIBUTING.md
  18. DEPS
  19. LICENSE
  20. OWNERS
  21. PRESUBMIT.py
  22. README.md
README.md

Goma client

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the Goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How Goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the Goma backend caches the compile result. If the same compile request comes, the cached result is returned from the Goma cache server.

How to build

Goma client can be built on Linux, Mac, and Win.

Prerequisite

  1. Use 64bit platform (Linux, Mac or Win).
  2. Install depot_tools.
  3. Install dependencies.
    • On Mac, install Xcode.
    • On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume the Goma client code is checked out to ${GOMA_SRC}.

If you want to develop goma client, make goma client source unmanaged by gclient. Open .gclient file, and check "managed" value. If it's True, changed it to False. Otherwise, gclient will manage your repository, so your checkout can be unintentionally changed with gclient sync.

Move to client directory (which is under git repo), and configure git repository with your username and emails.

$ cd client
$ git config user.email 'foo@example.com'
$ git config user.name 'Your Name'

Build

$ cd "${GOMA_SRC}/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "${GOMA_SRC}/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

Coding style

Follow Google code style.

For C++11 features, we prefer to follow chromium guideline: http://chromium-cpp.appspot.com/

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "${GOMA_SRC}/client/out/Release"  (Replace ${GOMA_SRC} to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

More details are avairable in chromium's build instructions.

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=${GOMA_SRC}/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to Goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for Goma.

For general development

  1. Build Goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.
  • You can use autoninja in depot_tools instead of specifying gomacc manually.