Google Git
Sign in
chromium / infra / goma / client / 324996360dd1e9248190fb2a231e124803224fd5
commit324996360dd1e9248190fb2a231e124803224fd5[log] [tgz]
authorGoma Authors <goma-dev@google.com>Wed Jun 06 04:36:04 2018
committerShinya Kawanaka <shinyak@google.com>Wed Jun 06 04:56:16 2018
treecdec97071f6fd3524a5eb2a25faa5c45f252f1f4
parent79ba6ccef68dd76b462d9e8ca25e7fca17d126a2 [diff]
Import changes for goma client

  - e990581a95c6cdad37b036ac1ef3d85b89d4e1d5 Fix typo
  - abddc12025be051b26ce1093381442e1e3980e0e Remove CompilerFlags::is_xxx
  - 640aaed30dae9fd4d774d555253c3fd29f39a70b refactoring: Split execreq normalizer tests
  - d6f15c2679014f84487ac07749e70f211f272586 clean up: remove unused functions from goma_ctl.
  - a8e508d90857bf556b6f810486fe750daf83791e Add local fallback test in simpletry
  - 2c24ac51890935e85b6cb4c5b722cab9ea89598c Revert "compile error if necessary default blacklist file...
  - 683d8f3bcb62adcfa7859b8a4a6b7cf40a7d1fee refactoring: Split CompilerType and CompilerTypeSpecific
  - 313ba87e0acd1fec78dddbadf3d3596e56500b3c Check CreateDir return value
  - 7afb0c757c5fcef0824b1aaddbcad9b185ddba42 "dump exec req" dumps ExecReq args,env,expanded_args actu...
  - df3629d0e28314d6f0b8a06e0b5af52f23f37bc8 refactoring: rename FixPath to FixNonSystemPath
  - 9bf941a51f29abafb7a75d8906e33e6d194937e7 Update boringssl to 81a6f6d8
  - 9b99423ae6f4ea9eabcbd32bee19fc75969ab3bb refactoring: Introduce ConfigurableExecReqNormalizer
  - 3f2196540f3cccd26d6b2c75464db8fc9e079e35 ExecReqNormalizer Keeps args for cl.exe and clang-cl.exe
  - 9eacd628dbf3ed774f7885f7a376f8ce933ae393 follow-up: cleanup code and add comment
  - 3d246e7afe962a492bdd82b0a3d097accc66cdf3 refactoring: Extract normalizer procedure as function
  - 5fa6383452d7addf6fdb9ccf196531bea7e4eb37 Don't normalize args in javac
  - 1f45b70a504a9d2ac5ba5615f7566d0965a8a85f cpp_include_processor: fix for clang-cl on linux
  - f4a485f891162016fd46031180ce07518c617c37 Apply clang-tidy readability-else-after-return
  - e6bcfb549545e94b44c2a53e27b9024b39d12af7 Split java files from client/ to client/java
  - d405c56ef27a676f5efb19ce2ff096104f18b37c Enable detach test for spawner_posix
  - 7b6bc9c3543302e2724674d0c3d71c57d2525f6e refactoring: Move IsXXXCommand to *_flags
  - 3b39b89d449b1b355fd58b0c1895f14cc45c1d2f MemoryLog has info only if SEND_USER_INFO is true.
  - d85b7d0ce8107a92241fc45c3b717d0c2e1f31d6 Add test for subprocess kill to subprocess_task_unittest
  - de092852942c9f654a2270aa3b5087773a8234c2 Remove compiler-specific code from compiler flags
  - 1e2a0f589715d98390282d1f7bc77ffe9709a63a change default for sending privacy info.
  - d7b1e35a8c26d2e6c9d6270da82af9287c2173a2 Fix SpawnerPosix::Kill
  - 13d3a76ea9c81ed0b40baed6f5b1feaa23af5312 show compiler_proxy_id_prefix on status page
  - 2ace1278638a9c9e4daf20764ca2d231d65dba01 Add diff viewer of ExecReq normalizer
  - 23494173a855c1229a71093e8944b7c42ce8cdf4 Add spawner_posix_unittest
  - 3bcfc52ebfd97cd3cc905069f0a46c4f43e01b89 refactoring: Use CompilerType in client/
  - 2a105644a399374adf9e7905cc9b092ae95fad50 Update boringssl to f86693df
  - 3c880fbcfbf2d64feaa5294ef456c66b72b7ce54 Enable lld on Win by default
  - f36c1adac9adb210370d824d311782f932054900 Set cache hit for local output cache
  - 3bd900f43cf2a4098dc4c5dd0688f9d4659cca4d refactoring: Introduce CompilerType
  - 83079f137ef3259d4b9fd7a8264908a94b605b08 goma_ctl.py: improve goma_ctl.py stat performance.
  - eb6f486fc4aa86d26d88c3ec648c703d9f743654 refactoring: Move gcc/clang test to gcc_flags_unittest
  - e1a4926352b83d00032a907de5c90db5d4064c33 refactoring: Move cl.exe/clang-cl.exe test to vc_flags_un...
  - de4c77bc43e5485a6e0942de78c1d5ec5eed630a Correct deps (atomic_stats_counter_unittest and callback_...
  - 6f7f0ba4df56c8de01953e5385e622983629c9a3 refactoring: Move gcc-only functions to gcc_flags
  - fb0e4654f624becf687fe3a879d2b994d71638e9 refactoring: Split compiler flags
  - c8e68e2971e4d49187c624492536f89b21a82ed3 Print tmpdir message to stderr
  - d35f5ca5fabc3f7a8ad1053e67b4223ea0951020 refactoring: Adopt new interface of CreateDir
  - 34f0be8f6b6836b73d9ae66a60062aea7878c97f Increase compiler info pool for faster build time
  - 138c24d1ad00f7828058835b46fcb7a355fd436e refactoring: move IsDirectory to filesystem
  - 8570faa6640b47c800c144aa67c45fcbcf8aad6e compile error if necessary default blacklist files are mi...
  - d744ff7e5a0b599977bb0d3b62d674e507e54892 refactoring: Adopt new interface of file::RecursivelyDelete
  - 1bdf4e7f9372304506faa39c32dc029621839412 Use clang-cl by default
  - f1d62b2e70746ff13412674d881e8952787fbaea bug fix: make -fno-sanitize-blacklist work.
  - eafb8fcf94ebb3cb22f09a0655b9c4109cecdc2a Use tempfile when gomacc does not exist in goma_ctl.py
  - 7c37d0e8fdcbd253dfe7e791b92fa767c14e759c clean-up: Run iwyu in benchmark/
  - a24e4454b198ccf72cb1ed4bebe1f8131809fd6f clean-up: Run iwyu in base/
  - 05f3c4fbac5a073b9c6c40ad34dddce95aa7bd27 Update boringssl to 9b2c6a93
  - 5028053d62a4f46f92c4540303c7dda07811424d Add -w to compiler_info_flags
  - a66dec481853985c9e20bf0117d4055788d392f4 Fix for compiler info failure due to warning output
  - 86e2ce1915fe5578cf7389ebed5da8fecb88a29c clean up: do EXPECT_DEATH only if GTEST_HAS_DEATH_TEST is...
  - 8c0a06e90ab8fbeae23667a986f41d5dea7aa96e Introduce ThinLTOImportProcessor
  - 7dabd702ceb66a4932b4958343c6b236aca5681e Set cache_hit when we use find from LocalOutputCache
  - 482a24f6305bc10257d6b828c680c00ba5d6cb1b Re-enable goma_ctl_test for silent update.
  - 2f7854a269e594e5da3b4a712272898ed4c6cd61 use proper default for open source client.
  - 53d039f966965c0ace3ad53e13e26dc7c52729ca Consider #line case when getting compiler info
  - 1e7f79db8bd48ef46de99ccda9fe78ddf2681895 goma_auth: fix oauth2 config
  - cce21dd38cfbcb1fe602582816fe50da13490440 Remove glog from linker related tests
  - 8ed8d3e27d1fbb548cfd189e51771ac8137ba92a improve goma_ctl.py ensure_start performance.
  - 1a2239dea4046f3fd4285c2a9665201e5680ff4f Split linker_input_processor_lib from compiler_proxy_lib
  - 0c8100e54b2bf43692c6c1a72901c441c6507fb1 Show local delay ms in status page
  - 422ab9ecc59e96fc59c08a90399bdd54ff2ab168 remove oauth2_cloud support
  - 8e2ea1c1c09bca7edc0fd09f68891bd4da5e7c38 Rename kBufSize to kNetworkBufSize
  - 3091531c3c09cb1c7fadd7eadf28addd89b26956 Remove 'creating crash dump dir' message
  - 50a47dfdc0e73a48cc6fce0bfa497bf70634e7ea Use _beginthreadex instead of CreateThread
  - 99f347cc9b3c2c2186e09c8efb736f896edc5f41 Rename IncludeProcessor to CppIncludeProcessor
  - 434b6f273ebe44a624264594821f6f81fa7357df Update boringssl to 69271b5d
  - ff25a1db48acb396f5f9ca23ded17139edee22c8 Introduce FlatSet
  - 7b477600253c30de059ba35ae2f6055dc4b10c3e Use C++ code to get GomaTmpDir instead of python code.
  - ec69a36cdcfc75ec058e0143a9fcbfe4983fcb45 oauth2_token: don't set / for url_path_prefix
  - a0c7e391b1a24563b3540c3407c2e2aebbeb5ec4 Split sha256_hash_cache_lib from compiler_proxy_lib
  - f99f3dcbb83b02e30072adb50e3207567e391dbe Roll abseil to f88b4e9cd876905ee694cee1720095330e0bd20f
  - 9fcfae8133811e0969e40dce8e2af3de8c7bcd64 Do not log inside lock
  - 59b8fdd07b354d89b6f6ab75204fb4f8e0c72113 Roll abseil to bd5a2bba8f4238db2e06c99ac808fc8327dec2a0
  - 21529c6556151a95dc7799743408ee5b02106dbc Use string_view as CppMacroEnv key
  - dd26f65fe60f0202106e0fe8573ce06bdd5f94d0 Reverse condition for error message join
  - d35359b0f04bd4cf285cdbc3d963c6c4853fd2c2 not normalize if -fcoverage-mapping is used with clang.
  - 5790074dd18369800a602c7affe347797c005060 Update boringssl to e30fac63
  - 43e169903e47df00a440c993d16287731abec241 Reduce allocation in JoinPathRespectAbsolute
  - 7ac1789c40992dad99797230f3f0f76616d2abe0 Support new location of clang implicit blacklist files
  - f2698c19cb259621e9595563d44c6fcbbaf312bb Increase Max subprocs for powerful machine
  - ab6c8e5865f4b972e6d7fa232be298f9a7a00825 Update gn to revision 554932
  - 6bf3abd0037de656a282100953438cb55fe027ac Update boringssl to c596415e
  - 514505fed4189d0ca179bfa867b5b1fee77aca08 Apply modernize-pass-by-value
  - 67bfc05bba35427ef9696128c88c49fed5289d9c buildfix: Goma Win64 GN builder
  - 3832098776f07bbb3fae864301971a7d790ca57c Apply modernize-make-unique
  - 482d008680fd21c646d1bbff8a503957c8b07a23 Extract CppMacroEnv from CppParser
  - 44d29e8f98046cf8666150e7e7c97ad6c092045b PyObjC is used with vpython.
  - 39046061a5ba1e1126c66fa4b42c13d13d464191 Pre-calculate include guard
  - b65a3ff2e626c04508b3dad94c13e948d078a3d3 Update boringssl to 48b276db
  - b67a5f6bd529ee41b2c6e1bbd29e8e3116bafa0d Split jarfile_parser from compiler_proxy
  - 87d035e5bf35bb2621c00a270f8eda4cdbfcf30b buildfix: clang-cl.exe does not support /ZI
  - 14ea4d5364b457293af98ea2b9d1a246c404c6dd Split arfile_lib from compiler_proxy_lib
  - 1e97099db095a50ad410735feff3239ad55af3f9 Split rand_util_lib from compiler_proxy_lib
  - 73244a2eabee81e4fda196c570b18b4a77457738 Use const for SharedCppDirectives
  - f16ae5bae54d9526b4e13bbd85d9702d5cf09fdd Enable libcxx on Linux by default
  (And 141 more changes)

GitOrigin-RevId: e990581a95c6cdad37b036ac1ef3d85b89d4e1d5
Change-Id: I797397fe6f533a08673c60c5918c26a97e3669a9
362 files changed
tree: cdec97071f6fd3524a5eb2a25faa5c45f252f1f4
  1. base/
  2. benchmark/
  3. build/
  4. buildtools/
  5. client/
  6. infra/
  7. lib/
  8. test/
  9. testing/
  10. third_party/
  11. tools/
  12. .clang-format
  13. .gitignore
  14. .gn
  15. BUILD.gn
  16. codereview.settings
  17. CONTRIBUTING.md
  18. DEPS
  19. LICENSE
  20. PRESUBMIT.py
  21. README.md
README.md

Goma

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

NOTE: currently the goma backend is not available for non googlers. We're working so that chromium developers can use it. Stay tuned.

How goma works

Goma hooks a compile request, and sends it to a backend compile server. If you have plenty of backend servers, a lot of compile can be processed in parallel, for example, -j100, -j500 or -j1000.

Also, the goma backend caches the compile result. If the same compile request comes, the cached result is returned from the goma cache server.

How to build

goma client can be built on Linux, Mac, and Win.

Install dependencies

  1. Install depot_tools.
  2. Install dependencies.

On debian or ubuntu,

$ sudo apt-get install libssl-dev libc6-dev-i386

On Mac, install Xcode.

On Windows, install Visual Studio 2017. Community edition is OK.

Checkout source

$ gclient config https://chromium.googlesource.com/infra/goma/client
$ gclient sync
$ cd client

We assume goma is checked out to $GOMA_SRC.

Build

$ cd "$GOMA_SRC/client"
$ gclient sync
$ gn gen --args='is_debug=false' out/Release
$ ninja -C out/Release

Several important gn args

The build option can be modified with gn args.

is_debug=true/false
  Do debug build if true.
dcheck_always_on=true/false
  Enable DCHECK always (even in release build).
is_asan=true/false
  Use ASan build (with clang).
use_link_time_optimization=true/false
  Currently working only on Win. If true, /LTCG is enable.
use_lld=true/false
  Use lld for link (it will be fast)

Run unittest

$ cd "$GOMA_SRC/client"
$ ./build/run_unittest.py --target=Release --build-dir=out

How to use

For Chromium/Android development

Goma can be integrated with Chromium/Android development easily.

  1. Build goma client
  2. Start compiler_proxy
$ "$GOMA_SRC/client/out/Release/goma_ctl.py" start

For Chromium

In Chromium src, specify the following args in gn args

use_goma = true
goma_dir = "$GOMA_SRC/client/out/Release"  (Replace $GOMA_SRC to your checkout)

Then build like the following:

$ cd /path/to/chromium/src/out/Release
$ ninja -j100 chrome

For Android

$ source build/envsetup.sh
$ lunch aosp_arm-eng
$ GOMA_DIR=$GOMA_SRC/client/out/Release USE_GOMA=true make -j4

Here, -j4 is not related to goma parallelism. Android internally sets -j500 (or -j with NINJA_REMOTE_NUM_JOBS environment variable) for goma.

For general development

  1. Build goma client
  2. Start compiler_proxy
$ ./goma_ctl.py ensure_start
  1. Change your build script so that gomacc is prepended to compiler command. For example:
$ gomacc clang++ -c foo.cc
  1. Build your product with make -j100, ninja -j100 or larger -j. Check http://localhost:8080 to see compiler_proxy is actually working.

Tips

  • You can use autoninja in depot_tools instead of specifying gomacc manually.