Google Git
Sign in
chromium / infra / goma / server / refs/tags/v0.0.15
commit1933b8e4bf1fbc16c62fbc310bad5bbe14697a91[log] [tgz]
authorGoma Authors <noreply@google.com>Fri Jul 31 03:39:40 2020
committerFumitoshi Ukai <ukai@google.com>Fri Jul 31 04:36:01 2020
tree93b69918267394bd2a620551a68fc6ec0873d451
parenteb5e1bd07bcf2c4d4f776ff7bb24192684cbd629 [diff]
Import changes for goma server

  - f94fd59331f0a08e04fadd87f57e45645a45619e Remove oppressive language
  - 705de0b783ac80773b599e9d98bf9615842014a6 rootRel ignores case for Windows path.
  - d5a03991dd3066e4c4d9cee750444256ffaebf5a remoteexec: add os-family and docker-runtime tag for rbe ...
  - 4fb8d0bf2803331e7c96543e22f7e8dcd108e88e remoteexec: export latency metrics
  - d6190334f40f171b612626002bf451a1b11a368f remoteexec: release blobs in gomaInputSource
  - 080a7f2aa574a36d59f286dd9fd1b427d0909512 Ignore case to detect common inputs on Windows.
  - 3cbca9e3a7baf68d3aca935d4af9e9f485451e2b roll cloud.google.com/go/pubsub v1.6.0 to v1.6.1
  - db2a9f565a4a2760941f01c1e4f2ef2f2f61b97c remove workaround for *.thinlto.bc
  - 78812dc5c8119dd5ee4e670a411f766136414ccf Fix /showIncludes arg support for clang-cl
  - 4d8ca76aa627be90da5764c20a8c078c18f7601f roll cloud.google.com/go/pubsub v1.5.0 to v1.6.0
  - 7a48e01a4a1924cd98c92d6af49f148a56cb8403 remoteexec: don't make FILE_META for just 2MiB.
  - 86cdb70a44908700510ada0c93d244de3c746673 remoteexec: set short timeout in each stage in Exec request
  - 8775e082bc88b4901318590313d0d6557aa09a51 fix digest cache metrics
  - 91b6886c0e6ea18306b37e7943d13355478bfc77 roll github.com/google/go-cmp v0.5.0 to v0.5.1
  - d6ca1f1339907f32addf4bcb6a4498f2eb1c3436 digest: need to export fileExt tag in view
  - 8e9279c6c7c325625e91f049bed6c9baa8d0b886 digest: count diget cache usage per file ext.
  - 63373c5fdd1cc0a6bdad3f6d16e5ddd32397d7fc Set exec call max retry count
  - 206814d1c610c101c29a1657528cce3f3babd050 remoteexec: set short timeout for cas download
  - ff4661038d624605a3260328bb0a72b5a75ce0e0 remoteexec: limit buffer for gomaInputReader
  - 0e8a5925e4a191114fcaa95723506dc433cbd207 use go 1.14.6
  - 35f990fccecc3afa201f791cab5ab9b5c242293e Limit the number of files reported as missing inputs.
  - db6cfd62b40b88516df9db98305d1d99c017050c make more flags relocatable
  - f43b4e6a64dfec4e7ff2ae275952706dcad5d5de execlog: raise max message size for LogService
  - ff78635fcd063f2e9adb8ff4b1d729d0cee84b8b Limit max retries for RESOURCE_EXHAUSTED to 5
  - 062970a35a0d7b1e2cad7edd89f040bbf862f992 handle -mllvm -basic-aa-recphi=0
  - 5f5f4a475d5a3100265293b37a33f42e776bd0d7 handle -include<file> --include<file> --include=<arg>
  - a8bf0d17646a86d0f30e6b3b69023e4e7921cb26 retry slower for RESOURCE_EXHAUSTED
  - faef9c8f8fa3b843a6389bd377ce0128351b786f use go 1.14.5
  - 31215806700785c31df2aa612a275c5e69749d07 make "-mllvm", "-instcombine-*" relocatable.
  - 8d49153bf56544fb9a2fb04c17e24985e713ece4 roll cloud.google.com/go v0.60.0 to v0.61.0
  - c0138a6eeced9a322914565982748514cee72388 raise max digest cache entries to 2M.
  - 9d3d9f28622f3ad288f2e58b33ac5b4ae6a2b3d4 digest: measure digest cache stats
  - 06b1fb83893d4c1a0c68b4acdfd95c55da321cc4 auth: log group id for auth error
  - 95c3a41098c13a08b8c490ec4ecc040f0879d501 remoteexec: fix logging
  - 7d92046c6989aa78a69c8248f2ed1053b90e13e3 roll google.golang.org/api v0.28.0 to v0.29.0
  - 7ce952609abb49b29be5f6dd90659314eeeea142 roll cloud.google.com/go/pubsub v1.4.0 to v1.5.0
  - 8d8edd431bc10d3098174a942479501bc272f9a4 digest cache: limit max entries
  - 3947635af7fcb3f6a14351b5b301da115a9d8ee7 workaround for *thinlto.bc
  - c5744e4a46dfa5426580cc62c0292ff72c3c7b19 Add 'idirafter' to gccRelocatableReq
  - 46e46a33b06bd783e85286fa7c810350f83b0eaa start more slowly and backing off more quickly for RESOUR...
  - 116eac06fc66757d5f03cd30a53551f9e0e6ecff roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 7da28764733edc653a402ef249f0b4add7fe2216 roll cloud.google.com/go v0.59.0 to v0.60.0

TBR=yyanagisawa@google.com, yekuang@google.com
GitOrigin-RevId: f94fd59331f0a08e04fadd87f57e45645a45619e
Change-Id: I743e4ff7e8bce8e40ab888863a40cc8153596313
33 files changed
tree: 93b69918267394bd2a620551a68fc6ec0873d451
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o