Google Git
Sign in
chromium / infra / goma / server / v0.0.25
commit4beab4e04a5a9137d44d09ec5bc5301fff0bfc18[log] [tgz]
authorGoma Authors <noreply@google.com>Fri Apr 22 00:42:26 2022
committerFumitoshi Ukai <ukai@google.com>Fri Apr 22 05:09:12 2022
tree692f6bf26d82fa1c6220fed7ef59d02429fb5f93
parentf495539064692b2cda075fb14894e0230ffa1492 [diff]
Import changes for goma server

  - 5978f51377cbcfc4e98b8e8862cfffb07f65f39f remoteexec: handle -iframework
  - e7b4bb1b3c3326849e73e312fa3380c6c75e291d remoteexec: make clang-cl /std:c* relocatable
  - 3d12525761a708dfa5ad4e7472e70e0f6ca4f3ca remoteexec: handle -mllvm -limited-coverage-experimental=...
  - 85546eb700ddd6237ddf8aa2fee3b9e18be51325 copybara: target branch should be main
  - b9d9f8b22d52412fac85d91eaf35c7dc9b5198bc remoteexec: recognize -Xclang -no-opaque-pointers as relo...
  - ae1a962f061f23786d2ecb76ac7b610991615a8c roll google.golang.org/api v0.74.0 -> v0.75.0
  - 8b37a8d401d20af159355f5fe32adc9f70b17a1b roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 4718dc3b5c3dfe1f8f61ae63e4a6dba7e421adbe cloud.google.com/go/monitoring v1.4.0 -> v1.5.0
  - 45189f9cc41c54f29b8bc490e13d87fae4090273 roll cloud.google.com/go/compute v1.5.0 -> v1.6.0
  - 5f71b9a045d9ac25272c4c400935d589132d2fcd auth: distinguish no membership and lookup error
  - 821854cbe54e7f87df6b064b853f7cd8257e0b64 roll github.com/googleapis/gax-go/v2 v2.2.0 -> v2.3.0
  - d9f391a7619727fe9ff9830269a691a50c80baeb roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 309b66da4b33d01490e77ac816a466eb107f002d roll cloud.google.com/go/storage v1.21.0 -> v1.22.0
  - 14d0b5920e7eb8884f260420aa9a0ebf4cb43e77 roll cloud.google.com/go/pubsub v1.19.0 -> v1.20.0
  - a877dc821a2218f0e33c23535020ae4a97b13ce4 use go 1.18.1
  - 88d906aea9cf66c2ef5a81f13d78077a62b26092 update nsjail/config.proto at nsjail-3.1
  - dd3d817dc9340f175acf9d739416d7d8155f9b2d mixer: return 403 not 500 if no backend for the requestin...
  - 80031084dd7c292729e0bc4a0b746edd6ef36a8c roll google.golang.org/protobuf v1.27.1 -> v1.28.0
  - 8171f495f1cafa3bce369b76775222c84f7e24a5 roll google.golang.org/api v0.71.0 -> v0.74.0
  - 5ccb2eca70b44e98556e255cc61e1c5bbb5b3fdb roll github.com/googleapis/gax-go/v2 v2.1.1 -> v2.2.0
  - 76e3284dbb56e10898c64c75bc9da25d2506ec54 roll cloud.google.com/go/pubsub v1.18.0 -> v1.19.0
  - 29687f93bb68e4473b641f29cf27e22f37dac341 Add configurable TTL to the redis cache and have it be di...
  - 9992eafab60e275fc74a8cf515cfa61bdb2ffd03 Use go 1.18
  - bc727443db79b28886d6cdad7cf4dc1a4ccd4608 auth-server: make --service-account-json-dir optional
  - 450f1f445c4e48b62e53ae188f44c9d79523cc17 Use go 1.17.8
  - c062ce6ea60d310a460a5341fa6c89d070edb55b roll cloud.google.com/go/monitoring v1.3.0 -> v1.4.0

Bug: 1318246
GitOrigin-RevId: f66c875ba83b94c4e247cce63018daa15737001f
Change-Id: I915ff773443b9af54c00b833d6bbc96f6962f778
39 files changed
tree: 692f6bf26d82fa1c6220fed7ef59d02429fb5f93
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o