Import changes for goma server

  - ea652f7ec375df95e58c0253031dcd295fb1e5b6 fix copybara for public release
  - e11c8c928207920ff15fec47124c56e70f4c69e8 force to load configs when retrying
  - 28fcc0c2a7701320ab35a0ce446720af14b0730f command: Next uses ctx, and return ErrWatcherClosed when ...
  - b7b4fe5f6962f4724994639d6de3f23346ef95bf mitigate missing compiler.
  - a655123a3a6f56f6a1382d5aaf8515835f0a76cb roll cloud.google.com/go/pubsub v1.15.0 -> v1.16.0
  - d2ab42543f983c2d1e2b54b92118279a77b0801c roll github.com/fsnotify/fsnotify 1.5.0 -> 1.5.1
  - 33055aaba37c7e9309e84c3b435a8f9b526a7270 Use go 1.17
  - bef24828dd4e48c2e2886756463666a58872099c roll github.com/fsnotify/fsnotify v1.4.9 -> v1.5.0
  - 5a98c02b6b320161d1b8d4f8d328bc52704b2f61 roll cloud.google.com/go v0.91.1 -> v0.93.3
  - 6f1377417c1692d6045c02d6d4820c20c955b2a7 fix nsjail seccomp filter for cros glibc update.
  - c6bea44b2ead61f86d3c5afb642fa8bbd39c996e roll google.golang.org/grpc v1.39.1 -> v1.40.0
  - 1996177b91433409e15eb1903722c4f1bc758a65 roll cloud.google.com/go/pubsub v1.14.0 -> v1.15.0
  - 3bc5010f0175fb59e4b868814f451c2387898066 roll cloud.google.com/go v0.90.0 -> v0.91.1
  - 3bb8d7760d2642c70b0a2fda074d9ab42a267481 roll google.golang.org/grpc v1.39.0 -> v1.39.1
  - 3bcfbd6d24ca3113209608bd62cfb49a7d267543 roll go.uber.org/zap v1.18.1 -> v1.19.0
  - 88b038ddaa9d960e19f4001558a434aa03856f38 roll cloud.google.com/go/pubsub v1.13.0 -> v1.14.0
  - aac8393a118158a6064156a425b1d578f97ed63c roll cloud.google.com/go v0.89.0 -> v0.90.0
  - 85bdd8faac81be7ffe926dfa4b99257ff37000d7 roll google.golang.org/api v0.51.0 -> v0.52.0
  - a65c6dad61423b5d064aee1a1f5ad5d6495d980e roll cloud.google.com/go v0.88.0 -> v0.89.0

GitOrigin-RevId: ea652f7ec375df95e58c0253031dcd295fb1e5b6
Change-Id: I29d051072bb168d58edb0203655a97f9527c3f27
12 files changed
tree: 6d43d4eaf7abfb63cfb08a1ad7291cf860c98998
  1. .gitallowed
  2. .gitignore
  3. CONTRIBUTING.md
  4. LICENSE
  5. OWNERS
  6. README.md
  7. auth/
  8. backend/
  9. buildsetup.sh
  10. bytestreamio/
  11. cache/
  12. cipd_manifest.txt
  13. cipd_manifest.versions
  14. cmd/
  15. command/
  16. exec/
  17. execlog/
  18. file/
  19. frontend/
  20. fswatch/
  21. go.mod
  22. go.sum
  23. hash/
  24. httprpc/
  25. infra/
  26. log/
  27. profiler/
  28. proto/
  29. remoteexec/
  30. rpc/
  31. server/
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o