Import changes for goma server

  - 39fe62dc1f2cc5b479b9a26556a9f428fbf345ec fix inputroot_test.go
  - 5efd6aea02d3a998774bee1c1c30183b090b690c roll github.com/google/uuid v1.1.1 to v1.1.2
  - 2a71dc4f4e03fa731c42ab1056b910f7ba0cc423 remoteexec: check system include paths for input root
  - 82a99479365d5a41f3f87095cc905a9fdc3a632a roll cloud.google.com/go/storage v1.10.0 to v1.11.0
  - 5a0bb045bfdd29a5d198805f88296b6dafe21a1b roll cloud.google.com/go v0.64.0 to v0.65.0
  - 62bdce74caafaa03ca03636c88125cb1e744cba2 Remove runsc=available label.
  - a57ea3781734b97ccae561c8586a9f1e4b0806cd roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 7184189e09fb9087a150bd8213e52edb6e958367 roll google.golang.org/grpc v1.31.0 to v1.31.1
  - 25e1a0e12142e025990e328e9652b23ba00caefb raise reportingInterval from 1m(60s) to 90s
  - 3d2aa60317d577037e9e331eaa7ad6bfa4e6a969 roll github.com/google/go-cmp v0.5.1 to v0.5.2
  - 676f94f8eff069a2991bf7a59d4263c7097ac17c digest: report cache-set fail as op=cache-set-fail
  - f24d165c7ab9321d2fbea708cee3c29e9fc14d8f remoteexec: don't wait semaphore if ctx is canceled
  - 61da6cb090ac8bfcac5274fbb0aaf4b43f5159dc remoteexec: narrow FileLookupSema
  - 66ee7d976ed7aa9b28d59990dc393c8b6efabdd3 remoteexec: span inside newInputTree
  - 00c4e591ec15a14f0e444fb8010485834cab155c roll cloud.google.com/go v0.63.0 to v0.64.0
  - ac65bf2aa9c197d9aef1f546b9cafa5ec50a885e use protoc 3.13.0
  - 7620cb78da0aced2ffe9525f25dac189a0c96187 roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 4ae8f1fbaabb0422209237edcd5a93ab8b5b16fe roll github.com/grpc-ecosystem/go-grpc-middleware v1.2.0 ...
  - d472650d02e679db7613df5b1639bad5a7377a4b Use longer timeout values for AC checks and CAS access.
  - 4d6e7e34afa9387d9fd2bdb6e9a7268a40143c14 use go 1.15
  - ecec79f44713eb11392408f61babc95e73cc632d roll cloud.google.com/go v0.62.0 to v0.63.0
  - de7bae43af8700718d7eb0716722c879935d7fe9 use go 1.14.7
  - bf399c36232223eb0057620466bafb678297e456 roll cloud.google.com/go v0.61.0 to v0.62.0
  - 2456330b34805cac9600212a87264ccd29792fac roll google.golang.org/grpc v1.30.0 to v1.31.0

GitOrigin-RevId: 39fe62dc1f2cc5b479b9a26556a9f428fbf345ec
Change-Id: I567a922e0c49886bf6a519bed1dd03b26d756c2f
TBR=yyanagisawa@google.com, yekuang@google.com
33 files changed
tree: d3e1ab684d802edc30ff4ad7b8bfd990e24ecb85
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o