Import changes for goma server

  - 591e386a763fae3f3a60c301813893bbb8abc20f remoteexec: log "LLVM ERROR:" message.
  - effcd2537c04d8131e7576b3fb03c0cc437a36ad remove ubuntu:14.04 usage
  - 8c78a5741ff0f56850e869d6743449d323047951 remoteexec: don't use chan in inputFiles
  - 1d3b27da1cff4aabf405fe6667686b831758235c use remote sampler for all servers
  - 0c04862e43cb7349427ea01722c08ef8937fc3ce exec: fix rundir
  - 64d8edadefffb543416f5bae221e89176b1b702e remoteexec: remove select from inputFiles
  - 21bae73338440611b770253d2ff12d3fa332606a roll cloud.google.com/go v0.48.0 to v0.49.0
  - 699932015acf4ba046925152b6ca3923819502a0 remoteexec_proxy: fix typo in --whitelisted-users help
  - e62d02c577fec9137404990fd9ec3f08e529bd8f remoteexec_proxy: whitelist by domain
  - b53d06d8cb1e0b7c66d2bd13937de38fed2f73bd roll google.golang.org/api v0.13.0 to v0.14.0
  - 2a5d3e6801203a1f1ad0acafc4238455af928b69 roll cloud.google.com/go v0.47.0 to v0.48.0
  - 9334f8dd97824abb7a73fdd22beb08b218e3d8c9 roll cloud.google.com/go/storage v1.2.1 to v1.3.0
  - 776ddbdd0009a015e05d2c2b381a0af645e3ebe9 roll google.golang.org/grpc v1.25.0 to v1.25.1
  - 4d6c407935664f8c579e48bee7f915a2a8818a30 roll go.opencensus.io v0.22.1 to v0.22.2
  - 22684ee9174606073a2604e1d58fbf03ba3ad564 roll cloud.google.com/go/storage v1.2.0 to v1.2.1
  - 80b0367a9bd4bfed174ac2875ab80dd9c9452637 roll google.golang.org/grpc from v1.24.0 to v1.25.0
  - a255fa7f3426a30793ccc9d2df36b7c35f83ecd5 roll cloud.google.com/go/storage from v1.1.2 to v1.2.0
  - 4f3358ab35ab018916183f7acb322300ace3b8b8 use go 1.13.4
  - 4ef76414023039afe425b9e9d9d5e4ce5adda5e8 roll contrib.go.opencensus.io/exporter/stackdriver from v...
  - 6cbdea757c934b993b544d36ef278e016a5aabd1 roll google.golang.org/api from v0.11.0 to v0.13.0
  - 07beae01ae88661449921352442003a9669a9ddc frontend: trace-sampling-qps-limit from 1.0 to 0.2

Bug: 1026058
TBR=yyanagisawa@google.com, sque@google.com, yekuang@google.com
GitOrigin-RevId: 591e386a763fae3f3a60c301813893bbb8abc20f
Change-Id: Iccd28d5ad68f4cc46285ecb61c54df70d6eea997
14 files changed
tree: f41b1e84368317e821a17699f325b185d21d9430
  1. .gitignore
  2. CONTRIBUTING.md
  3. LICENSE
  4. README.md
  5. auth/
  6. backend/
  7. buildsetup.sh
  8. bytestreamio/
  9. cache/
  10. cipd_manifest.txt
  11. cipd_manifest.versions
  12. cmd/
  13. command/
  14. exec/
  15. execlog/
  16. file/
  17. frontend/
  18. fswatch/
  19. go.mod
  20. go.sum
  21. hash/
  22. httprpc/
  23. infra/
  24. log/
  25. profiler/
  26. proto/
  27. remoteexec/
  28. rpc/
  29. server/
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o