Google Git
Sign in
chromium / infra / goma / server / v0.0.23
commit894fdea9bc412db820c175bebee725a25b002305[log] [tgz]
authorGoma Authors <noreply@google.com>Tue Oct 26 04:11:11 2021
committerFumitoshi Ukai <ukai@google.com>Wed Oct 27 23:52:33 2021
tree998463837330e607d33f1c252701bdafc305c7a5
parent3b2193c76cd9597e11bd27b63245b1f4ded7c25a [diff]
Import changes for goma server

  - 9fc77617f4f068b1363b5566a92fa48b905cfd9c Use go 1.17.2
  - 5c027676fafe917194ee2e1415f276a5f87a51c8 roll cloud.google.com/go/pubsub v1.17.0 -> v1.17.1
  - 153df272ab6d3213e55fcb43213ee521cb80d4f8 roll protoc 3.18.1 -> 3.19.0
  - e102254e4a8cdd590bcd3ff2d02ffff3fa8d4c0d roll google.golang.org/api v0.58.0 -> v0.59.0
  - dfc33091fb5456c53bb9424cbc4b362c71110426 roll cloud.google.com/go/monitoring v1.0.0 -> v1.1.0
  - 3116344eab64058a9fa0543cbc70757086c63a5e roll cloud.google.com/go/storage v1.18.1 -> v1.18.2
  - a4031eb99dcde3a09463b707117f762f469ff623 roll cloud.google.com/go/profiler v0.1.0 -> v0.1.1
  - 33a4815bcf26d60122ce6e1665347733a15af495 roll cloud.google.com/go/storage v1.18.0 -> v1.18.1
  - b50fc9b4acde3e1d0651ea8c8e4aac37399c75d0 roll cloud.google.com/go/storage v1.17.0 -> v1.18.0
  - ca98bbe13655ef55d0a197a4334e440b6c42fe15 exec_server: finish configServer ListenAndServe when shut...
  - df034053cfac7a0984ce49dc0695b22aacbc0f75 roll protoc 3.18.0 -> 3.18.1
  - 9f1e71f2ba2165303f8248ca805489df86fa3c51 roll google.golang.org/api v0.57.0 -> v0.58.0
  - 0da5f343f0df7bcfe6cb44f606560ef02665e002 roll cloud.google.com/go/storage v1.16.1 -> v1.17.0
  - 5e3f28c9ae7b7731bc9bdfdeb542e28ccfacffc1 roll cloud.google.com/go v0.95.0 -> v0.97.0

GitOrigin-RevId: 9fc77617f4f068b1363b5566a92fa48b905cfd9c
Change-Id: If704de1f40a4a0e4b86eaf630ac1c58a77e1ab01
25 files changed
tree: 998463837330e607d33f1c252701bdafc305c7a5
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o