Google Git
Sign in
chromium / infra / goma / server / refs/tags/v0.0.22
commit3b2193c76cd9597e11bd27b63245b1f4ded7c25a[log] [tgz]
authorGoma Authors <noreply@google.com>Mon Sep 27 01:15:08 2021
committerFumitoshi Ukai <ukai@google.com>Tue Sep 28 04:24:53 2021
tree3b407f82dbe4770651d2e6b204c08e62dcf32629
parentaa75748f3557a010ccb0332d0e70d3190c22507f [diff]
Import changes for goma server

  - 3393dcd6fb83d8bbfc68d343777c58cc68ac77cf roll google.golang.org/grpc v1.40.0 -> v1.41.0
  - 1f0c118120f08586f1f5978971a737ccf1bae2cb roll cloud.google.com/go/trace v0.1.0 -> v1.0.0
  - 00151d910920060fe549aece57ba79d444f391e7 roll cloud.google.com/go/monitoring v0.3.0 -> v1.0.0
  - 9817519ffd55b7cc5a319c0031a1988adbaf06a4 roll cloud.google.com/go/monitoring v0.2.0 -> v0.3.0
  - 1a8ec6e9650c3b3e12451000d31a0383e65be269 roll cloud.google.com/go v0.94.1 -> v0.95.0
  - e6206192f58a409dd91805ffc2a0e02220f88514 Update OWNERS.
  - 96cf07f53c575569431261d7a638746e10805e67 roll protoc 3.17.3 -> 3.18.0
  - db055dd66b95bba3f38a860ec2d0db42866a6b08 roll google.golang.org/api v0.56.0 -> v0.57.0
  - c661e7752b796d92c75b476b3ff286871c257b5a roll github.com/googleapis/gax-go/v2 v2.1.0 -> v2.1.1
  - 94aa7c0cf8a15a4ec92121cb822feffe661e7a9f roll cloud.google.com/go/monitoring v0.1.0 -> v0.2.0
  - 97448ca2a23714a145bb11b1162228f5eb09e987 Use go 1.17.1
  - c3332e166ddd8e854947c7e2ed856dec4ee0b748 use the same seccomp filter for chroot
  - 6c63566fe751e95b3392e5bb0d2db865ad6db2ae roll cloud.google.com/go/pubsub v1.16.0 -> v1.17.0
  - 4453f3809bb8b8db1c146b59342f974e845e636c roll go.uber.org/zap v1.19.0 -> v1.19.1
  - fc63e3f4061eb26f5d4caec643af183af5d05b3c roll google.golang.org/api v0.55.0 -> v0.56.0
  - 61d2ec5fca5747ff49a9e2dea6a3ac4e9467d218 roll cloud.google.com/go/storage v1.16.0 -> v1.16.1
  - 4e75144defdf532abd33007c1143f951cbf63eb1 roll cloud.google.com/go v0.93.3 -> v0.94.1

GitOrigin-RevId: 3393dcd6fb83d8bbfc68d343777c58cc68ac77cf
Change-Id: Id3b4a2533698cd9dfc4d2affbbf428dc090afe90
26 files changed
tree: 3b407f82dbe4770651d2e6b204c08e62dcf32629
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o