Google Git
Sign in
chromium / infra / goma / server / refs/tags/v0.0.20
commit45c4249080db5d792aba847f01fe3e27cd602689[log] [tgz]
authorGoma Authors <noreply@google.com>Tue Jul 27 08:29:22 2021
committerFumitoshi Ukai <ukai@google.com>Fri Jul 30 02:03:08 2021
tree54bb7d354c7017d96f2076053eb222b6ecef40c1
parent87359e1228877e8308ed40e04e22d5b498d8cb61 [diff]
Import changes for goma server

  - d7f124ffb4908ec6e5b27302b2faa94176503ee7 roll google.golang.org/api v0.50.0 -> v0.51.0
  - 6f18dcd01eaec81dd54af4ab5c8147970b10c323 roll cloud.google.com/go v0.87.0 -> v0.88.0
  - e264308347bf4b11d70b1d17276eb5f066e7ee97 roll cloud.google.com/go/pubsub v1.12.2 -> v1.13.0
  - 326d7c47f8649e64cc46d250ccaa6aee83e3835d use `go install` instead of `go get` to install go binary
  - c71a80674fa5e638bf622a3d1b06c0020a4678dd Don't try to report proc stats on non-linux machines
  - ff35c6cf9d67078b327f7ab3b15a49fc445022a1 roll cloud.google.com/go v0.86.0 -> v0.87.0
  - 48a9c90c4257d5f3f822f59d0323fe01ab967fc8 Use go 1.16.6
  - 1343e157b06c7746bc4a835f987dd514879a7358 roll github.com/google/uuid v1.2.0 -> v1.3.0
  - bdbe2dca74afe03b6a9c350a0c130de831f70f19 roll cloud.google.com/go/pubsub v1.12.1 -> v1.12.2
  - 71e90254e81ce50c7c433561b35050bfaa7bcc55 retry pubsub receive for Unknown error
  - 3a67471c8cb419011496dd79681ec4b312425489 roll cloud.google.com/go/pubsub v1.12.0 -> v1.12.1
  - 7e1a1e2b51915b2f9598cbdb8366db8e076be2b1 roll cloud.google.com/go v0.85.0 -> v0.86.0
  - 2a8baa108948ba4d5265390310f9c34936e2c4a2 [clang] Support flags used for ML inliner training in Goma
  - 00f2cffb67168affe05037b607f7f0ab9ffe82e7 roll google.golang.org/grpc v1.38.0 -> v1.39.0
  - 716c7a0f50ac3cd3534aed613adb9dad2219a274 roll google.golang.org/api v0.49.0 -> v0.50.0
  - 915dc559f48d832e0d70775179b967562e44fd9b roll go.uber.org/zap v1.17.0 -> v1.18.1
  - 061cfe046f363f7847bf0590d19371281abad6ec roll cloud.google.com/go/storage v1.15.0 -> v1.16.0
  - 39b9ae53af6a319a611e9c900f1acdeddb2de8e7 roll cloud.google.com/go v0.84.0 -> v0.85.0
  - 26bae0dd6025c135e13bc4097db7189acc5bc9a3 roll cloud.google.com/go/pubsub v1.10.3 -> v1.12.0
  - de28b3273ebf5faee75bba36aee4526a9fec3504 roll cloud.google.com/go v0.82.0 -> v0.84.0
  - 5060bbab18d10b5759e7bd5a40324414680dd4ee unfreeze grpc
  - dd4a8746c81ac493749d054fd1b6bf27ab9faaef Roll google.golang.org/grpc v1.37.1 -> v1.38.0 again
  - 67fd12c3ab95a88847a0cb5841c6a7e459d1c058 Reland "Roll go.uber.org/zap v1.16.0 -> v1.17.0"
  - 8b2504b52fe12b829cbf15deeac3dbed19c980e1 Revert "Roll go.uber.org/zap v1.16.0 -> v1.17.0"
  - 9dd86b304d393615658ca79b8c816946e6c70a0c remove github.com/protobuf/proto dependency
  - 24db5ed9be03c440664f08d8e0f9bd851ab3c5ff remoteexec: add -imsvc from %INCLUDE% for win cross
  - fe3a06a3bb94c3a006fe23685c29ee7d995424d7 Roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - cd8eae9ebfe15ea021f7d9fcf29e720a7da48280 Roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 7a0cd8b2e7dbf3d2ec65fd35bdc0558dd286699b Roll github.com/gomodule/redigo v1.8.4 -> v1.8.5
  - 1f39367269a4c0e08d8645ff4ef13598d0c3406f use go 1.16.5
  - 3c57acf680a66983b4da1fd4b626c3307d7ff5c6 roll protoc from 3.17.1 to 3.17.2
  - e0cfe907d8a13f35c8bd6b0bbb6ec9be4148cdb1 autoroller: freeze support
  - 236c798e99839cd73fe31a78a45eef1c7874f795 Revert "roll google.golang.org/grpc v1.37.1 -> v1.38.0"
  - b36a7a1d41ccc3ad61296b0937eb666f70015f5e execlog: add service_account tag to metrics.
  - c75592e9e1ee38f90275ca7bb101c5a7c9a5bfbc roll protoc from 3.17.1 to 3.17.2
  - 13802fc4dd0fdf96261661ac9f6bced8cff8668f roll cloud.google.com/go v0.82.0 -> v0.83.0

GitOrigin-RevId: d7f124ffb4908ec6e5b27302b2faa94176503ee7
Change-Id: I3ba04507f7dd547bc6819ea6623d9192544f205e
49 files changed
tree: 54bb7d354c7017d96f2076053eb222b6ecef40c1
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o