Google Git
Sign in
chromium / infra / goma / server / v0.0.19
commit43e598370433892f95f6b53af8cbdd59ecf6d852[log] [tgz]
authorGoma Authors <noreply@google.com>Mon May 31 00:51:39 2021
committerFumitoshi Ukai <ukai@google.com>Wed Jun 02 05:25:11 2021
tree046ddc5f2e5d213bbcb8b0fc98b19ad191b3e1f7
parent74c815e2a1bd351d547d319b83f76b6fbc50771f [diff]
Import changes for goma server

  - 8d6478c2fc6e025ffd2af9898c5edf0a2b3b5a90 roll cloud.google.com/go/pubsub v1.10.3 -> v1.11.0
  - c388208ca756f429954dd9b85838e1d2654604ae Roll go.uber.org/zap v1.16.0 -> v1.17.0
  - 1feb0c4ded2a47b1096cad05c5122dde3cee510e Roll github.com/google/go-cmp v0.5.5 -> v0.5.6
  - dbd4fb005259ecb134d803d805a34c3f0fe3bced roll protoc from 3.17.0 to 3.17.1
  - 2e5e3bab85b35dbb77240694aa2c5a9889886b1b roll google.golang.org/grpc v1.37.1 -> v1.38.0
  - b94ec430c17d60d99fc8ef478b1115ab96ed18d6 roll google.golang.org/api v0.46.0 -> v0.47.0
  - fc062745a3b7080bd97c275bb808c8b0507525a5 roll cloud.google.com/go v0.81.0 -> v0.82.0
  - 2384007fb8656d1573fa83d4eed6d74283d5be80 use infra/3pp/tools/protoc
  - 7266ae032ff7da3a9bda646b2bd19eb67d779f7e Fix OWNERS
  - 4d366a94f22ec95ae2d1c4c40cd7939bb0ef1df1 roll protoc from 3.15.6 to 3.17.0
  - 688c8d5a0e6863e0a0487954533c50743340099a roll google.golang.org/grpc v1.37.0 -> v1.37.1
  - 18f3cfcf1e87a82f9cd843c95ec3f0cab02d8174 roll google.golang.org/api v0.45.0 -> v0.46.0
  - f16e9a25534ac9c5769f0318208659db9fcb2377 roll github.com/grpc-ecosystem/go-grpc-middleware v1.2.2 ...
  - 5a143b4ca9e35dda21575f6b8f97c3c64bf7e228 roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 74bf168f93e9215c313fd555a98381975f53fe93 roll cloud.google.com/go/storage v1.14.0 -> v1.15.0
  - 1083406b65cdaea20bdec749ce88afa31b286488 roll cloud.google.com/go/pubsub v1.10.1 -> v1.10.3
  - bb90971dd42abf257bf8e58a02a9b9d0ae8d4dbb roll cloud.google.com/go v0.80.0 -> v0.81.0
  - 06459eb3c26c0be1da5ab82cc92db44191d5b8c2 use go 1.16.4
  - 163e8baa858ec1db3956bdd5e28762c592076b0c fix llvm windows cross failure
  - 698173b09713cfdae057082757f5bee32b5fb947 fix windows cross for skia
  - 2fb1292098232c2424df1c34c7f4b6454497cadc copybara: don't copy *.pb.go

GitOrigin-RevId: 8d6478c2fc6e025ffd2af9898c5edf0a2b3b5a90
Change-Id: I47de1d67bba93597a03b9d0d8705774cd3b21d15
29 files changed
tree: 046ddc5f2e5d213bbcb8b0fc98b19ad191b3e1f7
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o