Google Git
Sign in
chromium / infra / goma / server / refs/tags/v0.0.13
commitb7d2fa5a0a2789c4d3c2d48aca3539a5de4da035[log] [tgz]
authorGoma Authors <noreply@google.com>Wed May 27 05:36:45 2020
committerFumitoshi Ukai <ukai@google.com>Fri May 29 02:11:27 2020
tree1d968b8301ea87d746bb1791280c0c7adf2030bb
parent924c40c70a7c5a261e17cc78e40c9d4d95bd5183 [diff]
Import changes for goma server

  - 359be3990e566f2e69c48bb60eb204e9928456e1 remoteexec: Support more flags for clang-cl relocatable c...
  - b71ef8339d14b9b8c9f4e0deac2840450142494f Support -static-libgcc
  - 145591b21f79ef856c4ce77a09274af6b46b2fef remoteexec: implement clangclRelocatableReq
  - 6a04d35fa44717878506d89f0ec2ee5520829cb7 remoteexec: log user group of request
  - b79757327b70ed13215f2d684b3ea3e94bb18e8f remoteexec: InputRootAbsolutePath support for Windows
  - b44b229dc6d267eee2ea0463b81f830ad03cd53f roll google.golang.org/api v0.24.0 to v0.25.0
  - 2cf5b6d60b7e780954ff18fd68f62a9fe7fd0d19 use go 1.14.3
  - f792929863097064795fb12a41cfd5ebd58529a7 roll protoc to 3.12.0
  - 6be6fc6f3d659aab7808316039ba6d745f77cfa1 roll github.com/golang/protobuf v1.4.1 to v1.4.2
  - c0569523c6472d2aa5bc6dcb4c969a612a792c7c Remove reference to RBE documentation.
  - 01a4d419a70f66388c2a7f7bfa8fd3b17289897b roll cloud.google.com/go/storage v1.7.0 to v1.8.0
  - d1f2f0774a9e1fa871b28259b474df7418c3472b roll google.golang.org/api v0.23.0 to v0.24.0
  - b23097692c90759587f572314c416a518a1b7084 roll cloud.google.com/go v0.56.0 to v0.57.0
  - c77d37dcace158ce417e97c8ad0bd4e8da35664f roll cloud.google.com/go/storage v1.6.0 to v1.7.0
  - 617403fdc0cbf8d9ba23ebd5109f323b82ebecbf roll github.com/golang/protobuf v1.4.0 to v1.4.1
  - 58081c86461cddccbeafe34022711bac6089758b roll google.golang.org/protobuf v1.21.0 to v1.22.0
  - 53c29dcc78dfafb39a288755df3e40c29275c563 roll google.golang.org/api v0.22.0 to v0.23.0

TBR=yyanagisawa@google.com, sque@google.com, yekuang@google.com
GitOrigin-RevId: 359be3990e566f2e69c48bb60eb204e9928456e1
Change-Id: Iecf17209d66bddd9fa47f28c52cf70c1e0a1d4f1
30 files changed
tree: 1d968b8301ea87d746bb1791280c0c7adf2030bb
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o