commit	b8fc06406b031ad8fdbe18160199953f1c274664	[log] [tgz]
author	Goma Authors <noreply@google.com>	Fri May 01 00:54:27 2020
committer	Fumitoshi Ukai <ukai@chromium.org>	Fri May 01 01:21:59 2020
tree	c58eacb8658551959e8c8cc11672b86d0b0137e7
parent	af23dcce34b69bb8362fff46caed79227d5c28bd [diff]

Import changes for goma server

  - 94d6b985d74e79539863f5347ae0bfff87f3c1c3 fix copybara
  - 8cfb431277e0726a165534448f79814c0f12feeb roll google.golang.org/grpc v1.29.0 to v1.29.1
  - b8cdc60420844fb344cf84fdb4dc666a40215cae roll google.golang.org/grpc v1.28.1 to v1.29.0
  - d7a3cc9ba6359f71f6d90eeaa5a9246d035c5313 roll google.golang.org/api v0.21.0 to v0.22.0
  - 46a6fef2c52f1e708d19505f0faf07b182cdfcb7 remoteexec: retry for RESOURCE_EXHAUSTED error
  - b0ee93ffbe1e4eefbdfa90e3f80d5419901c2662 gcs: retry with randomized backoff if it got 429 error
  - e6f4986a7a51ba1167cc8ee811b2a9e5fee51901 roll protoc from 3.7.0 to 3.11.4
  - f2e1d3552d48159b61ad51bd1b01ca68bc3c3b66 roll github.com/google/protobuf to 1.4.0
  - 869c45eee6a2c5347cdbd85cb9c9fe3e34d6e221 remoteexec: use pointer for proto message
  - 008d45dc0f192bc1389f8458612b256908d0cf36 normalizer: use pointer for proto message
  - ae95424bd43cb7e34a11e9e84764d271de6a4344 proto: use option go_package
  - f4cb0d7c4c344ab2e372acf021e3f57530b939f6 remoteexec: use InputRootAbsolutePath property
  - d75f29d723206aa51cd0815ff0ddd3c02f3a91cf use go 1.14.2
  - b8d42a28bf453d83805d506f7242a7a6aa641b9b roll google.golang.org/grpc v1.28.0 v1.28.1
  - 56aa9b08b389f43f2617aeb860660f5803587435 roll google.golang.org/api v0.20.0 to v0.21.0
  - 68d9326baaa2723f363b99022f729a9332d2a538 roll cloud.google.com/go v0.55.0 to v0.56.0
  - db4762ed162ced413ccb6a8040ede1e601db081e roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 279ed94da32cc792de9b44fc50e25a416ea87469 file_server: set unhealthy if GC couldn't reduce the memory
  - 449f6b25e8aeedcade61c8f16142e220edc7b6b5 exec_server: set max msg size for file_server
  - 92c1bb98100620d154e5100348978c824bf26404 cas: Stop assuming BatchUpdateBlobsResponse_Response.Stat...

TBR=yyanagisawa@google.com, sque@google.com, yekuang@google.com
GitOrigin-RevId: 94d6b985d74e79539863f5347ae0bfff87f3c1c3
Change-Id: I04de808c39c3e3ae7c9a32a4627977d1957fc795

57 files changed

tree: c58eacb8658551959e8c8cc11672b86d0b0137e7

README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o