commit	eb5e1bd07bcf2c4d4f776ff7bb24192684cbd629	[log] [tgz]
author	Goma Authors <noreply@google.com>	Wed Jun 24 06:29:37 2020
committer	Fumitoshi Ukai <ukai@google.com>	Fri Jun 26 01:18:56 2020
tree	89e5b68c1d36984734987ea0e30a9310a4397b96
parent	85ab0b169b64e71eb843a67cf3fece0c55afa200 [diff]

Import changes for goma server

  - 06f903f9f5f02994b1ddcf2bc419bb75d74655d8 roll google.golang.org/protobuf v1.24.0 to v1.25.0
  - 868e7267cc73d65931f111e3a0275229feaf6695 roll cloud.google.com/go v0.58.0 to v0.59.0
  - e6ecfd86132940b48e79d361d2bf308837d21c46 roll google.golang.org/grpc v1.29.1 to v1.30.0
  - c50132ec76ede6f5a416adaa659771723d37500b exec-server: fetch-config-parallel defaults to true
  - 0044e3681c05e0c49e8f87cf942afbb9534ac9ea roll github.com/bazelbuild/remote-apis-sdks-2020011715525...
  - a291d308014f0243bb5de12a649fdd0e0c12cdf9 roll cloud.google.com/go/storage v1.9.0 to v1.10.0
  - dfa0669991291c2150e25f44029c14f58cab0fad roll google.golang.org/api v0.27.0 to v0.28.0
  - 2003a4c5445da332c326ef58e582788b0c8e8d5e roll github.com/google/go-cmp v0.4.1 to v0.5.0
  - c9a6a00621fea23384ee9d6cf59e1d4c6c81e342 roll go.opencensus.io v0.22.3 to v0.22.4
  - 87c244767fd2a295b365aef5f18879d1b7ee6a6c roll google.golang.org/api v0.26.0 to v0.27.0
  - 450daff9b935c824c0abcb92c3d18da55d843163 Use the more inclusive and descriptive term "allow list"
  - 6ecdbae6c5d199cffb187f61c0d99845c079419d roll cloud.google.com/go v0.57.0 to v0.58.0
  - 9fda3239444999309d29284812474ec81ce95b64 use go 1.14.4
  - f52fd5e3257adabc349143b536e02fec637ee7e5 roll cloud.google.com/go/storage v1.8.0 to v1.9.0
  - 3572d394d9793528ab299ba019ad75b69336f87d remoteexec: Pass env vars from ExecReq to remote exec for...
  - 41a635bb53c97da9a99e03f3032a8b290aa48a51 delete setupFakeGomaFile
  - 3c9cefb6772096d4e67bf44fa92bf31a4135e4c0 roll protoc to 3.12.3
  - 0089e3d6fe23bd87195d41ecea5c3b60180c22db roll google.golang.org/api v0.25.0 to v0.26.0
  - 2fd7c2da7661b459f0cecb149478390c9e7b2185 experiment RBE hardened mode
  - a327605dd2adbfd01598177f98d84ba593667de5 roll cloud.google.com/go/pubsub v1.3.1 to v1.4.0
  - 6b4d4b9d3f70071c2110aabb15a1e4e979852a0f roll google.golang.org/protobuf v1.23.0 to v1.24.0

GitOrigin-RevId: 06f903f9f5f02994b1ddcf2bc419bb75d74655d8
Change-Id: Ic0dc32f3d25f98b9e6264f08bc6144625b914d0c

29 files changed

tree: 89e5b68c1d36984734987ea0e30a9310a4397b96

README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o