commit	daa08de9a78b8a663f8415d9a996ffe46cd0a353	[log] [tgz]
author	Goma Authors <noreply@google.com>	Wed May 08 01:42:56 2019
committer	Fumitoshi Ukai <ukai@google.com>	Thu May 09 05:46:38 2019
tree	1acbdf41021f8a9f6d96dfa95e66b9d55ec9d49c
parent	9c2c6e4ca15d26a562678f46ba400dff12adb07f [diff]

Import changes for goma server

  - 911cc693fe6893f5cff9e1233660a95d81d65226 use github.com/bazelbuild/remote-apis
  - 1e1260fe16cebf8dbf9cf6492c41acb87fba379c roll cloud.google.com/go to 0.38.0
  - ce25504b9f0dc3014ff224a2b91d8bca2c1dfa5f roll google.golang.org/api to 0.4.0
  - ac2ccec0ffb4cf64403224fc5d5bf11ca03163a1 Allow arbitrary script name to as a wrapper script.
  - f71ce1e66b26dc811051e9e4f4db666a9639f041 remoteexec: no warn if op.metadata is nil
  - 69bd629b0b2ad20db1548ac22d9d262d9430cfe4 inputRootDir suggests use of overlay if it is needed.
  - 22550b34d191cb2429858d8d30aa59e728ff1f4c remoteexec: cwd agnostic test for ATS
  - 6d3b3f1f6ec031b7d4205c3ec7a674c23308e44e exec: Pick sets selector in cmdConfig
  - 46f16c7dff0436829fc61ff02e5c0f136da916e5 roll google.golang.org/grpc to 1.20.1
  - fed8af5e12f814b8b9ef4c04692bc2d7d84042c0 Exclude infra directory.
  - 543e146a14af12a3724403af910ef375635becc0 Reland "Allow to have multiple wrapper scripts."
  - 6e2ee97899a8a601580425ff8c74a2ec33599296 fix: error strings should not be capitalized & no arg err...
  - 8b529445fa3420c70fbf2d39eed7d994cc28554f remoteexec_proxy: add -file-cache-bucket
  - ff36c19b7ba7e768cae3ded4c074793f3e50aa73 Add function to confirm docker & workdir won't be overwri...
  - 134662af055e6805d6ee4ea926a25223cc77414d Revert "Allow to have multiple wrapper scripts."
  - 9b3fefa9841a321aa36455bf233f33b656325a96 Allow to have multiple wrapper scripts.
  - 4b191d09b6dbd8ab225784bc619f18d659d8f7ef remoteexec: Log input paths when no common dir found
  - 21ff47c58b5cbe63fc09d2f8647373d415deac53 remoteexec: Rename commonDirHasCommonPaths -> validCommon...
  - c755b3dfd3e260e445afa0a2abedf4ca4b80d859 remoteexec: Refactor inputroot.inputRootDir()
  - 37cf82aa8e6f6c9b31e2cd005e44a8ba9a471e4b goma_replay: count errors
  - 276a5308828814b736f3ee05072b2171db740b1f goma_replay: improve
  - 35340d6f853a4f9ed75758ab5a7da9658dcd1fa2 roll google.golang.org/grpc to 1.20.0
  - ca6e254669892e0508b4988ad42c77d93a393d5a remoteexec: Log all timestamps on one line

GitOrigin-RevId: 911cc693fe6893f5cff9e1233660a95d81d65226
Change-Id: I05e4f511cff49e0c594234c3156599c644558290

39 files changed

tree: 1acbdf41021f8a9f6d96dfa95e66b9d55ec9d49c

README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.

Log messages will be output to stderr. OpenCensus z-pages ara available at http://localhost:$PORT/debug/

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o