Google Git
Sign in
chromium / infra / goma / server / refs/tags/v0.0.7
commit14815ae6e9457f5ac202a7c3cbf402cec4843bb0[log] [tgz]
authorGoma Authors <noreply@google.com>Fri Sep 27 01:37:48 2019
committerFumitoshi Ukai <ukai@chromium.org>Fri Sep 27 01:59:18 2019
tree09d67f840a031ad683e0f198e74c3b5327a730d7
parent1723a31851df9684b23e08481b96f9b4cc9234c5 [diff]
Import changes for goma server

  - 90ed184205b36115cfcc7ea5aef6a4abd54ff613 roll google.golang.org/grpc from v1.23.1 to v1.24.0
  - e5531b346f92ea24951c2a3047a14b71a5b25415 remoteexec: handle bare -mllvm option
  - aa4d76f74dbcb6ead9383a2b856843d9bcace210 remoteexec: check -resource-dir for cwd agnostic
  - 30297c35eed95cbe8d68b009559ecbb0afd9b8d9 merkletree: remove "add .keep_me" log
  - 493259dcb5a47ce4376dbe68bea12da6aa29a20a log: use grpcLogger for GRPCUniaryInterceptor
  - 862a26595b2426b52d2cf43495958e5813ce0795 remoteexec: remove "out of root" warn log
  - c0236eb9dc3566c0d4b6d327a8a900f386b14d07 roll cloud.google.com/go from v0.45.1 to v0.46.3
  - 88fd24474d9cb8132c7086eb91cf8501569fc365 don't use zpages in file_server and remoteexec_server
  - 76c83420404f8aae7ec10ee14319e0b49f56c8bb Revert "use URL Host for trace span name"
  - adbdf830a09056959ded8161e7f8bba507bcf4c2 Revert "disable opencensus zpages"
  - 8932e7572d20aad69441f0c1bd3574f5e82d2430 roll github.com/grpc-ecosystem/go-grpc-middleware from v1...
  - 8f087302aca5f0d63f0718f21b8110d9894c6580 roll google.golang.org/grpc v1.23.0 to v1.23.1
  - 5163ecd8101c9714cd35bdf5031b0143f666ca51 disable opencensus zpages
  - 3d6f425d2b99bcb4b593abd7a065ef50a8439fad log: fix grpc log level check
  - 432a3f5e822957d593f55ef26a3a59cd2b32d1cb Do not treat out of root as err.
  - b6539d9736882522e33b2a783b12361b863a6a5a log: fix grpc level filter
  - 43121eed64af65d2d703e798ff33155169484130 use URL Host for trace span name
  - a04d649e1d1caccec463e3b10a8aea5f8b9d8c61 log: log grpc >= error level by default.
  - fe9ea40d37f161ee33a4d744c9442c38e3a56738 Revert "file: preserve conn as long as call is ok"
  - b98a8b1284b205b2f91f24b712098f8058477f60 file: preserve conn as long as call is ok
  - ac2ed4a086f7f5d67cb6a3c899a047f93d99c8db roll google.golang.org/api from v0.9.0 to v0.10.0
  - 452cae04855735c8d8ba09cda1d6cf5bf8554f91 roll contrib.go.opencensus.io/exporter/stackdriver from v...
  - f537de75686d293f2c3b4a2bc3f62d85cec30b1d roll cloud.google.com/go from v0.44.3 to v0.45.1
  - dd1bb305cd260ca6e3a4b08791451bd7cfe8f98e roll go.opencensus.io from v0.22.0 to v0.22.1
  - c5a9a6d893071611124dc714fda558fc354071fd update github.com/googleapis/google-cloud-go-testing
  - fb44fdae5531a18df65b3a88a1df7c64cb776708 Make absolute path symlink request to relative.
  - 50e4068b805568e0c602c16f4fc84b0981084ba5 Revert "file: load balance by dial per each request"
  - 0c033df2c4a594125d92ba70f9061a60c16e4dc3 roll contrib.go.opencensus.io/exporter/stackdriver v0.12....
  - b5718db01eac8660933eef5c0031eb5d0959af49 use go 1.13
  - 8467b677010661d5fa576722a99d564b02e1a2d4 update golang.org/x/build
  - d1fc6de391b055e3b96fdd14be16121835484728 file: load balance by dial per each request
  - b82944367376a4af30fe8243291b4bef07c30099 replace git.apache.org/thrift.git to github.com/apache/th...

TBR=yyanagisawa@google.com, tikuta@google.com, sque@google.com
GitOrigin-RevId: e10b3db23c0337c14f5645eac79dce7b6fa05aca
Change-Id: Ib6de1850e3051d4c024fdc7caa91539dcfc96853
19 files changed
tree: 09d67f840a031ad683e0f198e74c3b5327a730d7
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitignore
  22. buildsetup.sh
  23. cipd_manifest.txt
  24. cipd_manifest.versions
  25. CONTRIBUTING.md
  26. go.mod
  27. go.sum
  28. LICENSE
  29. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o