Google Git
Sign in
chromium / infra / goma / server / refs/tags/v0.0.4
commit02b31977fcb75117c8bb0ed43c837b847ff1aa1e[log] [tgz]
authorGoma Authors <noreply@google.com>Wed Jun 26 03:48:06 2019
committerFumitoshi Ukai <ukai@chromium.org>Wed Jun 26 04:29:21 2019
tree79cb8615c7816621012de55a334a26ecb84577bc
parentc6b7c66e103e05a58fb7177436b64cde21d2d9dd [diff]
Import changes for goma server

  - 021a72b6e00502aaef798ca56a55f8f199d11078 Relax max heap size and file size in nsjail config.
  - fd0ba83ac13ad61f85af192824e0c7caec944a3a auth: resp 504 when auth check failed in retry
  - ab2b317600157529c400db81d685dc1f8a86a381 roll github.com/bazelbuild/remote-apis to latest (a5c5773)
  - 084c463010afd6a4dbd77849204c7a20549f1bc3 remoteexec_proxy: show config on status page
  - d2833f4028bdca68e94ee8b2a8d3d6231b85a73c Add dummy PATH environment.
  - 6cbc346f29dae25e1717203cd2288cebada2b21f gcs: set ChunkSize
  - 0b6b8ad4eb1438cda8983e3f25ae8d39a77148e0 remoteexec: set max call send msg size
  - 255c697d449c4f6b94573adbd10da61ec18d453c roll contrib.go.opencensus.io/exporter/stackdriver from 0...
  - c6b79ee7793551b32c47c418abfb20aabdc797ce Use nsjail config instead of command line arguments.
  - 2a04c3f08e67303c70806b3ad37c0c1f9cd4d62d roll cloud.google.com/go from 0.39.0 to 0.40.0
  - 05f8c5088c311592178db87c40cdd84a8d441dc7 roll google.golang.org/api from 0.5.0 to 0.6.0
  - 26a1729a9a623f70273526efd46851ba0497cad4 remoteexec_proxy: add -exec-config-file
  - ab0ba1d55f0efa0f048d757d62e0503fa9b55bf4 file: report rpc error if resource exhausted
  - 4ef9ec9d0a999a5460d8b904b94f2b41e63b470a remoteexec: don't set credential on insecure connection
  - 383545afc875e3be09be9ed5d46c25b8d8d2fe04 file_server: protect from OOM
  - 85f94766beac9d2d90c46695809ef29fd5ba6334 Allow to have different infra/configs between internal an...
  - 25a2d424cb62237bdcdf65424692493b81c789a2 rpc: need register tag key for log
  - a75bbf4b361177cdc1222927ec2026e79e9f62dd roll google.golang.org/grpc from v1.21.0 to v1.21.1
  - 1b694d68aeaf6f840d61e38d80ae23fb36c34a80 remoteexec: remove too chatty debug log
  - 1c53131edac839119f1ef86d0ed5e19cc8a9dea7 Update nsjail from nsjail-2.4 to nsjail-2.8.
  - 8d692f639aaed837ac64f922f2213fbf840cc9f1 go mod tidy
  - c420ad5b135719736d813fd88de38939efc26774 roll contrib.go.opencensus.io/exporter/stackdriver from v...
  - 3289808e5fe3780d5e195048909491a8200a9135 roll go.opencensus.io from v0.21.0 to v0.22.0
  - 24d9545cee0f3293d2acd9d8c744a21a8e2b5660 exec: set error message when commandspec normalizer fails
  - 5fd89a7f99b51ae4d9a79eaf6700d9d54cf7f09b Allow RuntimeConfig to say having nsjail or not.
  - 09fc66a7f8999c7b6884e83a94dffdc8ef2bd224 Use CIPD to install go and protoc.
  - 260757e7472cffc4c53993a8787ec0a5c16f2605 Recover nsjail docker build.
  - 0e9a5a2a0313e39185327df894c32194f1a7b7c2 exec_server: allow no prebuilt if ATS enabled.
  - abe4ea6697ec0e9eee4d587c180ebee737d47f5d remoteexec: Update logging for upload blobs failure
  - 5124c730ffd6ae455749983455cb6b71c7026a2c frontend: reply 503, then 429 when memory exceeded
  - 74ccce2bad3d60a3538549161bcdcf1d35f7a196 remoteexec: Truncate logged time durations to milliseconds
  - 559d53e907d552314e7bc105856461c4e871909b remoteexec: Log more parts of time taken in exec-server
  - 276dcb48ebecd3e0f253cf6b422ef01e4c6e928c GC: don't run GC concurrently
  - 3ba574567f4cc52f8ca62093c5b3fa2c56b99b0a Allow to chroot or not will be decided by a platform conf...
  - 21d7a02f0a8273d503d8b12e3a3ecc25e09d697c update google.golang.org/grpc

GitOrigin-RevId: 2dd14a0db8aa454342c13eee46ff2971ff20c4f4
Change-Id: If2fc31cfef58ec0eb1909f88b3e609db9fa262df
33 files changed
tree: 79cb8615c7816621012de55a334a26ecb84577bc
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitignore
  22. buildsetup.sh
  23. cipd_manifest.txt
  24. cipd_manifest.versions
  25. CONTRIBUTING.md
  26. go.mod
  27. go.sum
  28. LICENSE
  29. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.

Log messages will be output to stderr. OpenCensus z-pages ara available at http://localhost:$PORT/debug/

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o