Google Git
Sign in
chromium / infra / goma / server / a94b8f4c7e9094c2d3784cfc1a969446feb704c1
commita94b8f4c7e9094c2d3784cfc1a969446feb704c1[log] [tgz]
authorGoma Authors <noreply@google.com>Tue Mar 31 03:19:07 2020
committerFumitoshi Ukai <ukai@google.com>Tue Mar 31 04:21:11 2020
tree82facf5807d035cc84c7eb73b10dad079f75a5c2
parent9387c31666744317502a28fc5a15ed3095019cc1 [diff]
Import changes for goma server

  - 786216f2f72f949629c33852ff63ce33510edc89 fix for publish_to_external_repo
  - a87d59872a64457efea7713ce74bbf603d06f70b Revert "remoteexec: use command.working_directory for non...
  - 255f0ece2468fea4ee16ac6369e4f1a6ead0a48f remoteexec: batch fetch for gomaInputReader
  - 0c2338e24694223b24f59eb000053d72d35da37a [toolchain config] Add feature to load config in parallel
  - 7f2dee79b5ef5ec7e22de9486d85804988eed280 remoteexec: fix -fdebug-compilation-dir handling
  - 895eb17049240200e8127400d5f970681627256f remoteexec: rename cwd agnostic to relocatable
  - 13745f68074d375d7d5482468651a5ae8c568e15 httprpc: increase timeout if ctx is timed-out
  - 1bc05465af290e200d74928bf8c4255bc6aff93b Add flag to disable server TLS cert verification
  - 23912419bd78500dcbac8f943f3d569c48372414 remoteexec: use slice for gcc.go:pathFlags
  - c6ab6f4278cac422bdc7be517db9f81e09faa673 Add OWNERS.
  - 50502850a7d3e81152502461412ce998de72f8e3 remoteexec: handle some feature flag for cwd agnotstic ch...
  - 3ff07d3135db5523b89bfaa26c55909c68a01939 use go 1.14.1
  - 44e354a7a5bd9b23efbae063d0e44f8b22ac672a roll cloud.google.com/go v0.54.0 to v0.55.0
  - 5a98cf13e6ca1aed3959be039fbcced7170fa0af cas: reduce memory usage in separateBlobsByByteLimit
  - e4a1b73baa5f5385231d531ad20e576200d73a4b handle additional arguments when determining cwdAgnostic
  - 36a3e8cb4b2d3fc149448a577978612e55e159bc roll cloud.google.com/go/pubsub v1.3.0 to v1.3.1
  - b1abc4046e85064e14f4bcbfddf26f8d6177388e roll github.com/golang/protobuf v1.3.4 to v1.3.5
  - 8b839a89b144b1f3bc57d918cb3263ff43fc0e9f roll google.golang.org/grpc v1.27.1 to v1.28.0
  - 6f8d3ef24a2ec5017ca3bd4f1b593de86155ee1b roll cloud.google.com/go/pubsub v1.2.0 v1.3.0
  - d07964413edc7f99b6261310a35629abd5da7727 roll cloud.google.com/go v0.53.0 to v0.54.0
  - a4ccdb5b9835cfdea026468e28ea64b7823759dc mixer: return StatusServiceUnavailable if token expired.
  - e96fb73af507d39679747135f97af39a881e500b roll google.golang.org/api v0.19.0 to v0.20.0
  - b12d8580773d1afc61377a2c0b03506d7f2f2b69 Revert "use go 1.14"
  - 9df336f29c934d537cff0943f148cb229ef7ab3a roll github.com/golang/protobuf v1.3.3 to v1.3.4
  - 6c161b3c40e4245d64004b93e01e9a8414e634a9 roll google.golang.org/api v0.18.0 to v0.19.0
  - 42578364e8b8d5d0fd7ef596519f5f59c0826fa2 use go 1.14
  - 8634f6c6448ff45c89957574cf16451da5a750e3 remoteexec: gcc treats hmap as non-cwd agnostic
  - 786f644e76df655907aaca52afd79ce8cd91f7cf remoteexec: Reduce size of ExecResp to < resp msg limit
  - d1f6c3a1d14263ba83326299b0c1f7ef5c2adefd roll google.golang.org/api v0.17.0 to v0.18.0
  - 3b38a0e70eaadfb67da9fb87d2e3d0b3c7161bda proto: Add new FileBlob::BlobType: FILE_REF
  - 2171e844c06091f753b761e7cdcf93175d1182ab remoteexec: Factor out function to create chunked file blob
  - d6272703df65f15e24f4e611aba25ad0075e259f remoteexec: call LookupFile() one blob at a time.
  - 82052d721336f525e5bb503271f2352f893665fb Record RBE's execution duration into the ExecResp
  - b9935b06dec093a18e48e50d4bdc5e8c16257e0d Update goma_data.proto
  - 9b0b387d5ac4503f35195432c0042c0f5150a2fb remoteexec: Add unit test for gomaOutput.toFileBlob
  - fd8b69d736133b33fdf72eadcb5a9ae50f6f259b Make gomaOutput.outputDirectory concurrent
  - 1a1b7a1311160ae5648046b71bba6cb61fb7b0ef remoteexec: Refactor gomaOutput.traverseTree
  - 6ab056629bc7313832979576de60080e2a800424 remoteexec: add unit test for gomaOutput.outputFile
  - a8a95cf0b9e52dee9040ed1c36eaa41e4ee1f206 remoteexec: Add unit test for traverseTree
  - acec3c2918f16f710f0af6339d522e594f4a3e19 remoteexec: Change order of traverseTree and outputDirect...
  - 5ce3ce9359a7caa980c32bb843d88ea44338cf96 remoteexec: Add helper funcs for gomaoutput_test
  - 0a042edfd6d76d70f3a267e253d9b69736667f6e remoteexec: use command.working_directory for non-Win
  - db9890e479d82b71656e815a775679f612dcdf4b roll cloud.google.com/go v0.52.0 to v0.53.0
  - b239f112ff79a0b5996495c559efbd67fae343d5 use go 1.13.8
  - a181e5a6449ebac79dee577677d459795c207f63 Revert "remoteexec: use command.working_directory"
  - b56b40cfbff6469c81475b097fc10ed93ae89450 remoteexec: use command.working_directory
  - 8c46d9ac9e240a8c270f8069b26beb55c3bbb702 roll contrib.go.opencensus.io/exporter/stackdriver v0.12....
  - 6f6f5357fc33e969a378cd868562bee9bebf59da roll google.golang.org/api v0.16.0 to v0.17.0
  - 6550b010d6604a95939b795712120645d6e72e28 roll google.golang.org/grpc v1.27.0 to v1.27.1
  - 686ecc2d45a88a4e2ae978151964e481da5d8955 roll cloud.google.com/go/pubsub v1.1.0 to v1.2.0
  - 6becb26481d8fc6c30486a8888e4033cd485248a roll google.golang.org/api v0.15.0 to v0.16.0
  - 36a475cc07e1159a3016a1cff12e08a9fbde5cda roll go.opencensus.io v0.22.2 to v0.22.3
  - aa9c9a7b99f14aa427da8a2365f8073991b92da8 roll github.com/grpc-ecosystem/go-grpc-middleware v1.1.0 ...

GitOrigin-RevId: 786216f2f72f949629c33852ff63ce33510edc89
Change-Id: Iefea5aecd28687323d5d6d6e69f5b66e3d601113
42 files changed
tree: 82facf5807d035cc84c7eb73b10dad079f75a5c2
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o