Google Git
Sign in
chromium / infra / goma / server / 7f1668a89914b84b1ba219299f2f25642137e118
commit7f1668a89914b84b1ba219299f2f25642137e118[log] [tgz]
authorGoma Authors <noreply@google.com>Mon Apr 26 08:03:42 2021
committerFumitoshi Ukai <ukai@google.com>Mon Apr 26 08:32:23 2021
treeb97f3fd13e6cfe606629a03fd638cc82a59d94fc
parent48e3e9026e60a470e440e865ff27b02dc9604410 [diff]
Import changes for goma server

  - 1a6bbe3274588ca7e97c865042c12d12edbc6690 update goma's base image to ubuntu 18.04
  - 80208c196d4287f40ce7c2ff962dbe5c04a0c9e6 remoteexec: add cross tag for RBE latency
  - d603b5908de4c5f0fd382c0c9785544a24f3367f fix -imsvc for windows cross
  - a99662ca8b8019c9ca27d53ece4a17bf4566082e fix -resource-dir for windows cross
  - f7d5df8c574fb2f4daa3478ef15585003bbaa4fa remoteexec: log cross compile type
  - 9ece49a555e96722fe1589b2e166ae4beb186759 fix for pnacl-clang.bat
  - 187521df7447d2978abab600af2f3c6af119781e fix pnacl-clang target
  - 3252da722b78a9a7dc94268ec337f09dd34ca76e use go 1.16.3
  - 61c5b528a3eddcee80a972c3b466934c46cb905d remoteexec: add test for windows ats & windows cross
  - 80736a74ae59d5ea4a6d114909dffee8a16f3ee2 remoteexec: fix windows dedup
  - 48d4fb1f2baead1f57e8bea8a3431514d6a5375e remoteexec: make --rtlib= relocatable
  - 453728e7c0fbd3a84a021e0eabd8870a7d9b25b1 remoteexec: support windows cross
  - 2fd0eca64142f7e1b271a0e8ddeec2777c254e01 descriptor: fix for pnacl windows cross compile
  - dd545a524f1edb2ccc2d85182e24a2875354049a dedup input filenames with ignoring case for Windows
  - d3ecbd19f4a697d56e9cc83315461d5814b09ca4 setup support windows cross
  - 032d4948194a2fb6734fbc9a8099fd1c27a07ff8 winpath: add ToPosix
  - ba840171d45a273ef2d47adcc667c4fbf81ed726 add windows_cross option
  - 8c280f1b00721ced84ef82dcfdf6a9061f40ce94 remoteexec: handle -Xclang -plugin-args-*
  - f25daa29f4146c25178503db9a46579e419cfe3d remoteexec: don't log error details if codes is OK
  - 816c57e5a65d57f469bc0a984761cecc24d528db remoteexec: record error details if exists
  - f7fe660d334f7cadc3a67e104a8cba8c29ad74bc roll google.golang.org/api v0.42.0 -> v0.43.0
  - 8a39e9b4750f25d2b5efec88463545a057247769 roll cloud.google.com/go v0.79.0 -> v0.80.0
  - 588653e059d80edca6534a367b6f2e822b9a88b3 add acl_lookup command
  - 56b296201042f5ce257650fffaa79d1d782e7dcc roll github.com/golang/protobuf v1.4.3 -> v1.5.1
  - 171c59e1266ab669895bae065fa1cf432e023506 roll google.golang.org/api v0.41.0 -> v0.42.0
  - 74d35147dc0608bdcf658b9cc6804005454a9cce roll protobuf to 3.15.6
  - bb7919124a39de18c0152914e73cb135b916aa2b use go 1.16.2
  - 99b0870036c05a650e5d5d5cf532cd0160bc57f1 [gcc_flags] Fix -ffile-compilation-dir relocatable bug
  - c3b4ff315dc9af90977250198442e33fd8ba0290 roll cloud.google.com/go v0.78.0 -> v0.79.0
  - 90550ace421f2a7585ae2d63e630963b2dcde7f4 roll google.golang.org/api v0.40.0 -> v0.41.0
  - 4aeba81435ec0a7347a2495f4fc5993834d81f79 remoteexec: ignore env var if /winsysroot or -imsvc is gi...
  - 637b12a53f0908dcf3d27a350836552a74be0b55 roll protobuf to 3.15.5
  - 97677d7bd10bccbd5564859c81c720d85262e995 remoteexec: use cas error for output file.
  - afadec5d6cf1303e214701c7b596f5273f9b93cc roll github.com/google/go-cmp v0.5.4 -> v0.5.5
  - 2a3dc2d1ddb24577aba5a7456be143668b2fe207 roll cloud.google.com/go/pubsub v1.10.0 -> v1.10.1
  - 8804ca5754ea18071c184d2d9c9e521ddb9ffcd1 migrate ptypes
  - 5ef81ec10990b583d15e9054870c9c1b1b6b56ca migrate protobuf library
  - 9eef9a1cd7b12ab6e2ca78a93131055935bb4ef5 remoteexec: handle -Xclang -target-feature
  - 126b0e2724fc11fc3dedee3a80952bc42e13fdce roll cloud.google.com/go/storage v1.13.0 -> v1.14.0
  - 14545dfae258ad5126d8c67bb3f7aeee149f756c [flags] Add support for '-fprofile-list=' flag
  - b2b7344397758cd264672d181cda56e35e1f1787 [flags] Support jointed form "-fdebug-compilation-dir" fl...
  - 94b1d90241a31ead35899302139e10b67f7443dc [flags] Add support for code coverage related flags
  - be882f5c2322393265d15395f1adadcdf687c7e7 remoteexec: relocatable feature flag for -Xclang
  - 9906e83ef1f369a8e22ab546ab094bdc6cca8c3d migrate github.com/golang/protobuf to google.golang.org/p...

GitOrigin-RevId: ec0138e0a575ad2ce4ae851dec6502ccc32921b1
Change-Id: I203208e108c52bf7d4a32ffc61c8370a0952888f
94 files changed
tree: b97f3fd13e6cfe606629a03fd638cc82a59d94fc
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o