Google Git
Sign in
chromium / infra / goma / server / v0.0.26
commit9d24c7bcaafab12eb0e99c795bf1a8288a6701ca[log] [tgz]
authorGoma Authors <noreply@google.com>Tue Sep 20 07:17:27 2022
committerFumitoshi Ukai <ukai@google.com>Fri Sep 30 05:58:48 2022
tree975567cabe5bbe00dcf49126c7d53a765c02ec0a
parentfe9f60ebe805a8e53c9bea869e7b6afb92223da7 [diff]
Import changes for goma server

  - 5a3493b56da0b792d36e6fdd92cf410923fa683e roll go.uber.org/goleak v1.1.12 -> v1.2.0
  - a31c73ae0674a0b6fa570c68d7b02a710fbd1a05 roll github.com/google/go-cmp v0.5.8 -> v0.5.9
  - 52180f068915f5276133a85924c22a0fd93f3ada roll github.com/aws/aws-sdk-go v1.44.86 -> v1.44.101
  - d2a75b74dfef5291b1f31bf112791f6108125056 roll cloud.google.com/go/storage v1.25.0 -> v1.26.0
  - 6c89ac80f2518becefb8fb69e48ef88cec2a0469 roll cloud.google.com/go/compute v1.9.0 -> v1.10.0
  - d2d3a557566a42efc342ba74d37ecce1f11fc3d0 roll google.golang.org/grpc v1.48.0 -> v1.49.0
  - 8fc83527575b5bdd266fdcdc7bed1e8fc4a6e0b0 roll google.golang.org/api v0.93.0 -> v0.94.0
  - 3ba1668bdf5f96b873781ed342a3e2ade9349804 roll go.uber.org/zap v1.22.0 -> v1.23.0
  - fa557adf4116d05fd33d2283c24dd00263a9ddc5 roll go.uber.org/atomic v1.9.0 -> v1.10.0
  - d7525e45120db680ff105fc63b6242502d2c0128 roll github.com/aws/aws-sdk-go v1.44.72 -> v1.44.86
  - f61c19a36b73bebedf3c45f39d460b8a9f59a949 roll cloud.google.com/go/storage v1.24.0 -> v1.25.0
  - 9718b37d6b86c49a2eb2dfb14ff8f6e6f04fe042 roll cloud.google.com/go/pubsub v1.24.0 -> v1.25.1
  - beb7dc8e13f852a407a45d79408fb2de299b7c96 roll cloud.google.com/go/compute v1.8.0 -> v1.9.0
  - 361508dab02961973faad22e1df2456d8c26432b auth: add metrics go.chromium.org/goma/server/auth.auth_b...
  - 755500ff0f978dbf0975ba0796012666c5334647 Revert "frontend: record ping requests by service account"
  - 302340e55d68a2da4d4f3dd5b53e55743fc9dbed frontend: record ping requests by service account
  - 441c4e6d02a8cb1471c46463f90543992a6245a4 goma-server: make the missing input limit configurable
  - 9c38d25ff291d587c3504b1d787299a52e190af6 roll go.uber.org/zap v1.21.0 -> v1.22.0
  - a14bca4a50db83205e3522a02f1a26d5cd10316e roll github.com/googleapis/gax-go/v2 v2.4.0 -> v2.5.1
  - 2ff97b804a62b3fb0886ebd187cd863b3b096071 roll github.com/aws/aws-sdk-go v1.44.66 -> v1.44.72
  - 654ce6573d798943fa482e3f3d4b7b46491d6427 roll cloud.google.com/go/monitoring v1.5.0 -> v1.6.0
  - e3299a4ecf63d1e83178132047a4498684e3d457 roll cloud.google.com/go/compute v1.7.0 -> v1.8.0
  - 94dd145e063bd6653c755f55fd97a3ca60414510 roll protoc from 21.4 to 21.5
  - 50e261a5e1cbddd21ded86b2248d00bd69a7bac0 Use go 1.19
  - 9532550e6673a2638eb5e536766e85b7777fc5fa roll google.golang.org/protobuf v1.28.0 -> v1.28.1
  - 29fd35ca2289ef0868aaf679e6a7fe6461465f13 roll google.golang.org/api v0.87.0 -> v0.90.0
  - 30e21baf0581f9dddd98cbea0db106d87dc4d65e roll github.com/aws/aws-sdk-go v1.44.57 -> v1.44.66
  - eb4d39bd6fd6c202974f91143f796b321fd3fa6c roll cloud.google.com/go/storage v1.23.0 -> v1.24.0
  - 14ce627e387cccc04dff6cfeb5c5d88ad1b0a2e0 roll protoc from 21.2 to 21.4
  - 7d762909b6b254ea2d55dae9d2ec21a6e0c362b6 Reland "allow rseq syscall in nsjail"
  - abd47f4c3dfccc49f1c82330eb34a6cbbb7979e8 Revert "allow rseq syscall in nsjail"
  - 314d1338521a477a399227b64d8f61231bf48917 allow rseq syscall in nsjail
  - 02f0383cfacc8de35693e6bf67704d9905287503 use option.WithTelemetryDisabled()
  - 816fdf47bac5c2e2796d178d30b5aa4cce147427 roll google.golang.org/grpc v1.47.0 -> v1.48.0
  - 089e0ba5a2946702441e380ebc34777ba022fc1d roll google.golang.org/api v0.86.0 -> v0.87.0
  - a861228c1309c67322f3c29acd7a92c648cef0f3 roll github.com/aws/aws-sdk-go v1.44.51 -> v1.44.57
  - 26b5b14af38b0e0f21a56c97cca246453384817f roll cloud.google.com/go/pubsub v1.23.1 -> v1.24.0
  - 87753defe7c8b2678cf146a8902522d2527d1c6a Use go 1.18.4
  - 59590ac4205ba29492a329762eb6d0ac3389a7ec Use permanent link in a comment of remoteexec/adapter.go
  - 2e3d3afbeb59f4e3801e92ef756659491336a6bd Use permanent github link in a comment
  - 1a0858b7c07a57939151f961b41f63c2fa14174f roll github.com/gomodule/redigo v1.8.8 -> v1.8.9
  - e4734d7cfdf8adbad17fc1046011cc88a5869d77 roll github.com/aws/aws-sdk-go v1.44.47 -> v1.44.51
  - b8db3f9029a5063f92998ed0c9b49762eebf4c93 roll google.golang.org/api v0.85.0 -> v0.86.0
  - b438f2adc709d9ea15801cf544f9dca4d8e5682e roll github.com/stretchr/testify v1.7.5 -> v1.8.0
  - 8042c551b39e6e3b11d2aa40aec0e22ba336eb04 roll github.com/aws/aws-sdk-go v1.44.42 -> v1.44.47
  - c06f2f6b495ce773e0f1c10cb893548364036cbf roll cloud.google.com/go/pubsub v1.23.0 -> v1.23.1
  - 56b00915f8f26977c5b8444ea5d20d3bfca533e7 roll protoc from 21.1 to 21.2
  - e699602d4328d7763c3e40b155410212df9d58ef roll github.com/stretchr/testify v1.7.2 -> v1.7.5
  - ce835466cedf8889b3786d42a6105e2ba01e8a57 roll github.com/aws/aws-sdk-go v1.44.37 -> v1.44.42
  - eeadc8edba9d94a844a85c9b3f6bec53b9ed151c roll cloud.google.com/go/storage v1.22.1 -> v1.23.0
  - f259e93e72d0d0d85c07a0bb5aaeaed83eaf9aaa roll cloud.google.com/go/pubsub v1.22.2 -> v1.23.0
  - 3fe82e8455803e1db4ad0a7071ac13bb1a98ab24 roll github.com/aws/aws-sdk-go v1.44.32 -> v1.44.37
  - d3bdd53d719a3dfd32dfeaf09ce31199a303e7a6 roll cloud.google.com/go/compute v1.6.1 -> v1.7.0
  - a8101119c31b4be8b3a0451b76799a4a0c10a7ef roll google.golang.org/api v0.82.0 -> v0.83.0
  - 928b76fac5fa5b16faf10645cb48ca5bd8791758 roll github.com/stretchr/testify v1.7.1 -> v1.7.2
  - e0e286675f902edcb900e62c71da072549d7e275 roll github.com/aws/aws-sdk-go v1.44.27 -> v1.44.32
  - a2405d645a5e0dc3e954bf634002edbc90cc383d roll google.golang.org/grpc v1.46.2 -> v1.47.0
  - 0531671ccae0ee2b017fefd7ad9d4c7b5dea96c0 roll google.golang.org/api v0.81.0 -> v0.82.0
  - d43d16b8301a8fc9f85b356780bca9dd06da1a16 roll github.com/aws/aws-sdk-go v1.44.24 -> v1.44.27
  - 1cacf68e456023e07d302c27c29d665f7beb84a9 roll cloud.google.com/go/pubsub v1.21.1 -> v1.22.2
  - 4dde1f078e08dfb82c8b246187b67eff5718c7e1 Use go 1.18.3
  - fab1c12fb5ea3015f16515c8f54c21183b8f157a roll google.golang.org/api v0.80.0 -> v0.81.0
  - c157491449f5a838e2a4e479a4ee0def16cd21db roll github.com/aws/aws-sdk-go v1.44.21 -> v1.44.24
  - 1e131c2c76b9b09c3099436e1be49b22d73dc12c roll protoc from 3.20.1 to 21.1
  - f2f5438b65e13c455cc0d78d4b090d43d52b1ee6 use go 1.18.2
  - 3932978782034895c04d5fb26b48d95681af9031 roll github.com/aws/aws-sdk-go v1.44.20 -> v1.44.21
  - ed08a46ed3a445ea7490a05582d87e618dca4787 roll github.com/aws/aws-sdk-go v1.44.19 -> v1.44.20
  - 184d0f3c232fcc21e7ceb7b49ec985ecde3c7a88 roll github.com/aws/aws-sdk-go v1.44.18 -> v1.44.19
  - b7157603fa4f1b03ba1d872abdce4db225526767 roll github.com/aws/aws-sdk-go v1.44.17 -> v1.44.18
  - 002934c79577de7c18ee930d2bd3e5cdc82de582 roll cloud.google.com/go/profiler v0.2.0 -> v0.3.0
  - 00773bb8080ddd4af5ca618c7e543e8f2ee589a1 roll google.golang.org/api v0.79.0 -> v0.80.0
  - 8140df6d9510e200042462f05db514540dc401d9 roll github.com/aws/aws-sdk-go v1.44.15 -> v1.44.17
  - ad0ad278b1c31fa5a5e309e3f058156185b49787 fix goma failure in windows llvm/cmake build
  - 114990a0da4f05de31b51f85db51b6d763a418ab roll google.golang.org/grpc v1.46.0 -> v1.46.2
  - f2a524dfbcdb3e672f91924963c67f02c4dde013 roll github.com/aws/aws-sdk-go v1.44.13 -> v1.44.15
  - 055bc7295e80dff1be77da72b2a5bb9257c6ed26 roll google.golang.org/api v0.78.0 -> v0.79.0
  - 026c56325ca3cfb1ca366201c3201befb6ab1c7c roll github.com/googleapis/gax-go/v2 v2.3.0 -> v2.4.0
  - 2e62f365654206aecc531ac5d55289989823c863 roll github.com/aws/aws-sdk-go v1.44.11 -> v1.44.13
  - d76656e18fb3a6ba33f611c570d7568138bb8fab roll github.com/aws/aws-sdk-go v1.44.10 -> v1.44.11
  - 732c4257ec421d56a1ff40ec497538b945d739b7 roll github.com/aws/aws-sdk-go v1.44.9 -> v1.44.10
  - 5a065790e57369e41ba6431a232a38da45d01464 roll google.golang.org/api v0.76.0 -> v0.78.0
  - 04442c656deddfc99cf41167d464f7c29eea8525 roll github.com/google/go-cmp v0.5.7 -> v0.5.8
  - 19bfb9a01716fcd2b14873540b720292d69ad9bb roll github.com/fsnotify/fsnotify v1.5.1 -> v1.5.4
  - 0c34b74025fa4b4c4079b14fa7dd8b139150151a roll github.com/aws/aws-sdk-go v1.43.31 -> v1.44.9
  - 5604943f7c0ea1d5881c6327dfc9b2021312f4f2 roll cloud.google.com/go/pubsub v1.20.0 -> v1.21.1
  - 7c27d28ed528094d2e7366dc989c56c504daac16 roll cloud.google.com/go/compute v1.6.0 -> v1.6.1
  - 32db5cc2c77166a4f331d251e8eefdbd21cd1959 fix github.com/prometheus/prometheus version
  - 91cf4b759c4fc6e45c29cbe7bf36109062c08891 [redis] Extend TTL on read
  - dce31968123e539f418d98d3bd4f23220becff85 reply http.StatusServiceUnavailable for auth internal err...
  - 81f2d7b66ceb1ada42018f0edef117a6e4f92900 roll protoc from 3.19.4 to 3.20.1
  - aa9af854c3121a4d743b971c92bb7d0780567a7f remoteexec: handle --unwindlib=

GitOrigin-RevId: 5a3493b56da0b792d36e6fdd92cf410923fa683e
Change-Id: Id4d5c571f6b2587b740b6f2991e628f4fa47eef1
82 files changed
tree: 975567cabe5bbe00dcf49126c7d53a765c02ec0a
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o