Google Git
Sign in
chromium / infra / goma / server / 753d55e9476bc11356ed0f1727768ca7fc80b9b9
commit753d55e9476bc11356ed0f1727768ca7fc80b9b9[log] [tgz]
authorGoma Authors <noreply@google.com>Mon Feb 28 05:14:13 2022
committerFumitoshi Ukai <ukai@google.com>Mon Feb 28 06:43:19 2022
tree819b410733c5621076aadba718603a26c6d54713
parent5d76e12f9d2bd4b72fc9cc46fada5050cf08eb1d [diff]
Import changes for goma server

  - bfd9edf5c5936f8f045cdafc5e75cd00213ba64d goma: look at outputs when deriving exec root
  - 3f6697801b051159f27c5d34334a946c7e3e8d16 Fix handling windows absolute command path
  - f2054f033980cc1454ad842dad5d5c80bdedb5c2 roll cloud.google.com/go/compute v1.3.0 -> v1.5.0
  - 34ef29d9abf89be71f0f7bde22ea53ef6f6591cb use go:embed for embedded config/script
  - 9dcedfbc238cd9058d9610376746e18fbd838f16 allow syscalls used by python3
  - f9827e11b59f97d3de9b083db21dc3e083f50454 roll cloud.google.com/go/storage v1.20.0 -> v1.21.0
  - 8f5b848d64b563104010abae56dfe5caccfaf75b roll google.golang.org/api v0.68.0 -> v0.69.0
  - f79f8daef542fb45b5488880bd4910364ceee921 roll cloud.google.com/go/profiler v0.1.2 -> v0.2.0
  - d29b6fedd85bf59763f8e57fa1dc1abfcaa8991e roll cloud.google.com/go/monitoring v1.2.0 -> v1.3.0
  - 113100af2681769ddeb76617bb3795140ac02916 roll cloud.google.com/go/errorreporting v0.1.0 -> v0.2.0
  - d86d21cd20649665f33b87e96dabf71040193912 roll cloud.google.com/go/compute v1.2.0 -> v1.3.0
  - ad96594f40d650fe27d1785709df49d45ad18a74 Use go 1.17.7
  - 58c9af7f7cf029da951bbf2185159dec16ec0c91 roll google.golang.org/api v0.67.0 -> v0.68.0
  - c882e07c981df758a7e79ce9a6410e0ef3573f92 roll cloud.google.com/go/pubsub v1.17.1 -> v1.18.0
  - 076862693d3506d99d1115884b52e0c713e629d9 roll go.uber.org/zap v1.20.0 -> v1.21.0
  - 752c52ff519db713b771186639627f6c53814286 roll google.golang.org/grpc v1.43.0 -> v1.44.0
  - d8f5c1d11dd4bd5a686fee3f4dcf769c04d01bcc roll google.golang.org/api v0.66.0 -> v0.67.0
  - d8584874fcd17cfd700d7daa36f9ff03782a48ea roll cloud.google.com/go/storage v1.18.2 -> v1.20.0
  - c0634d649d179d9dba8f45361400964ebc818d9f roll cloud.google.com/go/iam v0.1.0 -> v0.1.1
  - 3d851c8deb4eb12c34b0eaf62a75b2151bcf5b14 roll cloud.google.com/go/compute v1.0.0 -> v1.2.0
  - bc8a70fde9c766f76ae936c0ae0c69e1dc6f728e use protoc 3.19.4
  - d150709be93a0eb30819b280742fd54f187179ff Revert "goma-rbe-chromeos: short timeout at first and lon...
  - b879f701ebb7c3aed8b154ae78e6c8a95a748d80 fix nsjail seccomp filter for go runtime
  - ccd9179cd96fb4eb8209a9d603b5cc72136fc9cb Reland "roll google.golang.org/grpc v1.42.0 -> v1.43.0"
  - 219f98542fc803207b5d0e56cd7115011b4d1759 Revert "Revert go to 1.17.3"
  - 52c2cc75ea051aa9155aeb22bbc09a1fa9d1b1f8 Revert "Revert google.golang.org/api and code.google.com/...
  - b8f310b265b7939fde220130569bbf7bbcd55d90 Reland "roll github.com/gomodule/redigo v1.8.5 -> v1.8.8"
  - 0f88e2475f450b12ea95152f27f17f88003ef75d Reland "roll go.uber.org/zap v1.19.1 -> v1.20.0"
  - c6af554c078ba48adcde20428bf26cd966052ecb remoteexec: fix retry; need to upload new action message
  - a3db180af0f0c49e9386752a5c60533435412a63 remoteexec: fix retry for deadline exceeded in short time...
  - 8608551b026c1b630d1736f7aab174d91b209d78 remoteexec: fix exec-action-timeout-for-retry
  - b1608fa71fe26d8ffd63789006fdad397d35d9f4 goma-rbe-chromeos: short timeout at first and long timeou...
  - e4ab28294d6b1ec8845ec5e9bb1e25f0524f3018 use shorter exec-action-timeout in goma-rbe-chromeos
  - a39825f9cd190cdba48b196364441acf45c29d78 Revert "roll go.uber.org/zap v1.19.1 -> v1.20.0"
  - d2f1b12af9b50581662a8478b2dd8a62ebc0f1b7 Revert "roll github.com/gomodule/redigo v1.8.5 -> v1.8.8"
  - 1eaf8fd61acaaceb845c01e4b8a5e019be0abb68 Revert google.golang.org/api and code.google.com/go/*
  - 94f88a00c53bf67776a4465d6169a9c047a6346f Revert go to 1.17.3
  - 5f1deef096abed855db89dc250a70c3b85889363 Revert "roll google.golang.org/grpc v1.42.0 -> v1.43.0"
  - 0abff47dd4435a6d65de1ec6c9d6e0f816bc94aa use protoc 3.19.3
  - 50678e6b56ff13145cf4e83a9382fc74bdb1623e Use go 1.17.6
  - 61d2d120f493dc2bb5250662fdee2b5f5f3d980c roll google.golang.org/api v0.64.0 -> v0.65.0
  - c54785ecdef8ef51532bfee41da7ad27c23858c9 roll cloud.google.com/go/compute v0.1.0 -> v1.0.0
  - 279f9970d3bc550896497567f5263ab10ec4d677 roll google.golang.org/api v0.63.0 -> v0.64.0
  - b350304b928a1fa30aa67e9507e8a640dde37f25 roll go.uber.org/zap v1.19.1 -> v1.20.0
  - 7c174810172cd2f06e69d64a4c283be3377741ab roll github.com/gomodule/redigo v1.8.6 -> v1.8.8
  - 486bcef9439acd6502466502379b6f47655f282a roll cloud.google.com/go/profiler v0.1.1 -> v0.1.2
  - 43e1e4058b624983b80a58da87caa18fa75cdccd roll cloud.google.com/go/monitoring v1.1.0 -> v1.2.0
  - 0f4f7d5996d4ac5a5e1174cb7db508ec069438d3 roll cloud.google.com/go v0.99.0 -> v0.100.2
  - e8dd05a2e743e6d9d8e0a2346852bcc257ada21b Allow chdir, getpeername, nanosleep, readlinkat, sched_ge...
  - 94436a9465929705a51e824864dfec94f6e2c86c roll google.golang.org/grpc v1.42.0 -> v1.43.0
  - bdc9327244f5057979655dd96395f45e3558f169 roll google.golang.org/api v0.61.0 -> v0.63.0
  - c1a2ee5cd6d611c1be477aa4a45afaabc2e4630c Use go 1.17.5
  - 40a59adddbe4b43b4d086345fde5d59c7d033343 goma: reset memory buffer when retrying download of std{o...
  - 04c0cd99731aa071f851c8a461a6ef68ea333ba2 roll cloud.google.com/go v0.98.0 -> v0.99.0
  - 11e603260806b4ed5b51153cedbbcf66baf4b415 roll cloud.google.com/go v0.97.0 -> v0.98.0
  - e192588e60d48a5d2ebb3434304f1e94b05e0049 Use go 1.17.4
  - d04e61dd0dacc9c8674d99cc75bf7c37662858a6 roll google.golang.org/api v0.60.0 -> v0.61.0
  - b01f2e603ad4bdeab1053c53c444b3821a0b67c7 roll github.com/gomodule/redigo v1.8.5 -> v1.8.6
  - 769d3962df15d0428c804ad4d89602223c1cf8ce use exected output paths to detect exec root
  - 5f6b76397c15fd6817ab7d9438eb0c6470dc82dd allow newfstatat
  - 171fe73b7794306bca27313a1068f1b023ad7ea8 Use go 1.17.3
  - 316056642031b027287437f32ad11c1e0e58ffbc roll google.golang.org/grpc v1.41.0 -> v1.42.0
  - cec6dc4bbf9758601113066ed1d3495b8a620480 roll contrib.go.opencensus.io/exporter/stackdriver v0.13....
  - 7b70852bad8128c609bd70b8af30cefb4a74625b roll protoc 3.19.0 -> 3.19.1
  - 88c2b0d40fa876a464a4b74ee86ce945a5a8bff7 roll google.golang.org/api v0.59.0 -> v0.60.0
  - 4b7bdb9712eb5039f6725836a3ede85bc1dce151 update go version in go.mod

GitOrigin-RevId: bfd9edf5c5936f8f045cdafc5e75cd00213ba64d
Change-Id: If2b6fc9c71cad30bd011d29d80fb8d9615e90313
37 files changed
tree: 819b410733c5621076aadba718603a26c6d54713
  1. auth/
  2. backend/
  3. bytestreamio/
  4. cache/
  5. cmd/
  6. command/
  7. exec/
  8. execlog/
  9. file/
  10. frontend/
  11. fswatch/
  12. hash/
  13. httprpc/
  14. infra/
  15. log/
  16. profiler/
  17. proto/
  18. remoteexec/
  19. rpc/
  20. server/
  21. .gitallowed
  22. .gitignore
  23. buildsetup.sh
  24. cipd_manifest.txt
  25. cipd_manifest.versions
  26. CONTRIBUTING.md
  27. go.mod
  28. go.sum
  29. LICENSE
  30. OWNERS
  31. README.md
README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's internal Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image would be something like

FROM marketing.gcr.io/google/ubuntu1804:latest
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -y update \
  && \
  apt-get install -f -y build-essential lsb-release python \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --allowed-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o