commit	7d22a579ed3171bfd23e9a89a693f4c406fc878f	[log] [tgz]
author	Goma Authors <noreply@google.com>	Mon Feb 03 01:53:56 2020
committer	Fumitoshi Ukai <ukai@chromium.org>	Mon Feb 03 03:43:38 2020
tree	7188a599d3a8b4a9b5fdcd2bd4c114e6545c733b
parent	5563502754bd6d0bfe2c10e2f3610b61ac45e645 [diff]

Import changes for goma server

  - fc28b776ba28f56117aae335f7bef710da45c72b remoteexec: fix TestUploadInputFiles
  - c327a441b0510fd4f623f09050d01f59987db0dd use go 1.13.7
  - b0293a342ce0f3acc02c9862b740b565ad4e082d roll github.com/golang/protobuf v1.3.2 to v1.3.3
  - c6cbbe89b235735691b6e962484c6b965b5b1600 merkletree: remove .keep_me hack
  - 8382880bd530582d3d992dab035986930568d6c5 roll google.golang.org/grpc v1.26.0 to v1.27.0
  - 726253c4d73965be922bf4eda1b1a06a40e5143b remoteexec: Refactor uploadInputFiles
  - 942f049ddbd7ef23af1c87edf7e1c816eab6d470 remoteexec: uploadInputFiles returns errors
  - 0e97e016329d602d9f4395a9ed89fd2bbf53c65e remoteexec: upload files to file-server in batches
  - a1ba3f8a84a39e4bf8167e247cea6174b14795f3 remoteexec: gomaInput supports multiple blob upload
  - 66a80acaab9b5fdc2a44a67f1427e9667e8d82c9 remoteexec: Reduce message of successful inputFiles
  - 2a4b7380727fe086c34481bd23290d5b3e3285fc use go 1.13.6
  - 7856755c29d0dc76b9a789ed461a7a791bb7e206 remoteexec: Simplify result creation in inputFiles()
  - 55dfe14632e34f402875b5ec4026441cb7a155a9 exec_server: file-lookup-concurrency defaults to 20
  - c7f1560b5f7c94159db6c30cf6ef2c22907048fe roll cloud.google.com/go from v0.51.0 to v0.52.0
  - 144c70591ee7f3692be709d03d73bfea4fa18aad remoteexec/cas: Check BatchUpdateBlobsRequest proto size
  - bc8fe0340ac23c837b418375baf4dc1332abd111 exec_server: Reduce cas file lookup concurrency 100 -> 20
  - a19839bc86fef4aea7bc97b6970926f44c0f2d49 remoteexec/cas: Fix typo in cas_test.go
  - e9d616740893349e6f937861cd1442bc00666ff2 remoteexec/cas: Limit CAS batch updates to 1000
  - 60e5ee5e8155de5e9adb073611685b004223daf8 Allow the audience string stored in the git repository.
  - 0d349f7415cf3f2de74a423c5f3e0b4de3ab1b3b merkletree: duplicate symlinks/dirs are not error
  - d812400ff13d45e07b921abe508ff4ea3c424e86 roll contrib.go.opencensus.io/exporter/stackdriver v0.12....
  - 372222e44e945b700fb584b44bc59f731b14b0a5 remoteexec: number of requests per wrapper types
  - 2d381dbf4c4bbb2d4e94cbc8e2f120b15981b0f4 remoteexec: Adapter-wide sema for CAS lookupBlob
  - 7036cf5ad607cefa6bcd278bdd2c6ae35d3d483d remoteexec: Create FileLookupSema in Adapter
  - bff170368f7e2657f502e5015692698a616c359f remoteexec/cas: Concurrent blob lookup in file store
  - 7c632fe404f065740bc6c0924cfe9a8554e0e3c9 remoteexec/cas: Use RBE's fakes.CAS for testing
  - 70f2c93d85b296c3b6e8b7e096061d19f7207899 remoteexec: Support javac cwd agnostic check
  - 7f14b767f9c42e68d47e9274ac489048f111e433 remoteexec/cas: Pass struct ptr to struct functions
  - 0f94a6b3421da5d6a38640e85d7b259fbd7c159c remoteexec: Log CAS errors and return early
  - 6b537728242f687d779a921def88c3d773bd4636 remoteexec/cas: helper funcs for blob lookup and batch re...
  - c59b1b3664c2a31c79711fc0c856070070399828 roll cloud.google.com/go/storage v1.4.0 to v1.5.0
  - 6709ea49a3e6d0291a4265c895367fce513f6707 remoteexec/cas: Explicitly separate small/large blobs
  - 2c830147f02e4d45839ce6c4a9e36db3789bb577 remoteexec/cas: test CAS.Upload()'s bytestream upload
  - ae5b7f95c67047a429e9c07bc3106ac3b0461b61 remoteexec/cas: Add unit test for CAS.Upload()
  - 9b68e7167f149557163c0065312699c684b2462e remoteexec/cas: Add unit test for CAS.Missing()
  - 317c004525e555ae24ec59834e6ca544df42f2f2 server: drop trace fraction and qps
  - d17057ac9a59dde2e82480eeb03d17e762533bcf Reland "Reland "remoteexec: don't use sibling docker""
  - 3a11774b46ccc6c3d257740636aff632e6f9a4a8 Revert "Reland "remoteexec: don't use sibling docker""
  - 35ae2c4ac6887957307dfa81cec0e75832abae6a roll cloud.google.com/go v0.50.0 to v0.51.0
  - 30d950518d05977c41c8f0d1afed15d30248c2fd Reland "remoteexec: don't use sibling docker"
  - ee5243d8531cde95846e659287696b702a46be56 server: default trace fraction down from 1.0 to 0.5
  - c6a98a4d29879a9accb756d29ebb1939d66ccb8e normalizer: fix for armv7a-cros-linux-gnueabihf
  - b978c1c4087b012b48c479662d94d1df4fc8e3cb Revert "remoteexec: don't use sibling docker"
  - 3b20f501fd195694f4108f63ce2c1681502d4968 Revert "remoteexec: fix nsjail option"
  - a997ba139d9cac1b90b794dd93672f605f3d0b33 Revert "remoteexec: fix nsjail"
  - 852897bf7d1b7994f8846aeaaf24d006d16abb81 remoteexec: fix nsjail
  - 2a6bc4a4acc88175eb6eb6222abc14f09abb58df remoteexec: fix nsjail option
  - 994b7a4f97707a280c672fa8544ee6879112032b remoteexec: don't use sibling docker
  - 257f9c9c1cab29d6cad7e3401878197dbb57ee53 roll cloud.google.com/go v0.49.0 to v0.50.0
  - 0e5482a738df170bda7df8fbfcb6e2aa433d31d7 roll google.golang.org/grpc v1.25.1 to v1.26.0
  - a13936c9478838f6766a0ddc4fae7a212f32c7ad Revert "remoteexec: don't use sibling docker"
  - 17aff5222ce72a5776bde4064cd3d34e7652905d remoteexec: don't use sibling docker
  - 727ffd963e02aee0584c73b27cc8171e413416f2 toolchain.config managed in k8s configmap
  - 8cc8f1910fb0ea38e0bc94e1d891be4bb8aed378 remoteexec: introduce wrapperType
  - 4f96a9dd2f867760d7d920b5d655f7297c7e63f9 remoteexec: Move OpenCensus stats to stats.go
  - b459260bd395a891db9d27d5140bed59feddaf50 use go 1.13.5
  - 6959d4be1f0147defa862bfd36a5ed76eb4e1a65 Revert "command: pubsub-error uses last value"
  - 854a0bf7465499ff814027b61a321ecc523dd1a7 use trace limited sampler for all servers
  - ca0b8aefe7411371a79938ea634b232c62674898 Revert "use remote sampler for all servers"
  - 8dcbc3283bb16ebea3f9eb0e97eb6ccc18dc99fa Support -fdebug-compilation-dir when checking CWD agnostic

GitOrigin-RevId: fc28b776ba28f56117aae335f7bef710da45c72b
Change-Id: I631bb5122e4647e9f9dcf52830cfb0e3cd248bf1
TBR=yyanagisawa@google.com, sque@google.com, yekuang@google.com

42 files changed

tree: 7188a599d3a8b4a9b5fdcd2bd4c114e6545c733b

README.md

Goma server

Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.

This is reference implementation of server code to be used with Goma client.

Dependencies

The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.

To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).

How to build

Goma server can be built on Linux.

$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy

You will get the binary in $(go env GOPATH)/bin.

How to run

remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.

$ remoteexec_proxy --port $PORT \
   --platform-container-image "docker://...@sha256:..." \
   --remoteexec-addr $REMOTEEXEC_ADDR \
   --remote-instance-name $REMOTE_INSTANCE_NAME

for chromium, platform container image should be created with install-build-dep.sh.

Example Dockerfile

FROM ubuntu:16.04
ENV DEBIAN_FRONTEND noninteractive

ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
  && \
  apt-get -y update \
  && \
  apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
  && \
  bash /tmp/install-build-deps.sh --lib32 --no-prompt \
     --no-chromeos-fonts \
  && \
  rm -rf /var/lib/apt/lists/*

If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.

Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.

Log messages will be output to stderr.

How to use

Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.

$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma

or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.

Need to authenticate Goma client before use.

$ $HOME/goma/goma_auth.py login

Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.

$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running'
$ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running'
$ export GOMA_USE_SSL=false
$ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true

For example, if you are running remoteexec_proxy locally with --port 5050, use:

$ export GOMA_SERVER_HOST=localhost
$ export GOMA_SERVER_PORT=5050

Finally, start Goma client:

$ $HOME/goma/goma_ctl.py ensure_start

and in chromium tree.

$ rm -f out/Release/obj/base/base/base64.o
$ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o