| commit | fccf18d00cfc7ce9f6eca4d69c9e86e3e687e6d1 | [log] [tgz] |
|---|---|---|
| author | Goma Authors <noreply@google.com> | Tue Jul 23 07:21:39 2019 |
| committer | Fumitoshi Ukai <ukai@chromium.org> | Tue Jul 23 14:00:50 2019 |
| tree | 7bf546e2e59e7b0c16c3ec518942edcd0b0ca592 | |
| parent | 02b31977fcb75117c8bb0ed43c837b847ff1aa1e [diff] |
Import changes for goma server - a702111367735f8c0a46225ea9146dc57b3ba713 Should refer env_file_for_docker in WORK_DIR instead of c... - 9ee4ac3baa2bfe9a6c51292aa04763b31a6371a9 remoteexec: ExecuteAndWait returns error of resp status. - 18366bdf014af5e6e7cb79ed274ac86ad90182f0 remoteexec: don't add . for wrapper on windows - 7ee036e7a320085d751425b91f5fafd2949496bb remoteexec: fix retry by ExecuteResponse status - bc9eb1ff4b5995e32bedc0626580c1322b21dab2 roll cloud.google.com/go from 0.41.0 to 0.43.0 - 4bc2b91a156363be0d2404c150fd787d808d375d remoteexec: update request metadata name - c670f55ebb2dd5f2cb9ae58b406109f6424a212c User environment should directly goes to where the compil... - e839bb6deafc14cc6ddbd2390a9f77923b85ed34 roll github.com/golang/protobuf from 1.3.1 to 1.3.2 - c8c02f1a5bd9f80f73dffd1d6599a8288d7daac3 use go 1.12.7 - 8c85c6baeac7addfda80134578d2ac5304837ab7 Use "pwd -P" instead of pwd to get current working direct... - b2d77a97fd319e5bec7e8de559a79fd895bbc018 roll google.golang.org/grpc from 1.21.1 to 1.22.0 - a506584bd95b470c908698c1be2ecb94d2620aa0 roll cloud.google.com/go from 0.40.0 to 0.41.0 - 96a8f21ee37900a200b3e7021d7b3cdb95e17035 remoteexec: reduce log volume - c713279d342b08cffe62fb266c6890c9fd46598b remoteexec: set rpc id - 8640543157f9b1cab6be2e9b5970101c248f0329 remoteexec_proxy: add document how to deploy it on appeng... - f9dc9f386b2dc522458f68e71af052f88a184f00 nsjail: add dummy HOME directory. - 64808c917652706de338b858213626dda987b17c nsjail: make /dev/null available. - f0a4ba0be368a4edcbd259193d725b8700418914 nsjail: set up PATH env from ExecReq::ToolchainSpec. - f1a01e793953c325c69576f75af614b9bbe03174 roll google.golang.org/api from 0.6.0 to 0.7.0 GitOrigin-RevId: 509e19eb587f2a5deb66290e1e70b5665b9a50d5 Change-Id: I7c08d57a3bbe51ff6c199228308ff13c0f320a49 TBR=yyanagisawa@google.com, tikuta@google.com
Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.
This is reference implementation of server code to be used with Goma client.
The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.
To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).
Goma server can be built on Linux.
$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy
You will get the binary in $(go env GOPATH)/bin.
remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.
$ remoteexec_proxy --port $PORT \ --platform-container-image "docker://...@sha256:..." \ --remoteexec-addr $REMOTEEXEC_ADDR \ --remote-instance-name $REMOTE_INSTANCE_NAME
for chromium, platform container image should be created with install-build-dep.sh.
Example Dockerfile
FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive
ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
&& \
apt-get -y update \
&& \
apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
&& \
bash /tmp/install-build-deps.sh --lib32 --no-prompt \
--no-chromeos-fonts \
&& \
rm -rf /var/lib/apt/lists/*
If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.
Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.
Log messages will be output to stderr. OpenCensus z-pages ara available at http://localhost:$PORT/debug/
Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.
$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma
or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.
Need to authenticate Goma client before use.
$ $HOME/goma/goma_auth.py login
Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.
$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running' $ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running' $ export GOMA_USE_SSL=false $ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true
For example, if you are running remoteexec_proxy locally with --port 5050, use:
$ export GOMA_SERVER_HOST=localhost $ export GOMA_SERVER_PORT=5050
Finally, start Goma client:
$ $HOME/goma/goma_ctl.py ensure_start
and in chromium tree.
$ rm -f out/Release/obj/base/base/base64.o $ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o