| commit | ad3905624b89653ace5b881f1eb9ec979874d8b0 | [log] [tgz] |
|---|---|---|
| author | Goma Authors <noreply@google.com> | Thu Oct 24 05:39:26 2019 |
| committer | Fumitoshi Ukai <ukai@chromium.org> | Fri Oct 25 02:48:58 2019 |
| tree | f9de6b5be776f834b6ee29e7f75c71cd930249b0 | |
| parent | 14815ae6e9457f5ac202a7c3cbf402cec4843bb0 [diff] |
Import changes for goma server - 7f124bd0a4bdab90f1b96cd6738ea17f996d770f remoteexec: use buffered chan for inputFiles - fbb49a10cb8125a4e9d35aa029850f7fcbfe0520 remoteexec: refactor inputFiles - d1f837135f61e4465841a93174ee1de0fd7f7663 roll cloud.google.com/go/storage from v1.0.0 to v1.1.2 - 348fad9894e7d5b3e7cf4ef0428b562fd783f43a use go 1.13.3 - 7a25cb8485ac1e1fe05cba2087554f93d8cddbf5 roll cloud.google.com/go from v0.46.3 to v0.47.0 - b2b454123035e98568ada5b3792f682af106a763 cas: use sync.Pool for cas upload/download - 542b209958867a8fa132f013521b51762c29db7b merkletree: improve filepath join - 01ab083e39d5543d665b309f78bc79aa198380a6 remoteexec: rootRel expect cwd and rootDir are already cl... - 06aacfce418ac33f74935b9857cc5f1edfdc96e6 posixpath: eliminate growslice in Join - 2a1a31055a151d38a3d108b9dd6db36344cd1238 remoteexec: retry for RBE internal error. - 0f0601a06e4639de08ce4aca8353ee8b11d3e4b9 posixpath: eliminate growslice in SplitElem - ace54e6c77246b8944477d1e9d6a2f48ee12fd68 file: remove debug log - cb0f3fc8f0fc139e193c5783f6328dc96eed612c remove log detected by cloud profiler - 6b9a0958b2c2e31dab25b67204c86c796d9dca5a winpath: improve Clean - 28564e15a45c2c016816badaffd58b50444f415d posixpath: benchmark for abs path case - c77498427f070ac03d6e8bff53496dca53456ec9 roll google.golang.org/api from v0.10.0 to v0.11.0 - 32f988427805a67e8f12f819649b6c86381e3ff3 posixpath: improve Clean - 697b3728a270469b90d2c6b1fb9378b7ac763690 Make loadtest work - 0c7e9c96e2e745270f526fe40eb43b4158517d61 remoteexec_proxy: Set OSFamily - f178b358c6ffe5c24ee0b9133a42928ef76332c5 [dartanalyzer] Fix a bug that setup_cmd does not honor se... - a375b14e2bb5e4cd5f1addfb380085ccab044a2c auth: Print more detailed logging about email/domain mism... - d54ae14d159897ea0892c58147832677f29e238b log: remove unnecessary initialization log - 636b63aa13b88a5d29b740893499595617aec00a use go 1.13.1 TBR=yyanagisawa@google.com, tikuta@google.com, sque@google.com, yekuang@google.com GitOrigin-RevId: 7f124bd0a4bdab90f1b96cd6738ea17f996d770f Change-Id: Ibea59eee65481c82a4f310f23effc927611a4e33
Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.
This is reference implementation of server code to be used with Goma client.
The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.
To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).
Goma server can be built on Linux.
$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy
You will get the binary in $(go env GOPATH)/bin.
remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.
$ remoteexec_proxy --port $PORT \ --platform-container-image "docker://...@sha256:..." \ --remoteexec-addr $REMOTEEXEC_ADDR \ --remote-instance-name $REMOTE_INSTANCE_NAME
for chromium, platform container image should be created with install-build-dep.sh.
Example Dockerfile
FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive
ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
&& \
apt-get -y update \
&& \
apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
&& \
bash /tmp/install-build-deps.sh --lib32 --no-prompt \
--no-chromeos-fonts \
&& \
rm -rf /var/lib/apt/lists/*
If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.
Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.
Log messages will be output to stderr.
Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.
$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma
or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.
Need to authenticate Goma client before use.
$ $HOME/goma/goma_auth.py login
Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.
$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running' $ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running' $ export GOMA_USE_SSL=false $ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true
For example, if you are running remoteexec_proxy locally with --port 5050, use:
$ export GOMA_SERVER_HOST=localhost $ export GOMA_SERVER_PORT=5050
Finally, start Goma client:
$ $HOME/goma/goma_ctl.py ensure_start
and in chromium tree.
$ rm -f out/Release/obj/base/base/base64.o $ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o