| commit | 030736e0c390c356f8361c64a9a45e4111f765cc | [log] [tgz] |
|---|---|---|
| author | Goma Authors <noreply@google.com> | Thu May 23 04:06:43 2019 |
| committer | Fumitoshi Ukai <ukai@google.com> | Thu May 23 05:27:27 2019 |
| tree | e536f22a51d50e99d9a29f7c98d0d8e683a52e95 | |
| parent | 48224f31513604484d7c74afa5fd9d7e621e1691 [diff] |
Import changes for goma server - a6135e0847e80da878fee74e1df115d09aaef398 [remoteexec] use bytes.Equal - a52f45b457a5e627aaa34e1e7134221b8f34c896 Run with chroot if needed and allowed. - 389164d8293b716fbddf91468c4c4e6af07d3fbd mixer: clients5.google.com should not be default - a4ef9cd20704d8abf7023deb7723b8564b7c464d descriptor: allow new clang-cl version string - 9409d5f7df1f1e0865525cedb5bd93049e12eb8d goma_replay: use oauth2/v2 for tokeninfo - cc18352ff9233b7be1215d08c84446c3ac6fa36a goma_replay: add service account support - 305013abf6d560d643e2df792e52abdd9b352ee7 try hard not to do multiple DNS requests for the same host - cd10dea2c680ea41a0e4acfcdc47188bcd1379ca Fix typos in goma_replay and mixer-dev README - cca3bcde3ecaf368b9766725727678adba8b2f88 goma_replay: resolve hostname once - 9f6a10623f93760bf71d86055bda907d025b905f use gofmt -s - 2165cf292e73a4881d3bb3d7f2be2544efa9d4b1 remoteexec: use run-at-dir.sh if exists on container image. - 4d1f6983a108617312b84d9bbbdd1fa8de5e483b cloudbuild: Use default RBE instance for Windows - 7eaa7683584c99cbfa950a568664b20cbf4066a7 add buildsetup.sh for docker-less build - e885905a4a25c6e7cf268f15a93f30e1f4e7c589 mitigate grpc timeout - fffa2c244b55a9f834e1a4d64f7fde411bb6f725 remoteexec_proxy: add -insecure-remoteexec - 30df1ff5e37b9d9e7239a259c19b59a5c01257d5 roll cloud.google.com/go 0.38.0 to 0.39.0 - a84976e9b58a46afdec04db5e20ecd398358dbd7 remoteexec: fix wrong missing input report - 78737b28b8afa248441b3921ac7d71d05d44f8c1 roll google.golang.org/api from 0.4.0 to 0.5.0 - 91624ddad5e3abb02a0ae40d0d6e5d6c877801eb [proto] remove some ptotobuf protos - 29677cfa48530035517c12b64b2c49bbfd777e65 update zap to v1.10.0 - a0419fa35a7622201412b662afba287143358237 [proto] remove unused google api proto GitOrigin-RevId: a6135e0847e80da878fee74e1df115d09aaef398 Change-Id: Ic1aa40ae7c4fdf71154e5509d2a0843c6e3750d3
Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.
This is reference implementation of server code to be used with Goma client.
The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.
To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).
Goma server can be built on Linux.
$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy
You will get the binary in $(go env GOPATH)/bin.
remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.
$ remoteexec_proxy --port $PORT \ --platform-container-image "docker://...@sha256:..." \ --remoteexec-addr $REMOTEEXEC_ADDR \ --remote-instance-name $REMOTE_INSTANCE_NAME
for chromium, platform container image should be created with install-build-dep.sh.
Example Dockerfile
FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive
ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
&& \
apt-get -y update \
&& \
apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
&& \
bash /tmp/install-build-deps.sh --lib32 --no-prompt \
--no-chromeos-fonts \
&& \
rm -rf /var/lib/apt/lists/*
If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.
Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.
Log messages will be output to stderr. OpenCensus z-pages ara available at http://localhost:$PORT/debug/
Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.
$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma
or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.
Need to authenticate Goma client before use.
$ $HOME/goma/goma_auth.py login
Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.
$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running' $ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running' $ export GOMA_USE_SSL=false $ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true
For example, if you are running remoteexec_proxy locally with --port 5050, use:
$ export GOMA_SERVER_HOST=localhost $ export GOMA_SERVER_PORT=5050
Finally, start Goma client:
$ $HOME/goma/goma_ctl.py ensure_start
and in chromium tree.
$ rm -f out/Release/obj/base/base/base64.o $ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o