| commit | 1723a31851df9684b23e08481b96f9b4cc9234c5 | [log] [tgz] |
|---|---|---|
| author | Goma Authors <noreply@google.com> | Mon Aug 26 04:50:37 2019 |
| committer | Fumitoshi Ukai <ukai@chromium.org> | Tue Aug 27 08:23:15 2019 |
| tree | eec3ed20288ead79ef6423c01e2af46e48209269 | |
| parent | fccf18d00cfc7ce9f6eca4d69c9e86e3e687e6d1 [diff] |
Import changes for goma server - 385321ccbc2988a9c4aedfd8d14576685c862edd server: use debug.FreeOSMemory instead of runtime.GC - 42d70fca0b08e96a1131fcf1b2e2f0ad013da146 file-server: run gc if it closes to memory limit - 9429c487a29f3ea81339f7a95319f1e926e0b511 roll cloud.google.com/go v0.43.0 -> v0.44.0 - 3f170eff09fc736e254180059e8e3cc48fc729ab Revert "gcs: use minimum object attrs" - cac1c4d83526f60224aba62a8b5cd8bac331f039 roll google.golang.org/api v0.7.0 -> v0.9.0 - 01fdf185317e4caf233dcce5712fb8dc32b399ea gcs: use minimum object attrs - 14ec7e530bfe82ff5d656b422d00323955300f95 [inventory] Fix a potential DoS bug in pickCmd - a52be572c6807d30456e825e5a7698362d968562 [descriptor] Add dart toolchain support - 75e121b560bdd00623d861a09c0857cf0ad8fc5d memroy -> memory - 9c1f9e3da7dc4e30a92dd2aaf0cbd29ea852ca15 roll google.golang.org/grpc 1.22.1 -> 1.23.0 - 134589fd7a3ebd74517e286897d58b44480a26ad use go 1.12.8 - edde7c0ba6454173f3ec5a2d74c81f6d4a7528e0 remoteexec: retry for status Aborted - 8e18012b0006f36ff3020a3aadb92ce456f4c35c command: pubsub-error uses last value - 2610736422dbd6900481649d871480e3afb5e2c5 command: reports pubsubErrors=0 if pubsub is available - aa10576e7eec25a69153ec4c5d7128223914cec3 command: set expiration policy to subscriptions - 0a93bf24a2eef1314f7e419e739161ca843aa82a remoteexec: retry if aborted but caller ctx is active. - fd6b433ad85171b8c2d8a07694ec9e7c2f810399 command: fallback poller if subscription is not available - f020d98f49a687a2b67d21ddecc3cd62a7cc4a3a roll google.golang.org/grpc 1.22.0 to 1.22.1 GitOrigin-RevId: 385321ccbc2988a9c4aedfd8d14576685c862edd Change-Id: Ida45d03d064ba365a882f8d4ed0ba3d5111704ac TBR=yyanagisawa@google.com, tikuta@google.com, sque@google.com
Goma is a distributed compiler service for open-source project such as Chromium and Android. It's some kind of replacement of distcc+ccache.
This is reference implementation of server code to be used with Goma client.
The Goma server uses a backend service that implements the Remote Execution API to distribute compile requests across a collection of worker machines and to cache the results of compilations. The Remote Execution API is an open-source standard, with multiple service implementations. The Goma server has been tested with Google's Remote Build Execution service, but could use other service implementations with some minor tweaks to the service code.
To learn more about Remote Build Execution, which is currently an alpha service, you can view the whitelisted documentation by joining this group. If you would like to try using Goma with Remote Build Execution, you can follow these instructions. (note: you must join the documentation whitelist group to view the instructions).
Goma server can be built on Linux.
$ GO111MODULE=on go get go.chromium.org/goma/server/cmd/remoteexec_proxy
You will get the binary in $(go env GOPATH)/bin.
remoteexec_proxy is a single server that acts as proxy server between Goma client and Remote Execution API.
$ remoteexec_proxy --port $PORT \ --platform-container-image "docker://...@sha256:..." \ --remoteexec-addr $REMOTEEXEC_ADDR \ --remote-instance-name $REMOTE_INSTANCE_NAME
for chromium, platform container image should be created with install-build-dep.sh.
Example Dockerfile
FROM ubuntu:14.04
ENV DEBIAN_FRONTEND noninteractive
ADD https://chromium.googlesource.com/chromium/src/+/master/build/install-build-deps.sh?format=TEXT /tmp/install-build-deps.sh.base64
RUN base64 -d /tmp/install-build-deps.sh.base64 > /tmp/install-build-deps.sh \
&& \
apt-get -y update \
&& \
apt-get install -f -y build-essential gcc-4.8 g++-4.8 docker.io \
&& \
bash /tmp/install-build-deps.sh --lib32 --no-prompt \
--no-chromeos-fonts \
&& \
rm -rf /var/lib/apt/lists/*
If Remote Execution API requires service account, specify service account JSON file for Remote Execution API by --service-account-json.
Running user is granted by default. If you need to allow other users, you need to specify them by --whitelisted-users.
Log messages will be output to stderr. OpenCensus z-pages ara available at http://localhost:$PORT/debug/
Install goma client. We provide prebuilt binary with cipd, which is available in depot_tools.
$ cipd install infra/goma/client/linux-amd64 -root ${HOME}/goma
or follow the build instructions to build your own local version of Goma client before running the server code, and install it in $HOME/goma.
Need to authenticate Goma client before use.
$ $HOME/goma/goma_auth.py login
Specify hostname in $GOMA_SERVER_HOST and port in $GOMA_SERVER_PORT, along with a few other environment flags.
$ export GOMA_SERVER_HOST='host-of-remoteexec_proxy-running' $ export GOMA_SERVER_PORT='port-of-remoteexec_proxy-running' $ export GOMA_USE_SSL=false $ export GOMA_ARBITRARY_TOOLCHAIN_SUPPORT=true
For example, if you are running remoteexec_proxy locally with --port 5050, use:
$ export GOMA_SERVER_HOST=localhost $ export GOMA_SERVER_PORT=5050
Finally, start Goma client:
$ $HOME/goma/goma_ctl.py ensure_start
and in chromium tree.
$ rm -f out/Release/obj/base/base/base64.o $ GOMA_USE_LOCAL=false autoninja -C out/Release obj/base/base/base64.o