| commit | f6089187c6580bc27cee25b01ff994a74ae8658d | [log] [tgz] |
|---|---|---|
| author | Ed Schouten <eschouten@apple.com> | Fri Oct 21 20:39:43 2022 |
| committer | GitHub <noreply@github.com> | Fri Oct 21 20:39:43 2022 |
| tree | 795d2d48c0ec7ea45bf8c5f2e9092977e8ccbcae | |
| parent | aa29b91f336b9be2c5370297210b67a6654c0b72 [diff] |
Add a hint for indicating that a Tree is topologically sorted (#230) * Regenerate the Go source code for the Remote Execution protocol * Add a hint for indicating that a Tree is topologically sorted I'm currently trying to improve the performance of handling of large output directories (Tree messages), having sizes in the order of hundreds of megabytes. In the process, I have realised that there is a lot of value in enforcing that the Directory messages contained in them are topologically sorted. Two practical use cases: - When instantiating the contents of a Tree on a local file system, having the Tree be topologically sorted allows you to immediately create files and directories in the right place. - When needing to resolve the properties of a single file by path, a topologically sorted Tree permits resolution by doing a simple forward scan. Especially when other features like compression are taken into account, it's useful if Tree messages can be processed in a streaming manner. One practical issue is that most Protobuf libraries don't offer APIs for processing messages in a streaming manner. This means that implementors who want to achive these optimisations will need to write their own message parsers; at least for the Tree tree itself. To make this as painless as possible, we also require that the Tree is stored in some normal form. Fixes: #229
This repository contains a collection of APIs which work together to enable large scale distributed execution and caching on source code and other inputs. It describes how to upload inputs, request the execution, monitor for results, and cache those results. It‘s overall aim is to enable large scale parallel executions that wouldn’t be feasible on a single system, while minimizing the amount of uploads and executions needed by storing data in a content-addressable format and caching results.
The Remote Execution API is an API that, at its most general, allows clients to request execution of binaries on a remote system. It is intended primarily for use by build systems, such as Bazel, to distribute build and test actions through a worker pool, and also provide a central cache of build results. This allows builds to execute faster, both by reusing results already built by other clients and by allowing many actions to be executed in parallel, in excess of the resource limits of the machine running the build.
The Remote Asset API is an API to associate Qualifiers and URIs to Digests stored in Content Addressable Storage. It is primary intended to allow clients to use semantically relevant identifiers, such as a git repository or tarball location, to get the corresponding Digest. This mapping may be pushed by a client directly, or dynamically resolved and added to CAS by the asset server when fetched by a client.
The Remote Logstream API is an API supporting ordered reads and writes of LogStream resources. It is intented primarily for streaming the stdout and stderr of ongoing Action executions, enabling clients to view them while the Action is executing instead of waiting for it's completion.
There are a number of clients and services using these APIs, they are listed below.
These tools use the Remote Execution API to distribute builds to workers.
These applications implement the Remote Execution API to serve build requests from the clients above. These are then distributed to workers; some of these workers implement the Remote Worker API.
The Remote Execution APIs group hosts discussions related to the APIs in this repository.
Interested parties meet monthly via VC to discuss issues related to the APIs, and several contributors have organized occasional meetups, hack-a-thons, and summits. Joining the email discussion group will automatically add you to the Google Calendar invite for the monthly meeting.
The APIs in this repository refer to several general-purpose APIs published by Google in the Google APIs repository. You will need to refer to packages from that repository in order to generate code using this API. If you build the repository using the included BUILD files, Bazel will fetch the protobuf compiler and googleapis automatically.
The repository contains BUILD files to build the protobuf library with Bazel. If you wish to use them with your own project in Bazel, you will possibly want to declare cc_proto_library, java_proto_library, etc. rules that depend on them.
Other build systems will have to run protoc on the protobuf files, and link in the googleapis and well-known proto types, manually.
This repository contains the generated Go code for interacting with the API via gRPC. Get it with:
go get github.com/bazelbuild/remote-apis
Import it with, for example:
repb "github.com/bazelbuild/remote-apis/build/bazel/remote/execution/v2"
Enable the git hooks to automatically generate Go proto code on commit:
git config core.hooksPath hooks/
This is a local setting, so applies only to this repository.