blob: e4d8b6a8db2b46fd88611cdc00ebe9814b57367e [file] [log] [blame]
// Copyright 2019 The LUCI Authors.
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// See the License for the specific language governing permissions and
// limitations under the License.
// Package luciexe documents the "LUCI Executable" protocol, and contains
// constants which are part of this protocol.
// Summary
// A LUCI Executable ("luciexe") is a binary which implements a protocol to:
// * Pass the initial state of the 'build' from a parent process to the luciexe.
// * Understand the build's local system contracts (like the location of cached data).
// * Asynchronously update the state of the build as it runs.
// This protocol is recursive; A luciexe can run another luciexe such that the
// child's updates are reflected on the parent's output.
// The protocol has 3 parts:
// * Host Application - This sits at the top of the luciexe process
// hierarchy and sets up singleton environmental requirements for the whole
// tree.
// * Invocation of a luciexe binary - This invocation process occurs both
// for the topmost luciexe, as well as all internal invocations of other
// luciexe's within the process hierarchy.
// * The luciexe binary - The binary has a couple of responsibilities to be
// compatible with this protocol. Once the binary has fulfilled it's
// responsibilities it's free to do what it wants (i.e. actually do its
// task).
// In general, we strive where possible to minimize the complexity of the
// luciexe binary. This is because we expect to have a small number of 'host
// application' implementations, and a relatively large number of 'luciexe'
// implementations.
// The Host Application
// At the root of every tree of luciexe invocations there is a 'host'
// application which sets up an manages all environmental singletons (like the
// Logdog 'butler' service, the LUCI ambient authentication service, etc.). This
// Host Application is responsible for intercepting and merging all
// 'build.proto' streams emitted within this tree of luciexes (see "Recursive
// Invocation"). The Host Application may choose what happens to these
// intercepted build.proto messages.
// The Host Application MUST:
// * Run a logdog butler service and expose all relevant LOGDOG_* environment
// variables such that the following client libraries can stream log data:
// * Golang:
// * Python: infra_libs.logdog.bootstrap
// * Hook the butler to intercept and merge build.proto streams into a single
// build.proto (zlib-compressed) stream.
// * Set up a local LUCI ambient authentication service which luciexe's can
// use to mint auth tokens.
// * Prepare an empty directory which will house tempdirs and workdirs for
// all luciexe invocations. The Host Application MAY clean this directory
// up, but it may be useful to leak it for debugging. It's permissible for
// the Host Application to defer this cleanup to an external process (e.g.
// buildbucket's agent may defer this to swarming).
// The Host Application MAY hook additional streams for debugging/logging; it is
// frequently convenient to hook the stderr/stdout streams from the top level
// luciexe and tee them to the Host Application's stdout/stderr.
// For example: the `` binary forwards
// these merged build.proto messages to the Buildbucket service, and also
// uploads all streams to them to the Logdog cloud service. Other host
// implementations may instead choose to write all streams to disk, send them to
// /dev/null or render them as html. However, from the point of view of the
// luciexe that they run, this is transparent.
// Host Applications MAY implement 'backpressure' on the luciexe binaries by
// throttling the rate at which the Logdog butler accepts data on its various
// streams. However, doing this could introduce timing issues in the luciexe
// binaries as they try to run, so this should be done thoughtfully.
// If a Host Application detects a protocol violation from a luciexe within its
// purview, it SHOULD report the violation (in a manner of its choosing) and
// MUST consider the entire Build status to be INFRA_FAILURE. In addition the
// Host Application SHOULD attempt to kill (via process group SIGTERM/SIGKILL on
// *nix, and CTRL+BREAK/Terminate on windows) the luciexe hierarchy. The host
// application MAY provide a window of time between the initial "please stop"
// signal and the "you die now" signal, but this comes with the usual caveats of
// cleanups and deadlines (notably: best-effort clean up is just that:
// best-effort. It cannot be relied on to run to completion (or to run
// completely)).
// Invocation
// When invoking a luciexe, the parent process has a couple responsibilities. It
// must:
// * Set $TEMPDIR, $TMPDIR, $TEMP, $TMP and $MAC_CHROMIUM_TMPDIR to all point
// to the same, empty directory.
// This directory MUST be located on the same file system as CWD.
// This directory MUST NOT be the same as CWD.
// * Set $LUCI_CONTEXT["luciexe"]["cache_dir"] to a cache dir which makes sense
// for the luciexe.
// The cache dir MAY persist/be shared between luciexe invocations.
// The cache dir MAY NOT be on the same filesystem as CWD.
// * Set the $LOGDOG_NAMESPACE to a prefix which namespaces all logdog streams
// generated from the luciexe.
// * Set the Status of initial buildbucket.v2.Build message to `STARTED`.
// * Set the CreateTime and StartTime of initial buildbucket.v2.Build message.
// * Clear following fields in the initial buildbucket.v2.Build message.
// - EndTime
// - Output
// - StatusDetails
// - Steps
// - SummaryMarkdown
// - UpdateTime
// The CWD is up to your application. Some contexts (like Buildbucket) will
// guarantee an empty CWD, but others (like recursive invocation) may explicitly
// share CWD between multiple luciexe's.
// The tempdir and workdir paths SHOULD NOT be cleaned up by the invoking
// process. Instead, the invoking process should defer to the Host Application
// to provide this cleanup, since the Host Application may be configured to leak
// these for debugging purposes.
// The invoker MUST attach the stdout/stderr to the logdog butler as text
// streams. These MUST be located at `$LOGDOG_NAMESPACE/std{out,err}`.
// Typical luciexe implementations will use these for debug logging and output,
// but are not required to do so.
// The invoker MUST write a binary-encoded buildbucket.v2.Build to the stdin of
// the luciexe which contains all the input parameters that the luciexe needs to
// know to run successfully.
// The luciexe binary
// Once running, the luciexe MUST read a binary-encoded buildbucket.v2.Build
// message from stdin until EOF.
// As per the invoker's responsibility, the luciexe binary MAY expect the
// status of the initial Build message to be `STARTED` and CreateTime and
// StartTime are populated. It MAY also expect following fields to be empty.
// It MUST NOT assume other fields in the Build message are set. However, the
// Host Application or invoker MAY fill in other fields they think are useful.
// EndTime
// Output
// StatusDetails
// Steps
// SummaryMarkdown
// Tags
// UpdateTime
// As per the Host Application's responsibilities, the luciexe binary MAY expect
// the "luciexe" and "local_auth" sections of LUCI_CONTEXT to be filled. Other
// sections of LUCI_CONTEXT MAY also be filled. See the LUCI_CONTEXT docs:
// !!NOTE!! The paths supplied to the luciexe MUST NOT be considered stable
// across invocations. Do not hard-code these, and try not to rely on their
// consistency (e.g. for build reproducibility).
// The luciexe binary - Updating the Build state
// A luciexe MAY update the Build state by writing to a "build.proto" Logdog
// stream named "$LOGDOG_NAMESPACE/build.proto". A "build.proto" Logdog stream
// is defined as:
// Content-Type: "application/luci+proto; message=buildbucket.v2.Build"
// Type: Datagram
// Additionally, a build.proto stream MAY append "; encoding=zlib" to the
// Content-Type (and compress each message accordingly). This is useful for when
// you potentially have very large builds.
// Each datagram MUST be a valid binary-encoded buildbucket.v2.Build message.
// The state of the build is defined as the last Build message sent on this
// stream. There's no implicit accumulation between sent Build messages.
// The Step.Log.Url field in the emitted Build messages can be either absolute
// (has a valid url scheme) or relative. If an absolute `Url` is provided, the
// luciexe is also responsible for supplying corresponding `ViewUrl` and the
// host application won't validate or make any adjustments to those urls. This
// allows luciexe to include logs from other sources into this build. If a
// relative `Url` is provided, the host application will namespace the `Url`
// with $LOGDOG_NAMESPACE of the build.proto stream. For example, if the host
// application is parsing a Build.proto in a stream named
// "logdog://host/project/prefix/+/something/build.proto", then a Log with a Url
// of "hello/world/stdout" will be transformed into:
// Url: logdog://host/project/prefix/+/something/hello/world/stdout
// ViewUrl: <implementation defined>
// The `ViewUrl` field in this case SHOULD be left empty, and will be filled in
// by the host application running the luciexe (if supplied it will be
// overwritten).
// The following Build fields will be read from the luciexe-controlled
// build.proto stream:
// EndTime
// Output
// Status
// StatusDetails
// Steps
// SummaryMarkdown
// Tags
// UpdateTime
// The luciexe binary - Reporting final status
// A luciexe MUST report its success/failure by sending a Build message with
// a terminal `status` value before exiting. If the luciexe exits before sending
// a Build message with a terminal Status, the invoking application MUST
// interpret this as an INFRA_FAILURE status. The exit code of the luciexe
// SHOULD be ignored, except for advisory (logging/reporting) purposes. The host
// application MUST detect this case and fill in a final status of
// INFRA_FAILURE, but MUST NOT terminate the process hierarchy in this case.
// Recursive Invocation
// To support recursive invocation, a luciexe MUST accept the flag:
// --output=path/to/file.{pb,json,textpb}
// The value of this flag MUST be an absolute path to a non-existent file in
// an existing directory. The extension of the file dictates the data format
// (binary, json or text protobuf). The luciexe MUST write it's final Build
// message to this file in the correct format. If `--output` is specified,
// but no Build message (or an invalid/improperly formatted Build message)
// is written, the caller MUST interpret this as an INFRA_FAILURE status.
// NOTE: JSON outputs SHOULD be written with the original proto field names,
// not the lowerCamelCase names; downstream users may not be using jsonpb
// unmarshallers to interpret the JSON data.
// This may need to be revised in a subsequent version of this API
// specification.
// LUCI Executables MAY invoke other LUCI Executables as sub-steps and have the
// Steps from the child luciexe show in the parent's Build updates. This is one
// of the responsibilities of the Host Application.
// The parent can achieve this by recording a Step S (with no children), and
// a Step.MergeBuild.from_logdog_stream which points to a "build.proto" stream
// (see "Updating the Build State"). This is called a "Merge Step", and is
// a directive for the host to merge Build messages from in the
// from_logdog_stream log here. Note that this stream name should follow the
// same provisions specified for "Step.Log.Url". It's the invoker's
// responsibility to populate $LOGDOG_NAMESPACE with the new, full, namespace.
// Failure to do this will result in missing logs/step data.
// (SIDENOTE: There's an internal proposal to improve the Logdog "butler" API so
// that application code only needs to handle the relative namespaces as well,
// which would make this much less confusing).
// The Host Application MUST append all steps from the child build.proto
// stream to the parent build as substeps of step S and copy the following
// fields of the child Build to the equivalent fields of step S *only if*
// step S has *non-final* status. It is the caller's responsibility to populate
// rest of the fields of step S if the caller explicitly marks the step
// status as final.
// SummaryMarkdown
// Status
// EndTime
// Output.Logs (appended)
// This rule applies recursively, i.e. the child build MAY have Merge Step(s).
// Each luciexe's step names should be emitted as relative names. e.g. say
// a build runs a sub-luciexe with the name "a|b". This sub-luciexe then runs
// a step "x|y|z". The top level build.proto stream will show the step
// "a|b|x|y|z".
// The graph of datagram streams MUST be a tree. This follows from the
// namespacing rules of the Log.Url fields; Since Log.Url fields are relative to
// their build's namespace, it's only possible to have a merge step point
// further 'down' the tree, making it impossible to create a cycle.
// Recursive Invocation - Legacy ChromeOS style
// Note that there's an option in MergeBuild called 'legacy_global_namespace'.
// This mode is NOT RECOMMENDED, and was only added to support legacy ChromeOS
// builders which relied on this functionality prior to luciexe.
// When this option is turned on, output properties on the build stream will be
// merged into the top-level build containing the Merge Step, and it will also
// de-namespace the step names (i.e. a MergeStep called "Parent" whose
// build.proto stream contains a name "Step" would normally show on the build as
// "Parent|Step", will instead simply show as "Step" in this mode. This also
// means that these merge step processes must make sure not to emit a step which
// duplicates the name of one emitted in the parent build (otherwise the overall
// build will be invalid)).
// Using this functionality has the downside that properties emitted by the
// parent build WILL BE OVERWRITTEN by the merged build stream, if written to
// the same top-level property keys. In ChromeOS's case this is "OK" because the
// legacy recipes do a short bootstrap followed by the main payload which does
// all the actual stuff.
// There's a version of this feature which would be more-supportable, which
// would be to introduce namespaced outputs so that the top-level build could
// reflect the outputs from a given step (i.e. allow Step to actually have its
// own Output message). If you feel like you need this functionality, please
// contact ChOps Foundation team to discuss, rather than trying to use the
// legacy_global_namespace option.
// Related Libraries
// For implementation-level details, please refer to the following:
// * "" - low-level protocol details (this module).
// * "" - the Host Application library.
// * "" - luciexe invocation.
// * "" - luciexe binary helper library.
// Other Client Implementations
// Python Recipes (
// implement the LUCI Executable protocol using the "luciexe" subcommand.
// TODO(iannucci): Implement a luciexe binary helper in `infra_libs` analogous
// to and implement Recipes' support in terms
// of this.
// LUCI Executables on Buildbucket
// Buildbucket accepts LUCI Executables as CIPD packages containing the
// luciexe to run with the fixed name of "luciexe".
// On Windows, this may be named "luciexe.exe" or "luciexe.bat". Buildbucket
// will prefer the first of these which it finds.
// Note that the ` bundle` command generates a "luciexe" wrapper
// script for compatibility with Buildbucket.
package luciexe