lucicfg
is a tool for generating low-level LUCI configuration files based on a high-level configuration given as a Starlark script that uses APIs exposed by lucicfg
. In other words, it takes a *.star file (or files) as input and spits out a bunch of *.cfg files (such us cr-buildbucket.cfg
and luci-scheduler.cfg
) as outputs. A single entity (such as a luci.builder(...) definition) in the input is translated into multiple entities (such as Buildbucket‘s builder{...} and Scheduler’s job{...}) in the output. This ensures internal consistency of all low-level configs.
Using Starlark allows to further reduce duplication and enforce invariants in the configs. A common pattern is to use Starlark functions that wrap one or more basic rules (e.g. luci.builder(...) and luci.console_view_entry(...)) to define more “concrete” entities (for example “a CI builder” or “a Try builder”). The rest of the config script then uses such functions to build up the actual configuration.
lucicfg
is distributed as a single self-contained binary as part of depot_tools, so if you use them, you already have it. Additionally it is available in PATH on all LUCI builders. The rest of this doc also assumes that lucicfg
is in PATH.
If you don't use depot_tools, lucicfg
can be installed through CIPD. The package is infra/tools/luci/lucicfg/${platform}, and the canonical stable version can be looked up in the depot_tools CIPD manifest.
Finally, you can always try to build lucicfg
from the source code. However, the only officially supported distribution mechanism is CIPD packages.
lucicfg
can be found here.Create main.star
file with the following content:
#!/usr/bin/env lucicfg luci.project( name = "hello-world", buildbucket = "cr-buildbucket.appspot.com", swarming = "chromium-swarm.appspot.com", ) luci.bucket(name = "my-bucket") luci.builder( name = "my-builder", bucket = "my-bucket", executable = luci.recipe( name = "my-recipe", cipd_package = "recipe/bundle/package", ), )
Now run lucicfg generate main.star
. It will create a new directory generated
side-by-side with main.star
file. This directory contains project.cfg
and cr-buildbucket.cfg
files, generated based on the script above.
Equivalently, make the script executable (chmod a+x main.star
) and then just execute it (./main.star
). This is the exact same thing as running generate
subcommand.
Now make some change in main.star
(for example, rename the builder), but do not regenerate the configs yet. Instead run lucicfg validate main.star
. It will produce an error, telling you that files on disk (in generated/*
) are stale. Regenerate them (./main.star
), and run the validation again.
If you have never done this before or haven't used any other LUCI tools, you are now asked to authenticate by running lucicfg auth-login
. This is because lucicfg validate
in addition to checking configs locally also sends them for a more thorough validation to the LUCI Config service, and this requires you to be authenticated. Do lucicfg auth-login
and re-run lucicfg validate main.star
. It should succeed now. If it still fails with permissions issues, you are probably not in config-validation
group (this should be rare, please contact luci-eng@google.com if this is happening).
lucicfg validate
is meant to be used from presubmit tests. If you use depot_tools' PRESUBMIT.py
, there's a canned check that wraps lucicfg validate
.
This is it, your first generated config! It is not very functional yet (e.g. builders without Swarming dimensions are useless), but a good place to start. Keep iterating on it, modifying the script, regenerating configs, and examining the output in generated
directory. Once you are satisfied with the result, commit both Starlark scripts and generated configs into the repository, and then configure LUCI Config service to pull configuration from generated
directory (how to do it is outside the scope of this doc).
This process is mostly manual, but it is aided by lucicfg semantic-diff
command that can be used to verify the generated configs match the original ones. Roughly, the idea is to start with broad strokes, and then refine details until old and new configs match:
main.star
in the same directory that contains existing configs (like cr-buildbucket.cfg
). Add luci.project(...) and all luci.bucket(...) definitions there. Generated configs will be stored in generated
subdirectory, which is not yet really used for anything.cr-buildbucket.cfg
contains all builders (but their details are not necessarily are correct yet).lucicfg semantic-diff main.star cr-buildbucket.cfg
. It will normalize the original and the generated Buildbucket configs (by expanding all mixins, sorting fields, etc) and run git diff ...
to compare them. Our goal is to reduce this diff to zero.cr-buildbucket.cfg
is zero.luci-scheduler.cfg
, luci-milo.cfg
, commit-queue.cfg
, etc.generated
directory are semantically identical to the existing configs. Switch LUCI Config to use generated
as source of configs, deleted old configs.lucicfg
, not a generic Starlark interpreter. Also this is advanced stuff. Its full understanding is not required to use lucicfg
effectively.Each individual Starlark file is called a module. Several modules under the same root directory form a package. Modules within a single package can refer to each other (in load(...) and exec(...)) using their relative or absolute (if start with //
) paths. The root of the main package is taken to be a directory that contains the entry point script (usually main.star
) passed to lucicfg
, i.e. main.star
itself can be referred to as //main.star
.
Modules can either be “library-like” (executed via load(...) statement) or “script-like” (executed via exec(...) function). Library-like modules can load other library-like modules via load(...), but may not call exec(...). Script-like modules may use both load(...) and exec(...).
Dicts of modules loaded via load(...) are reused, e.g. if two different scripts load the exact same module, they'll get the exact same symbols as a result. The loaded code always executes only once. The interpreter may load modules in parallel in the future, libraries must not rely on their loading order and must not have side effects.
On the other hand, modules executed via exec(...) are guaranteed to be processed sequentially, and only once. Thus ‘exec’-ed scripts essentially form a tree, traversed exactly once in the depth first order.
All entities manipulated by lucicfg
are represented by nodes in a directed acyclic graph. One entity (such as a builder) can internally be represented by multiple nodes. A function that adds nodes and edges to the graph is called a rule (e.g. luci.builder(...) is a rule).
Each node has a unique hierarchical key, usually constructed from entity's properties. For example, a builder name and its bucket name are used to construct a unique key for this builder (roughly <bucket>/<builder>
). These keys are used internally by rules when adding edges to the graph.
To refer to entities from public API, one just usually uses strings (e.g. a builder name to refer to the builder). Rules' implementation usually have enough context to construct correct node keys from such strings. Sometimes they need some help, see Resolving naming ambiguities. Other times entities have no meaningful global names at all (for example, luci.console_view_entry(...)). For such cases, one uses a return value of the corresponding rule: rules return opaque pointer-like objects that can be passed to other rules as an input in place of a string identifiers. This allows to “chain” definitions, e.g.
luci.console_view( ... entries = [ luci.console_view_entry(...), luci.console_view_entry(...), ... ], )
It is strongly preferred to either use string names to refer to entities or define them inline where they are needed. Please avoid storing return values of rules in variables to refer to them later. Using string names is as powerful (lucicfg
verifies referential integrity), and it offers additional advantages (like referring to entities across file boundaries).
To aid in using inline definitions where makes sense, many rules allow entities to be defines multiple times as long as all definitions are identical (this is internally referred to as “idempotent nodes”). It allows following usage style:
def my_recipe(name): return luci.recipe( name = name, cipd_package = 'my/recipe/bundle', ) luci.builder( name = 'builder 1', executable = my_recipe('some-recipe'), ... ) luci.builder( name = 'builder 2', executable = my_recipe('some-recipe'), ... )
Here some-recipe
is formally defined twice, but both definitions are identical, so it doesn't cause ambiguities. See the documentation of individual rules to see whether they allow such redefinitions.
There are 3 stages of lucicfg gen
execution:
main.star
code and all modules it exec's. This builds a graph in memory (via calls to rules), and registers a bunch of generator callbacks (via lucicfg.generator(...)) that will traverse this graph in the stage 3.main.star
code. At this point we have a (potentially incomplete) graph and a list of registered generator callbacks.luci.builder("name") refers to undefined luci.bucket("bucket") at <stack trace of the corresponding luci.builder(...) definition>
.lucicfg
core code, not touching Starlark at all. It doesn't need to understand the semantics of graph nodes, and thus used for all sorts of configs (LUCI configs are just one specific application).Presently all this machinery is mostly hidden from the end user. It will become available in future versions of lucicfg
as an API for extending lucicfg
, e.g. for adding new entity types that have relation to LUCI, or for repurposing lucicfg
for generating non-LUCI conifgs.
Builder names are scoped to buckets. For example, it is possible to have the following definition:
# Runs pre-submit tests on Linux. luci.builder( name = 'Linux', bucket = 'try', ... ) # Runs post-submit tests on Linux. luci.builder( name = 'Linux', bucket = 'ci', ... )
Here Linux
name by itself is ambiguous and can't be used to refer to the builder. E.g. the following chunk of code will cause an error:
luci.list_view_entry( builder = 'Linux', # but which one?... ... )
The fix is to prepend the bucket name:
luci.list_view_entry( builder = 'ci/Linux', # ah, the CI one ... )
It is always correct to use “full” name like this. But in practice the vast majority of real world configs do not have such ambiguities and requiring full names everywhere is a chore. For that reason lucicfg
allows to omit the bucket name if the resulting reference is non-ambiguous. In the example above, if we remove one of the builders, builder = 'Linux'
reference becomes valid.
Some LUCI Services allow one project to refer to resources in another project. For example, a luci.console_view(...) can display builders that belong to another LUCI project, side-by-side with the builders from the project the console belongs to.
Such external builders can be referred to via their fully qualified name in the format <project>:<bucket>/<name>
. Note that <bucket>
part can't be omitted.
For example:
luci.console_view_entry( builder = 'chromium:ci/Linux Builder', ... )
luci.builder(...) and luci.gitiles_poller(...) rules have schedule
field that defines how often the builder or poller should run. Schedules are given as strings. Supported kinds of schedules (illustrated via examples):
* 0 * * * *
: a crontab expression, in a syntax supported by https://github.com/gorhill/cronexpr (see its docs for full reference). LUCI will attempt to start the job at specified moments in time (based on UTC clock). Some examples:
0 */3 * * * *
- every 3 hours: at 12:00 AM UTC, 3:00 AM UTC, ...0 */3 * * *
- the exact same thing (the last field is optional).0 1/3 * * *
- every 3 hours but starting 1:00 AM UTC.0 2,10,18 * * *
- at 2 AM UTC, 10 AM UTC, 6 PM UTC.0 7 * * *
- at 7 AM UTC, once a day.If a previous invocation is still running when triggering a new one, an overrun is recorded and the new scheduled invocation is skipped. The next attempt to start the job happens based on the schedule (not when the currently running invocation finishes).
with 10s interval
: run the job in a loop, waiting 10s after finishing an invocation before starting a new one. Moments when the job starts aren't synchronized with the wall clock at all.
with 1m interval
, with 1h interval
: same format, just using minutes and hours instead of seconds.
continuously
is alias for with 0s interval
, meaning to run the job in a loop without any pauses at all.
triggered
schedule indicates that the job is only started via some external triggering event (e.g. via LUCI Scheduler API), not periodically.
lucicfg.version()
Returns a triple with lucicfg version: (major, minor, revision)
.
lucicfg.check_version(min, message = None)
Fails if lucicfg version is below the requested minimal one.
Useful when a script depends on some lucicfg feature that may not be available in earlier versions. lucicfg.check_version(...) can be used at the start of the script to fail right away with a clean error message:
lucicfg.check_version( min = '1.5.5', message = 'Update depot_tools', )
Or even
lucicfg.check_version('1.5.5')
major.minor.revision
with minimally accepted version. Required.lucicfg.config( # Optional arguments. config_service_host = None, config_dir = None, tracked_files = None, fail_on_warnings = None, )
Sets one or more parameters for the lucicfg
itself.
These parameters do not affect semantic meaning of generated configs, but influence how they are generated and validated.
Each parameter has a corresponding command line flag. If the flag is present, it overrides the value set via lucicfg.config
(if any). For example, the flag -config-service-host <value>
overrides whatever was set via lucicfg.config(config_service_host=...)
.
lucicfg.config
is allowed to be called multiple times. The most recently set value is used in the end, so think of lucicfg.config(var=...)
just as assigning to a variable.
lucicfg
binary, usually luci-config.appspot.com
...
is allowed. If set via -config-dir
command line flag, it is relative to the current working directory. Will be created if absent. If -
, the configs are just printed to stdout in a format useful for debugging. Default is “generated”.config_dir
that are considered generated. Each entry is either <glob pattern>
(a “positive” glob) or !<glob pattern>
(a “negative” glob). A file under config_dir
is considered tracked if its slash-separated path matches any of the positive globs and none of the negative globs. If a pattern starts with **/
, the rest of it is applied to the base name of the file (not the whole path). If only negative globs are given, single positive **/*
glob is implied as well. tracked_files
can be used to limit what files are actually emitted: if this set is not empty, only files that are in this set will be actually written to the disk (and all other files are discarded). This is beneficial when lucicfg
is used to generate only a subset of config files, e.g. during the migration from handcrafted to generated configs. Knowing the tracked files set is also important when some generated file disappears from lucicfg
output: it must be deleted from the disk as well. To do this, lucicfg
needs to know what files are safe to delete. If tracked_files
is empty (default), lucicfg
will save all generated files and will never delete any file (in this case it is responsibility of the caller to make sure no stale output remains).lucicfg.config
and you want to override it to False via command line flags use -fail-on-warnings=false
.lucicfg.enable_experiment(experiment)
Enables an experimental feature.
Can be used to experiment with not yet released features that may later change in a non-backwards compatible way or even be removed completely. Primarily intended for lucicfg developers to test their features before they are “frozen” to be backward compatible. If you rely on an experimental feature and a lucicfg update breaks your config, this is a problem in your config, not in lucicfg.
Enabling an experiment that doesn‘t exist logs a warning, but doesn’t fail the execution. Refer to the documentation and the source code for the list of available experiments.
lucicfg.generator(impl = None)
Registers a callback that is called at the end of the config generation stage to modify/append/delete generated configs in an arbitrary way.
The callback accepts single argument ctx
which is a struct with the following fields and methods:
output: a dict {config file name -> (str | proto)}
. The callback is free to modify ctx.output
in whatever way it wants, e.g. by adding new values there or mutating/deleting existing ones.
declare_config_set(name, root): proclaims that generated configs under the given root (relative to config_dir
) belong to the given config set. Safe to call multiple times with exact same arguments, but changing an existing root to something else is an error.
func(ctx) -> None
.lucicfg.emit(dest, data)
Tells lucicfg to write given data to some output file.
In particular useful in conjunction with io.read_file(...) to copy files into the generated output:
lucicfg.emit( dest = 'tricium.cfg', data = io.read_file('//tricium.cfg'), )
Note that lucicfg.emit(...) cannot be used to override generated files. dest
must refer to a path not generated or emitted by anything else.
config_dir
(see lucicfg.config(...)). Must not start with ../
. Required.dest
. Proto messages are serialized using text protobuf encoding. Required.lucicfg.current_module()
Returns the location of a module being currently executed.
This is the module being processed by a current load(...) or exec(...) statement. It has no relation to the module that holds the top-level stack frame. For example, if a currently loading module A
calls a function in a module B
and this function calls lucicfg.current_module(...), the result would be the module A
, even though the call goes through code in the module B
(i.e. lucicfg.current_module(...) invocation itself resided in a function in module B
).
Fails if called from inside a generator callback. Threads executing such callbacks are not running any load(...) or exec(...).
A struct(package='...', path='...')
with the location of the module.
lucicfg.var(default = None, validator = None, expose_as = None)
Declares a variable.
A variable is a slot that can hold some frozen value. Initially this slot is usually empty. lucicfg.var(...) returns a struct with methods to manipulate it:
set(value)
: sets the variable‘s value if it’s unset, fails otherwise.get()
: returns the current value, auto-setting it to default
if it was unset.Note the auto-setting the value in get()
means once get()
is called on an unset variable, this variable can't be changed anymore, since it becomes initialized and initialized variables are immutable. In effect, all callers of get()
within a scope always observe the exact same value (either an explicitly set one, or a default one).
Any module (loaded or exec'ed) can declare variables via lucicfg.var(...). But only modules running through exec(...) can read and write them. Modules being loaded via load(...) must not depend on the state of the world while they are loading, since they may be loaded at unpredictable moments. Thus an attempt to use get
or set
from a loading module causes an error.
Note that functions exported by loaded modules still can do anything they want with variables, as long as they are called from an exec-ing module. Only code that executes while the module is loading is forbidden to rely on state of variables.
Assignments performed by an exec-ing module are visible only while this module and all modules it execs are running. As soon as it finishes, all changes made to variable values are “forgotten”. Thus variables can be used to implicitly propagate information down the exec call stack, but not up (use exec's return value for that).
Generator callbacks registered via lucicfg.generator(...) are forbidden to read or write variables, since they execute outside of context of any exec(...). Generators must operate exclusively over state stored in the node graph. Note that variables still can be used by functions that build the graph, they can transfer information from variables into the graph, if necessary.
The most common application for lucicfg.var(...) is to “configure” library modules with default values pertaining to some concrete executing script:
This is more magical but less wordy alternative to either passing specific default values in every call to library functions, or wrapping all library functions with wrappers that supply such defaults. These more explicit approaches can become pretty convoluted when there are multiple scripts and libraries involved.
Another use case is to allow parameterizing configs with values passed via CLI flags. A string-typed var can be declared with expose_as=<name>
argument, making it settable via -var <name>=<value>
CLI flag. This is primarily useful in conjunction with -emit-to-stdout
CLI flag to use lucicfg as a “function call” that accepts arguments via CLI flags and returns the result via stdout to pipe somewhere else, e.g.
lucicfg generate main.star -var environ=dev -emit-to-stdout all.json | ...
Danger: Using -var
without -emit-to-stdout
is generally wrong, since configs generated on disk (and presumably committed into a repository) must not depend on undetermined values passed via CLI flags.
get()
if it was unset.validator(value)
from set(value)
and inside lucicfg.var(...) declaration itself (to validate default
or a value passed via CLI flags). Must be a side-effect free idempotent function that returns the value to be assigned to the variable (usually just value
itself, but conversions are allowed, including type changes).-var <expose_as>=<value>
. If there's no such flag, the variable is auto-initialized to its default value (which must be string or None). Variables declared with expose_as
are not settable via set()
at all, they appear as “set” already the moment they are declared. If multiple vars use the same expose_as
identifier, they will all be initialized to the same value.A struct with two methods: set(value)
and get(): value
.
lucicfg.rule(impl, defaults = None)
Declares a new rule.
A rule is a callable that adds nodes and edges to an entity graph. It wraps the given impl
callback by passing one additional argument ctx
to it (as the first positional argument).
ctx
is a struct with the following fields:
The callback is expected to return a graph.keyset(...) with the set of graph keys that represent the added node (or nodes). Other rules use such keysets as inputs.
ctx
. The rest of the arguments define the API of the rule. Required.rule.defaults.<name>.set(...)
. impl
callback can get them via ctx.defaults.<name>.get()
. It is up to the rule's author to define vars for fields that can have defaults, document them in the rule doc, and finally use them from impl
callback.A special callable.
Time module provides a simple API for defining durations in a readable way, resembling golang's time.Duration.
Durations are represented by integer-like values of time.duration(...) type, which internally hold a number of milliseconds.
Durations can be added and subtracted from each other and multiplied by integers to get durations. They are also comparable to each other (but not to integers). Durations can also be divided by each other to get an integer, e.g. time.hour / time.second
produces 3600.
The best way to define a duration is to multiply an integer by a corresponding “unit” constant, for example 10 * time.second
.
Following time constants are exposed:
Constant | Value (obviously) |
---|---|
time.zero | 0 milliseconds |
time.millisecond | 1 millisecond |
time.second | 1000 * time.millisecond |
time.minute | 60 * time.second |
time.hour | 60 * time.minute |
time.day | 24 * time.hour |
time.week | 7 * time.day |
time.duration(milliseconds)
Returns a duration that represents the integer number of milliseconds.
time.duration value.
time.epoch(layout, value, location)
Returns epoch seconds for value interpreted as a time per layout in location.
int epoch seconds for value.
time.truncate(duration, precision)
Truncates the precision of the duration to the given value.
For example time.truncate(time.hour+10*time.minute, time.hour)
is time.hour
.
Truncated time.duration value.
time.days_of_week(spec)
Parses e.g. Tue,Fri-Sun
into a list of day indexes, e.g. [2, 5, 6, 7]
.
Monday is 1, Sunday is 7. The returned list is sorted and has no duplicates. An empty string results in the empty list.
Tue
), or a range (e.g. Wed-Sun
). Required.A list of 1-based day indexes. Monday is 1.
luci.project( # Required arguments. name, # Optional arguments. config_dir = None, buildbucket = None, logdog = None, milo = None, notify = None, scheduler = None, swarming = None, acls = None, )
Defines a LUCI project.
There should be exactly one such definition in the top-level config file.
config_dir
in lucicfg.config(...)) to place generated LUCI configs under. Default is .
. A custom value is useful when using lucicfg
to generate LUCI and non-LUCI configs at the same time.luci.logdog(gs_bucket = None)
Defines configuration of the LogDog service for this project.
Usually required for any non-trivial project.
luci.bucket(name, acls = None)
Defines a bucket: a container for LUCI resources that share the same ACL.
ci
or try
. Required.luci.executable(name, cipd_package = None, cipd_version = None)
Defines an executable.
Builders refer to such executables in their executable
field, see luci.builder(...). Multiple builders can execute the same executable (perhaps passing different properties to it).
Executables must be available as cipd packages.
The cipd version to fetch is usually a lower-cased git ref (like refs/heads/master
), or it can be a cipd tag (like git_revision:abc...
).
A luci.executable(...) with some particular name can be redeclared many times as long as all fields in all declaration are identical. This is helpful when luci.executable(...) is used inside a helper function that at once declares a builder and an executable needed for this builder.
refs/heads/master
. Supports the module-scoped default.luci.recipe( # Required arguments. name, # Optional arguments. cipd_package = None, cipd_version = None, recipe = None, )
Defines an executable that runs a particular recipe.
Recipes are python-based DSL for defining what a builder should do, see recipes-py.
Builders refer to such executable recipes in their executable
field, see luci.builder(...). Multiple builders can execute the same recipe (perhaps passing different properties to it).
Recipes are located inside cipd packages called “recipe bundles”. Typically the cipd package name with the recipe bundle will look like:
infra/recipe_bundles/chromium.googlesource.com/chromium/tools/build
Recipes bundled from internal repositories are typically under
infra_internal/recipe_bundles/...
But if you're building your own recipe bundles, they could be located elsewhere.
The cipd version to fetch is usually a lower-cased git ref (like refs/heads/master
), or it can be a cipd tag (like git_revision:abc...
).
A luci.recipe(...) with some particular name can be redeclared many times as long as all fields in all declaration are identical. This is helpful when luci.recipe(...) is used inside a helper function that at once declares a builder and a recipe needed for this builder.
recipe
is None, also specifies the recipe name within the bundle. Required.refs/heads/master
. Supports the module-scoped default.name
. Useful if recipe names clash between different recipe bundles. When this happens, name
can be used as a non-ambiguous alias, and recipe
can provide the actual recipe name. Defaults to name
.luci.builder( # Required arguments. name, bucket, executable, # Optional arguments. properties = None, service_account = None, caches = None, execution_timeout = None, dimensions = None, priority = None, swarming_host = None, swarming_tags = None, expiration_timeout = None, schedule = None, triggering_policy = None, build_numbers = None, experimental = None, task_template_canary_percentage = None, repo = None, triggers = None, triggered_by = None, notifies = None, )
Defines a generic builder.
It runs some executable (usually a recipe) in some requested environment, passing it a struct with given properties. It is launched whenever something triggers it (a poller or some other builder, or maybe some external actor via Buildbucket or LUCI Scheduler APIs).
The full unique builder name (as expected by Buildbucket RPC interface) is a pair (<project>, <bucket>/<name>)
, but within a single project config this builder can be referred to either via its bucket-scoped name (i.e. <bucket>/<name>
) or just via it‘s name alone (i.e. <name>
), if this doesn’t introduce ambiguities.
The definition of what can potentially trigger what is defined through triggers
and triggered_by
fields. They specify how to prepare ACLs and other configuration of services that execute builds. If builder A is defined as “triggers builder B”, it means all services should expect A builds to trigger B builds via LUCI Scheduler‘s EmitTriggers RPC or via Buildbucket’s ScheduleBuild RPC, but the actual triggering is still the responsibility of A's executable.
There‘s a caveat though: only Scheduler ACLs are auto-generated by the config generator when one builder triggers another, because each Scheduler job has its own ACL and we can precisely configure who’s allowed to trigger this job. Buildbucket ACLs are left unchanged, since they apply to an entire bucket, and making a large scale change like that (without really knowing whether Buildbucket API will be used) is dangerous. If the executable triggers other builds directly through Buildbucket, it is the responsibility of the config author (you) to correctly specify Buildbucket ACLs, for example by adding the corresponding service account to the bucket ACLs:
luci.bucket( ... acls = [ ... acl.entry(acl.BUILDBUCKET_TRIGGERER, <builder service account>), ... ], )
This is not necessary if the executable uses Scheduler API instead of Buildbucket.
os
), and values are either strings (e.g. Linux
), swarming.dimension(...) objects (for defining expiring dimensions) or lists of thereof. Supports the module-scoped defaults. They are merged (non-recursively) with the explicitly passed dimensions.k:v
strings) to assign to the Swarming task that runs the builder. Each tag will also end up in swarming_tag
Buildbucket tag, for example swarming_tag:builder:release
. Supports the module-scoped defaults. They are joined with the explicitly passed tags.dimensions
) before canceling the build and marking it as expired. If None, defer the decision to Buildbucket service. Supports the module-scoped default.https://
) associated with the builder, if known. It is in particular important when using luci.notifier(...) to let LUCI know what git history it should use to chronologically order builds on this builder. If unknown, builds will be ordered by creation time. If unset, will be taken from the configuration of luci.gitiles_poller(...) that trigger this builder if they all poll the same repo.notified_by
field in luci.notifier(...) or luci.tree_closer(...).luci.gitiles_poller( # Required arguments. name, bucket, repo, # Optional arguments. refs = None, path_regexps = None, path_regexps_exclude = None, schedule = None, triggers = None, )
Defines a gitiles poller which can trigger builders on git commits.
It periodically examines the state of watched refs in the git repository. On each iteration it triggers builders if either:
A watched ref‘s tip has changed since the last iteration (e.g. a new commit landed on a ref). Each new detected commit results in a separate triggering request, so if for example 10 new commits landed on a ref since the last poll, 10 new triggering requests will be submitted to the builders triggered by this poller. How they are converted to actual builds depends on triggering_policy
of a builder. For example, some builders may want to have one build per commit, others don’t care and just want to test the latest commit. See luci.builder(...) and scheduler.policy(...) for more details.
A ref belonging to the watched set has just been created. This produces a single triggering request.
Commits that trigger builders can also optionally be filtered by file paths they touch. These conditions are specified via path_regexps
and path_regexps_exclude
fields, each is a list of regular expressions against Unix file paths relative to the repository root. A file is considered “touched” if it is either added, modified, removed, moved (both old and new paths are considered “touched”), or its metadata has changed (e.g. chmod +x
).
A triggering request is emitted for a commit if only if at least one touched file is not matched by any path_regexps_exclude
and simultaneously matched by some path_regexps
, subject to following caveats:
path_regexps = [".+"]
will not match commits which modify no files (aka empty commits) and as such this situation differs from the default case of not specifying any path_regexps
.A luci.gitiles_poller(...) with some particular name can be redeclared many times as long as all fields in all declaration are identical. This is helpful when luci.gitiles_poller(...) is used inside a helper function that at once declares a builder and a poller that triggers this builder.
https://
. Required.refs/heads/[^/]+
or refs/branch-heads/\d+\.\d+
. The regular expression should have a literal prefix with at least two slashes present, e.g. refs/release-\d+/foobar
is not allowed, because the literal prefix refs/release-
contains only one slash. The regexp should not start with ^
or end with $
as they will be added automatically. Each supplied regexp must match at least one ref in the gitiles output, e.g. specifying refs/tags/v.+
for a repo that doesn't have tags starting with v
causes a runtime error. If empty, defaults to ['refs/heads/master']
.^
and $
are implied and should not be specified manually. See the explanation above for all details.^
and $
are implied and should not be specified manually. See the explanation above for all details.luci.milo( # Optional arguments. logo = None, favicon = None, monorail_project = None, monorail_components = None, bug_summary = None, bug_description = None, )
Defines optional configuration of the Milo service for this project.
Milo service is a public user interface for displaying (among other things) builds, builders, builder lists (see luci.list_view(...)) and consoles (see luci.console_view(...)).
Can optionally be configured with a reference to a Monorail project to use for filing bugs via custom bug links on build pages. The format of a new bug is defined via bug_summary
and bug_description
fields which are interpreted as Golang text templates. They can either be given directly as strings, or loaded from external files via io.read_file(...).
Supported interpolations are the fields of the standard build proto such as:
{{.Build.Builder.Project}} {{.Build.Builder.Bucket}} {{.Build.Builder.Builder}}
Other available fields include:
{{.MiloBuildUrl}} {{.MiloBuilderUrl}}
If any specified placeholder cannot be satisfied then the bug link is not displayed.
storage.googleapis.com
.storage.googleapis.com
.>
-separated format, e.g. Infra>Client>ChromeOS>CI
. Required if monorail_project
is set, otherwise must not be used.monorail_project
is unset.monorail_project
is unset.luci.list_view( # Required arguments. name, # Optional arguments. title = None, favicon = None, entries = None, )
A Milo UI view that displays a list of builders.
Builders that belong to this view can be specified either right here:
luci.list_view( name = 'Try builders', entries = [ 'win', 'linux', luci.list_view_entry('osx'), ], )
Or separately one by one via luci.list_view_entry(...) declarations:
luci.list_view(name = 'Try builders') luci.list_view_entry( builder = 'win', list_view = 'Try builders', ) luci.list_view_entry( builder = 'linux', list_view = 'Try builders', )
Note that list views support builders defined in other projects. See Referring to builders in other projects for more details.
name
.storage.googleapis.com
. Defaults to favicon
in luci.milo(...).luci.list_view_entry(builder = None, list_view = None)
A builder entry in some luci.list_view(...).
Can be used to declare that a builder belongs to a list view outside of the list view declaration. In particular useful in functions. For example:
luci.list_view(name = 'Try builders') def try_builder(name, ...): luci.builder(name = name, ...) luci.list_view_entry(list_view = 'Try builders', builder = name)
Can also be used inline in luci.list_view(...) declarations, for consistency with corresponding luci.console_view_entry(...) usage. list_view
argument can be omitted in this case:
luci.list_view( name = 'Try builders', entries = [ luci.list_view_entry(builder = 'Win'), ... ], )
list_view_entry
is used inline inside some luci.list_view(...) declaration.luci.console_view( # Required arguments. name, repo, # Optional arguments. title = None, refs = None, exclude_ref = None, header = None, include_experimental_builds = None, favicon = None, default_commit_limit = None, default_expand = None, entries = None, )
A Milo UI view that displays a table-like console where columns are builders and rows are git commits on which builders are triggered.
A console is associated with a single git repository it uses as a source of commits to display as rows. The watched ref set is defined via refs
and optional exclude_ref
fields. If refs
are empty, the console defaults to watching refs/heads/master
.
exclude_ref
is useful when watching for commits that landed specifically on a branch. For example, the config below allows to track commits from all release branches, but ignore the commits from the master branch, from which these release branches are branched off:
luci.console_view( ... refs = ['refs/branch-heads/\d+\.\d+'], exclude_ref = 'refs/heads/master', ... )
For best results, ensure commits on each watched ref have committer timestamps monotonically non-decreasing. Gerrit will take care of this if you require each commit to go through Gerrit by prohibiting “git push” on these refs.
Builders that belong to the console can be specified either right here:
luci.console_view( name = 'CI builders', ... entries = [ luci.console_view_entry( builder = 'Windows Builder', short_name = 'win', category = 'ci', ), # Can also pass a dict, this is equivalent to passing # luci.console_view_entry(**dict). { 'builder': 'Linux Builder', 'short_name': 'lnx', 'category': 'ci', }, ... ], )
Or separately one by one via luci.console_view_entry(...) declarations:
luci.console_view(name = 'CI builders') luci.console_view_entry( builder = 'Windows Builder', console_view = 'CI builders', short_name = 'win', category = 'ci', )
Note that consoles support builders defined in other projects. See Referring to builders in other projects for more details.
Consoles can have headers which are collections of links, oncall rotation information, and console summaries that are displayed at the top of a console, below the tree status information. Links and oncall information is always laid out to the left, while console groups are laid out to the right. Each oncall and links group take up a row.
Header definitions are based on Header
message in Milo's project.proto. There are two way to supply this message via header
field:
Pass an appropriately structured dict. Useful for defining small headers inline:
luci.console_view( ... header = { 'links': [ {'name': '...', 'links': [...]}, ... ], }, ... )
Pass a string. It is treated as a path to a file with serialized Header
message. Depending on its extension, it is loaded ether as JSONPB-encoded message (*.json
and *.jsonpb
paths), or as TextPB-encoded message (everything else):
luci.console_view( ... header = '//consoles/main_header.textpb', ... )
name
.https://
. Required.refs/heads/[^/]+
or refs/branch-heads/\d+\.\d+
. The regular expression should have a literal prefix with at least two slashes present, e.g. refs/release-\d+/foobar
is not allowed, because the literal prefix refs/release-
contains only one slash. The regexp should not start with ^
or end with $
as they will be added automatically. If empty, defaults to ['refs/heads/master']
.refs
and refs_regexps
. Note that force pushes to this ref are not supported. Milo uses caching assuming set of commits reachable from this ref may only grow, never lose some commits.storage.googleapis.com
. Defaults to favicon
in luci.milo(...).luci.console_view_entry( # Optional arguments. builder = None, short_name = None, category = None, console_view = None, )
A builder entry in some luci.console_view(...).
Used inline in luci.console_view(...) declarations to provide category
and short_name
for a builder. console_view
argument can be omitted in this case:
luci.console_view( name = 'CI builders', ... entries = [ luci.console_view_entry( builder = 'Windows Builder', short_name = 'win', category = 'ci', ), ... ], )
Can also be used to declare that a builder belongs to a console outside of the console declaration. In particular useful in functions. For example:
luci.console_view(name = 'CI builders') def ci_builder(name, ...): luci.builder(name = name, ...) luci.console_view_entry(console_view = 'CI builders', builder = name)
term1|term2|...
that describes the hierarchy of the builder columns. Neighboring builders with common ancestors will have their column headers merged. In expanded view, each leaf category or builder under a non-leaf category will have it's own column. The recommendation for maximum densification is not to mix subcategories and builders for children of each category.console_view_entry
is used inline inside some luci.console_view(...) declaration.luci.notifier( # Required arguments. name, # Optional arguments. on_occurrence = None, on_new_status = None, on_failure = None, on_new_failure = None, on_status_change = None, on_success = None, failed_step_regexp = None, failed_step_regexp_exclude = None, notify_emails = None, notify_rotang_rotations = None, notify_blamelist = None, blamelist_repos_whitelist = None, template = None, notified_by = None, )
Defines a notifier that sends notifications on events from builders.
A notifier contains a set of conditions specifying what events are considered interesting (e.g. a previously green builder has failed), and a set of recipients to notify when an interesting event happens. The conditions are specified via on_*
fields, and recipients are specified via notify_*
fields.
The set of builders that are being observed is defined through notified_by
field here or notifies
field in luci.builder(...). Whenever a build finishes, the builder “notifies” all luci.notifier(...) objects subscribed to it, and in turn each notifier filters and forwards this event to corresponding recipients.
Note that luci.notifier(...) and luci.tree_closer(...) are both flavors of a luci.notifiable
object, i.e. both are something that “can be notified” when a build finishes. They both are valid targets for notifies
field in luci.builder(...). For that reason they share the same namespace, i.e. it is not allowed to have a luci.notifier(...) and a luci.tree_closer(...) with the same name.
SUCCESS
, FAILURE
, and INFRA_FAILURE
. Default is None.SUCCESS
, FAILURE
, and INFRA_FAILURE
. Default is None.on_new_status
or on_occurrence
instead. If True, notify on each build failure. Ignores transient (aka “infra”) failures. Default is False.on_new_status
or on_occurrence
instead. If True, notify on a build failure unless the previous build was a failure too. Ignores transient (aka “infra”) failures. Default is False.on_new_status
or on_occurrence
instead. If True, notify on each change to a build status (e.g. a green build becoming red and vice versa). Default is False.on_new_status
or on_occurrence
instead. If True, notify on each build success. Default is False.on_new_status
.failed_step_regexp
, but negated - this regex must not match any failed steps for a notification to be sent. Mutually exclusive with on_new_status
.repo
field in luci.builder(...). Default is False.https://host/repo
) to restrict the blamelist calculation to. If empty (default), only the primary repository associated with the builder is considered, see repo
field in luci.builder(...).default
is defined in the project somewhere, it is used implicitly by the notifier.notifies
field in luci.builder(...).luci.tree_closer( # Required arguments. name, tree_status_host, # Optional arguments. failed_step_regexp = None, failed_step_regexp_exclude = None, template = None, notified_by = None, )
Defines a rule for closing or opening a tree via a tree status app based on a status of the observed builders.
crbug.com/1054172
experiment.The set of builders that are being observed is defined through notified_by
field here or notifies
field in luci.builder(...). Whenever a build finishes, the builder “notifies” all (but usually none or just one) luci.tree_closer(...) objects subscribed to it, so they can decide whether to close or open the tree in reaction to the new builder state.
Note that luci.notifier(...) and luci.tree_closer(...) are both flavors of a luci.notifiable
object, i.e. both are something that “can be notified” when a build finishes. They both are valid targets for notifies
field in luci.builder(...). For that reason they share the same namespace, i.e. it is not allowed to have a luci.notifier(...) and a luci.tree_closer(...) with the same name.
tree_status_host
in luci.cq_group(...). Required.failed_step_regexp
, in which case it must also have a failed step matching that regular expression.default_tree_status
is defined in the project somewhere, it is used implicitly by the tree closer.notifies
field in luci.builder(...).luci.notifier_template(name, body)
Defines a template to use for notifications sent by luci.notifier(...) and luci.tree_closer(...).
The main template body should have format <subject>\n\n<body>
where subject is one line of text/template and body is an html/template. The body can either be specified inline right in the starlark script or loaded from an external file via io.read_file(...).
The input to both templates is a TemplateInput Go struct derived from TemplateInput proto message.
The following functions are available to templates in addition to the standard ones.
A {{.Build.Builder.Builder}} build completed <a href="https://ci.chromium.org/b/{{.Build.Id}}">Build {{.Build.Number}}</a> has completed with status {{.Build.Status}} on `{{.Build.EndTime | time}}`
A template can “import” subtemplates defined in all other luci.notifier_template(...). When rendering, all templates defined in the project are merged into one. Example:
# The actual email template which uses subtemplates defined below. In the real # life it might be better to load such large template from an external file # using io.read_file. luci.notifier_template( name = 'default', body = '\n'.join([ 'A {{.Build.Builder.Builder}} completed', '', 'A <a href="https://ci.chromium.org/b/{{.Build.Id}}">build</a> has completed.', '', 'Steps: {{template "steps" .}}', '', '{{template "footer"}}', ]), ) # This template renders only steps. It is "executed" by other templates. luci.notifier_template( name = 'steps', body = '{{range $step := .Build.Steps}}<li>{{$step.name}}</li>{{end}', ) # This template defines subtemplates used by other templates. luci.notifier_template( name = 'common', body = '{{define "footer"}}Have a nice day!{{end}}', )
preview_email command can render a template file to stdout.
bb get -json -A 8914184822697034512 | preview_email ./default.template
This example uses bb tool, available in depot_tools.
Command preview_email
is available in infra Go env and as a CIPD package.
If a user-defined template fails to render, a built-in template is used to generate a very short email with a link to the build and details about the failure.
^[a-z][a-z0-9\_]*$
. Required.luci.cq( # Optional arguments. submit_max_burst = None, submit_burst_delay = None, draining_start_time = None, status_host = None, )
Defines optional configuration of the CQ service for this project.
CQ is a service that monitors Gerrit CLs in a configured set of Gerrit projects, launches presubmit jobs (aka tryjobs) whenever a CL is marked as ready for CQ, and submits the CL if it passes all checks.
This optional rule can be used to set global CQ parameters that apply to all luci.cq_group(...) defined in the project.
submit_burst_delay
. This feature today applies to all attempts processed by CQ, across all luci.cq_group(...) instances. Optional, by default there's no limit. If used, requires submit_burst_delay
to be set too.submit_max_burst
is used.2017-12-23T15:47:58Z
and Z is mandatory.luci.cq_group( # Required arguments. name, watch, # Optional arguments. acls = None, allow_submit_with_open_deps = None, allow_owner_if_submittable = None, tree_status_host = None, retry_config = None, cancel_stale_tryjobs = None, verifiers = None, )
Defines a set of refs to be watched by the CQ and a set of verifiers to run whenever there's a pending approved CL for a ref in the watched set.
acl.CQ_*
roles are allowed here. By default ACLs are inherited from luci.project(...) definition. At least one acl.CQ_COMMITTER
entry should be provided somewhere (either here or in luci.project(...)).Code-Review
and other approvals regardless of acl.CQ_COMMITTER
or acl.CQ_DRY_RUNNER
roles. Only cq.ACTION_*
are allowed here. Default is cq.ACTION_NONE
which grants no additional permissions. CL owner is user owning a CL, i.e. its first patchset uploader, not to be confused with OWNERS files. WARNING: using this option is not recommended if you have sticky Code-Review
label because this allows a malicious developer to upload a good looking patchset at first, get code review approval, and then upload a bad patchset and CQ it right away.cq.RETRY_*
constants that define how CQ should retry failed builds. See CQ for more info. Default is cq.RETRY_TRANSIENT_FAILURES
.cq_tryjob_verifier(..., cancel_stale=...)
instead. Current default is NO, but see https://crbug.com/1001182 for the up-to-date information.luci.cq_tryjob_verifier(**entry)
and a string entry is an alias for luci.cq_tryjob_verifier(builder = entry)
.luci.cq_tryjob_verifier( # Required arguments. builder, # Optional arguments. cq_group = None, result_visibility = None, cancel_stale = None, includable_only = None, disable_reuse = None, experiment_percentage = None, location_regexp = None, location_regexp_exclude = None, owner_whitelist = None, equivalent_builder = None, equivalent_builder_percentage = None, equivalent_builder_whitelist = None, )
A verifier in a luci.cq_group(...) that triggers tryjobs for verifying CLs.
When processing a CL, the CQ examines a list of registered verifiers and launches new corresponding builds (called “tryjobs”) if it decides this is necessary (per the configuration of the verifier and the previous history of this CL).
The CQ automatically retries failed tryjobs (per configured retry_config
in luci.cq_group(...)) and only allows CL to land if each builder has succeeded in the latest retry. If a given tryjob result is too old (>1 day) it is ignored.
The CQ can examine a set of files touched by the CL and decide to skip this verifier. Touching a file means either adding, modifying or removing it.
This is controlled by location_regexp
and location_regexp_exclude
fields:
location_regexp
is specified and no file in a CL matches any of the location_regexp
, then the CQ will not care about this verifier.location_regexp_exclude
, then this file won't be considered when matching location_regexp
.location_regexp_exclude
is specified, but location_regexp
is not, location_regexp
is implied to be .*
.location_regexp
nor location_regexp_exclude
are specified (default), the verifier will be used on all CLs.The matches are done against the following string:
<gerrit_url>/<gerrit_project_name>/+/<cl_file_path>
The file path is relative to the repo root, and it uses Unix /
directory separator.
The comparison is a full match. The pattern is implicitly anchored with ^
and $
, so there is no need add them.
The pattern must use Google Re2 library syntax, documented here.
This filtering currently cannot be used in any of the following cases:
allow_submit_with_open_deps = True
.Please talk to CQ owners if these restrictions are limiting you.
Enable the verifier for all CLs touching any file in third_party/WebKit
directory of the chromium/src
repo, but not directory itself:
luci.cq_tryjob_verifier( location_regexp = [ 'https://chromium-review.googlesource.com/chromium/src/[+]/third_party/WebKit/.+', ], )
Match a CL which touches at least one file other than one.txt
inside all/
directory of the Gerrit project repo
:
luci.cq_tryjob_verifier( location_regexp = ['https://example.com/repo/[+]/.+'], location_regexp_exclude = ['https://example.com/repo/[+]/all/one.txt'], )
Match a CL which touches at least one file other than one.txt
in any repository or belongs to any other Gerrit server. Note, in this case location_regexp
defaults to .*
:
luci.cq_tryjob_verifier( location_regexp_exclude = ['https://example.com/repo/[+]/all/one.txt'], )
For builders which may be useful only for some CLs, predeclare them using includable_only=True
flag. Such builders will be triggered by CQ if and only if a CL opts in via CQ-Include-Trybots: <builder>
in its description.
For example, default verifiers may include only fast builders which skip low level assertions, but for coverage of such assertions one may add slower “debug” level builders into which CL authors opt-in as needed:
# triggered & required for all CLs. luci.cq_tryjob_verifier(builder="win") # triggered & required if only if CL opts in via # `CQ-Include-Trybots: project/try/win-debug`. luci.cq_tryjob_verifier(builder="win-debug", includable_only=True)
cq_tryjob_verifier
is used inline in luci.cq_group(...) declarations to provide per-builder verifier parameters. cq_group
argument can be omitted in this case:
luci.cq_group( name = 'Main CQ', ... verifiers = [ luci.cq_tryjob_verifier( builder = 'Presubmit', disable_reuse = True, ), ... ], )
It can also be associated with a luci.cq_group(...) outside of luci.cq_group(...) declaration. This is in particular useful in functions. For example:
luci.cq_group(name = 'Main CQ') def try_builder(name, ...): luci.builder(name = name, ...) luci.cq_tryjob_verifier(builder = name, cq_group = 'Main CQ')
cq_tryjob_verifier
is used inline inside some luci.cq_group(...) declaration.cq.COMMENT_LEVEL_FULL
and cq.COMMENT_LEVEL_RESTRICTED
constants. Default is to give full visibility: builder name and full summary markdown are included in the Gerrit comment.CQ-Include-Trybots:
on CL description. Default is False. See the explanation above for all details. For builders with experiment_percentage
or location_regexp
or location_regexp_exclude
, don't specify includable_only
. Such builders can already be forcefully added via CQ-Include-Trybots:
in CL description.equivalent_builder
instead of builder
. A choice itself is made deterministically based on CL alone, hereby all CQ attempts on all patchsets of a given CL will trigger the same builder, assuming CQ config doesn't change in the mean time. Note that if equivalent_builder_whitelist
is also specified, the choice over which of the two builders to trigger will be made only for CLs owned by the accounts in the whitelisted group. Defaults to 0, meaning the equivalent builder is never triggered by the CQ, but an existing build can be re-used.Below is the table with role constants that can be passed as roles
in acl.entry(...).
Due to some inconsistencies in how LUCI service are currently implemented, some roles can be assigned only in luci.project(...) rule, but some also in individual luci.bucket(...) or luci.cq_group(...) rules.
Similarly some roles can be assigned to individual users, other only to groups.
Role | Scope | Principals | Allows |
---|---|---|---|
acl.PROJECT_CONFIGS_READER | project only | groups, users | Reading contents of project configs through LUCI Config API/UI. |
acl.LOGDOG_READER | project only | groups | Reading logs under project's logdog prefix. |
acl.LOGDOG_WRITER | project only | groups | Writing logs under project's logdog prefix. |
acl.BUILDBUCKET_READER | project, bucket | groups, users | Fetching info about a build, searching for builds in a bucket. |
acl.BUILDBUCKET_TRIGGERER | project, bucket | groups, users | Same as BUILDBUCKET_READER + scheduling and canceling builds. |
acl.BUILDBUCKET_OWNER | project, bucket | groups, users | Full access to the bucket (should be used rarely). |
acl.SCHEDULER_READER | project, bucket | groups, users | Viewing Scheduler jobs, invocations and their debug logs. |
acl.SCHEDULER_TRIGGERER | project, bucket | groups, users | Same as SCHEDULER_READER + ability to trigger jobs. |
acl.SCHEDULER_OWNER | project, bucket | groups, users | Full access to Scheduler jobs, including ability to abort them. |
acl.CQ_COMMITTER | project, cq_group | groups | Committing approved CLs via CQ. |
acl.CQ_DRY_RUNNER | project, cq_group | groups | Executing presubmit tests for CLs via CQ. |
acl.entry( # Required arguments. roles, # Optional arguments. groups = None, users = None, projects = None, )
Returns an ACL binding which assigns given role (or roles) to given individuals, groups or LUCI projects.
Lists of acl.entry structs are passed to acls
fields of luci.project(...) and luci.bucket(...) rules.
An empty ACL binding is allowed. It is ignored everywhere. Useful for things like:
luci.project( acls = [ acl.entry(acl.PROJECT_CONFIGS_READER, groups = [ # TODO: members will be added later ]) ] )
acl.entry object, should be treated as opaque.
swarming.cache(path, name = None, wait_for_warm_cache = None)
Represents a request for the bot to mount a named cache to a path.
Each bot has a LRU of named caches: think of them as local named directories in some protected place that survive between builds.
A build can request one or more such caches to be mounted (in read/write mode) at the requested path relative to some known root. In recipes-based builds, the path is relative to api.paths['cache']
dir.
If it's the first time a cache is mounted on this particular bot, it will appear as an empty directory. Otherwise it will contain whatever was left there by the previous build that mounted exact same named cache on this bot, even if that build is completely irrelevant to the current build and just happened to use the same named cache (sometimes this is useful to share state between different builders).
At the end of the build the cache directory is unmounted. If at that time the bot is running out of space, caches (in their entirety, the named cache directory and all files inside) are evicted in LRU manner until there's enough free disk space left. Renaming a cache is equivalent to clearing it from the builder perspective. The files will still be there, but eventually will be purged by GC.
Additionally, Buildbucket always implicitly requests to mount a special builder cache to ‘builder’ path:
swarming.cache('builder', name=some_hash('<project>/<bucket>/<builder>'))
This means that any LUCI builder has a “personal disk space” on the bot. Builder cache is often a good start before customizing caching. In recipes, it is available at api.path['cache'].join('builder')
.
In order to share the builder cache directory among multiple builders, some explicitly named cache can be mounted to builder
path on these builders. Buildbucket will not try to override it with its auto-generated builder cache.
For example, if builders A and B both declare they use named cache swarming.cache('builder', name='my_shared_cache')
, and an A build ran on a bot and left some files in the builder cache, then when a B build runs on the same bot, the same files will be available in its builder cache.
If the pool of swarming bots is shared among multiple LUCI projects and projects mount same named cache, the cache will be shared across projects. To avoid affecting and being affected by other projects, prefix the cache name with something project-specific, e.g. v8-
.
api.path['cache']
). Must use POSIX format (forward slashes). In most cases, it does not need slashes at all. Must be unique in the given builder definition (cannot mount multiple caches to the same path). Required.path
itself. Must be unique in the given builder definition (cannot mount the same cache to multiple paths).swarming.cache struct with fields path
, name
and wait_for_warm_cache
.
swarming.dimension(value, expiration = None)
A value of some Swarming dimension, annotated with its expiration time.
Intended to be used as a value in dimensions
dict of luci.builder(...) when using dimensions that expire:
luci.builder( ... dimensions = { ... 'device': swarming.dimension('preferred', expiration=5*time.minute), ... }, ... )
swarming.dimension struct with fields value
and expiration
.
swarming.validate_caches(attr, caches)
Validates a list of caches.
Ensures each entry is swarming.cache struct, and no two entries use same name or path.
Validates list of caches (may be an empty list, never None).
swarming.validate_dimensions(attr, dimensions, allow_none = None)
Validates and normalizes a dict with dimensions.
The dict should have string keys and values are swarming.dimension, a string or a list of thereof (for repeated dimensions).
{string: string|swarming.dimension}
. Required.Validated and normalized dict in form {string: [swarming.dimension]}
.
swarming.validate_tags(attr, tags)
Validates a list of k:v
pairs with Swarming tags.
Validated list of tags in same order, with duplicates removed.
scheduler.policy( # Required arguments. kind, # Optional arguments. max_concurrent_invocations = None, max_batch_size = None, log_base = None, )
Policy for how LUCI Scheduler should handle incoming triggering requests.
This policy defines when and how LUCI Scheduler should launch new builds in response to triggering requests from luci.gitiles_poller(...) or from EmitTriggers RPC call.
The following batching strategies are supported:
scheduler.GREEDY_BATCHING_KIND
: use a greedy batching function that takes all pending triggering requests (up to max_batch_size
limit) and collapses them into one new build. It doesn't wait for a full batch, nor tries to batch evenly.scheduler.LOGARITHMIC_BATCHING_KIND
: use a logarithmic batching function that takes log(N) pending triggers (up to max_batch_size
limit) and collapses them into one new build, where N is the total number of pending triggers. The base of the logarithm is defined by log_base
.*_BATCHING_KIND
values above. Required.LOGARITHMIC_BATCHING_KIND
, ignored otherwise. Must be larger or equal to 1.0001 for numerical stability reasons.An opaque triggering policy object.
scheduler.greedy_batching(max_concurrent_invocations = None, max_batch_size = None)
A shortcut for scheduler.policy(scheduler.GREEDY_BATCHING_KIND, ...).
See scheduler.policy(...) for all details.
scheduler.logarithmic_batching(log_base, max_concurrent_invocations = None, max_batch_size = None)
A shortcut for scheduler.policy(scheduler.LOGARITHMIC_BATCHING_KIND, ...)
.
See scheduler.policy(...) for all details.
CQ module exposes structs and enums useful when defining luci.cq_group(...) entities.
cq.ACTION_*
constants define possible values for allow_owner_if_submittable
field of luci.cq_group(...):
CQ_COMMITTER
or CQ_DRY_RUNNER
(if any).CQ_DRY_RUNNER
role.CQ_COMMITTER
role.cq.RETRY_*
constants define some commonly used values for retry_config
field of luci.cq_group(...):
cq.COMMENT_LEVEL_*
constants define possible values for result_visibility
field of luci.cq_group(...):
cq.refset(repo, refs = None)
Defines a repository and a subset of its refs.
Used in watch
field of luci.cq_group(...) to specify what refs the CQ should be monitoring.
https://
. Only repositories hosted on *.googlesource.com
are supported currently. Required.refs/heads/.+
. If not set, defaults to refs/heads/master
.An opaque struct to be passed to watch
field of luci.cq_group(...).
cq.retry_config( # Optional arguments. single_quota = None, global_quota = None, failure_weight = None, transient_failure_weight = None, timeout_weight = None, )
Collection of parameters for deciding whether to retry a single build.
All parameters are integers, with default value of 0. The returned struct can be passed as retry_config
field to luci.cq_group(...).
Some commonly used presents are available as cq.RETRY_*
constants. See CQ for more info.
cq.retry_config struct.
Refer to the list of built-in constants and functions exposed in the global namespace by Starlark itself.
In addition, lucicfg
exposes the following functions.
__load(module)
Loads another Starlark module (if it haven't been loaded before), extracts one or more values from it, and binds them to names in the current module.
A load statement requires at least two “arguments”. The first must be a literal string, it identifies the module to load. The remaining arguments are a mixture of literal strings, such as 'x'
, or named literal strings, such as y='x'
.
The literal string ('x'
), which must denote a valid identifier not starting with _
, specifies the name to extract from the loaded module. In effect, names starting with _
are not exported. The name (y
) specifies the local name. If no name is given, the local name matches the quoted name.
load('//module.star', 'x', 'y', 'z') # assigns x, y, and z load('//module.star', 'x', y2='y', 'z') # assigns x, y2, and z
A load statement within a function is a static error.
See also Modules and packages for how load(...) interacts with exec(...).
//path/within/current/package.star
or @<pkg>//path/within/pkg.star
or ./relative/path.star
. Required.exec(module)
Executes another Starlark module for its side effects.
See also Modules and packages for how load(...) interacts with exec(...).
//path/within/current/package.star
or @<pkg>//path/within/pkg.star
or ./relative/path.star
. Required.A struct with all exported symbols of the executed module.
fail(msg, trace = None)
Aborts the execution with an error message.
fail
is called.stacktrace(skip = None)
Captures and returns a stack trace of the caller.
A captured stacktrace is an opaque object that can be stringified to get a nice looking trace (e.g. for error messages).
struct(**kwargs)
Returns an immutable struct object with fields populated from the specified keyword arguments.
Can be used to define namespaces, for example:
def _func1(): ... def _func2(): ... exported = struct( func1 = _func1, func2 = _func2, )
Then _func1
can be called as exported.func1()
.
to_json(value)
Serializes a value to a compact JSON string.
Doesn't support integers that do not fit int64. Fails if the value has cycles.
proto.to_textpb(msg)
Serializes a protobuf message to a string using ASCII proto serialization.
proto.to_jsonpb(msg)
Serializes a protobuf message to a string using JSONPB serialization.
proto.to_wirepb(msg)
Serializes a protobuf message to a string using binary wire encoding.
proto.from_textpb(ctor, text)
Deserializes a protobuf message given its ASCII proto serialization.
Deserialized message constructed via ctor
.
proto.from_jsonpb(ctor, text)
Deserializes a protobuf message given its JSONPB serialization.
Deserialized message constructed via ctor
.
proto.from_wirepb(ctor, blob)
Deserializes a protobuf message given its wire serialization.
Deserialized message constructed via ctor
.
proto.struct_to_textpb(s = None)
Converts a struct to a text proto string.
A str containing a text format protocol buffer message.
io.read_file(path)
Reads a file and returns its contents as a string.
Useful for rules that accept large chunks of free form text. By using io.read_file
such text can be kept in a separate file.
//
) an absolute path within the currently executing package. If it is a relative path, it must point somewhere inside the current package directory. Required.The contents of the file as a string. Fails if there‘s no such file, it can’t be read, or it is outside of the current package directory.
io.read_proto(ctor, path, encoding = None)
Reads a serialized proto message from a file, deserializes and returns it.
//
) an absolute path within the currently executing package. If it is a relative path, it must point somewhere inside the current package directory. Required.jsonpb
or textpb
or auto
to detect based on the file extension. Default is auto
.Deserialized proto message constructed via ctor
.