Recipes are a domain-specific language (embedded in python) for specifying sequences of subprocess calls in a cross-platform and testable way.
Chromium uses BuildBot for its builds. It requires master restarts to change bot configs, which slows bot changes down.
With Recipes, most build-related things happen in scripts that run on the slave, which means that the master does not need to be restarted in order to change something about a build configuration.
Recipes also provide a way to unit test build scripts, by mocking commands and recording “expectations” of what will happen when the script runs under various conditions. This makes it easy to verify that the scope of a change is limited.
This README will seek to teach the ways of Recipes, so that you may do one or more of the following:
The document will build knowledge up in small steps using examples, and so it's probably best to read the whole doc through from top to bottom once before using it as a reference.
Recipes are a means to cause a series of commands to run on a machine.
All recipes take the form of a python file whose body looks like this:
DEPS = ['recipe_engine/step'] def RunSteps(api): api.step('Print Hello World', ['echo', 'hello', 'world'])
The RunSteps function is expected to take at least a single argument api (we'll get to that in more detail later), and run a series of steps by calling api functions. All of these functions will eventually make calls to api.step(), which is the only way to actually get anything done on the machine. Using python libraries with OS side-effects is prohibited to enable testing.
For these examples we will work out of the tools/build repository.
Put this in a file under scripts/slave/recipes/hello.py. You can then run this recipe by calling
$ scripts/slave/recipes.py run hello
run_recipe on the setup_build step which provides a precise invocation for run_recipe.py correlating exactly with the current recipe invocation. This is useful to locally repro a failing build without having to guess at the parameters to run_recipe.py.All recipes MUST have corresponding tests, which achieve 100% code coverage.
So, we have our recipe. Let's add a test to it.
DEPS = ['recipe_engine/step'] def RunSteps(api): api.step('Print Hello World', ['echo', 'hello', 'world']) def GenTests(api): yield api.test('basic')
This causes a single test case to be generated, called ‘basic’, which has no input parameters. As your recipe becomes more complex, you'll need to add more tests to make sure that you maintain 100% code coverage.
In order to run the tests, run
$ scripts/slave/recipes.py test train --filter hello
This will write the file build/scripts/slave/recipes/hello.expected/basic.json summarizing the actions of the recipe under the boring conditions specified by api.test('basic').
[
{
"cmd": [
"echo",
"hello",
"world"
],
"cwd": "[SLAVE_BUILD]",
"name": "Print Hello World"
}
]
In order to do something useful, we need to pull in parameters from the outside world. There's one primary source of input for recipes, which is properties.
Properties are a relic from the days of BuildBot, though they have been dressed up a bit to be more like we‘ll want them in the future. If you’re familiar with BuildBot, you'll probably know them as factory_properties and build_properties. The new properties object is a merging of these two, and is provided by the properties api module.
This is now abstracted into the PROPERTIES top level declaration in your recipe. You declare a dictionary of properties that your recipe accepts. The recipe engine will extract the properties your recipe cares about from all the properties it knows about, and pass them as arguments to your RunSteps function.
Let's see an example!
from recipe_engine.recipe_api import Property DEPS = [ 'step', 'properties', ] PROPERTIES = { 'target_of_admiration': Property( kind=str, help="Who you love and adore.", default="Chrome Infra"), } def RunSteps(api, target_of_admiration): verb = 'Hello, %s' if target_of_admiration == 'DarthVader': verb = 'Die in a fire, %s!' api.step('Greet Admired Individual', ['echo', verb % target_of_admiration]) def GenTests(api): yield api.test('basic') + api.properties(target_of_admiration='Bob') yield api.test('vader') + api.properties(target_of_admiration='DarthVader') yield api.test('infra rocks')
The property list is a whitelist, so if the properties provided as inputs to the current recipe run were
{ 'target_of_admiration': 'Darth Vader', 'some_other_chill_thing': 'so_chill', }
then the recipe wouldn't know about the other some_other_chill_thing property at all.
Note that properties without a default are required. If you don't want a property to be required, just add default=None to the definition.
Yes, elements of a test specification are combined with + and it's weird.
To specify property values in a local run:
build/scripts/tools/run_recipe.py <recipe-name> opt=bob other=sally
Or, more explicitly::
build/scripts/tools/run_recipe.py --properties-file <path/to/json>
Where <path/to/json> is a file containing a valid json object (i.e. key:value pairs).
Note that we need to put a dependency on the ‘properties’ module in the DEPS because we use it to generate our tests, even though we don't actually call the module in our code. See this crbug.com/532275 for more info.
There are all sorts of helper modules. They are found in the recipe_modules directory alongside the recipes directory where the recipes go.
Notice the DEPS line in the recipe. Any modules named by string in DEPS are ‘injected’ into the api parameter that your recipe gets. If you leave them out of DEPS, you'll get an AttributeError when you try to access them. The modules are located primarily in recipe_modules/, and their name is their folder name.
There are a whole bunch of modules which provide really helpful tools. You should go take a look at them. scripts/slave/recipes.py is a pretty helpful tool. If you want to know more about properties, step and path, I would suggest starting with scripts/slave/recipes.py doc, and then delving into the helpful docstrings in those helpful modules.
Modules are for grouping functionality together and exposing it across recipes.
So now you feel like you're pretty good at recipes, but you want to share your echo functionality across a couple recipes which all start the same way. To do this, you need to add a module directory.
recipe_modules/
step/
properties/
path/
hello/
__init__.py # (Required) Contains optional `DEPS = list([other modules])`
api.py # (Required) Contains single required RecipeApi-derived class
config.py # (Optional) Contains configuration for your api
*_config.py # (Optional) These contain extensions to the configurations of
# your dependency APIs
First add an __init__.py with DEPS:
# recipe_modules/hello/__init__.py from recipe_api import Property DEPS = ['properties', 'step'] PROPERTIES = { 'target_of_admiration': Property(default=None), }
And your api.py should look something like:
from slave import recipe_api class HelloApi(recipe_api.RecipeApi): def __init__(self, target_of_admiration): self._target = target_of_admiration def greet(self, default_verb=None): verb = default_verb or 'Hello %s' if self._target == 'DarthVader': verb = 'Die in a fire %s!' self.m.step('Hello World', ['echo', verb % self._target])
Note that all the DEPS get injected into self.m. This logic is handled outside of the object (i.e. not in __init__).
Because dependencies are injected after module initialization, you do not have access to injected modules in your APIs
__init__method!
And now, our refactored recipe:
DEPS = ['hello'] def RunSteps(api): api.hello.greet() def GenTests(api): yield api.test('basic') + api.properties(target_of_admiration='Bob') yield api.test('vader') + api.properties(target_of_admiration='DarthVader')
NOTE: all of the modules are also require 100% code coverage, but you only need coverage from SOME recipe.
The basic form of tests is:
def GenTests(api): yield api.test('testname') + # other stuff
Some modules define interfaces for specifying necessary step data; these are injected into api from DEPS similarly to how it works for RunSteps. There are a few other methods available to GenTests's api. Common ones include:
api.properties(buildername='foo_builder') sets properties as we have seen.api.platform('linux', 32) sets the mock platform to 32-bit linux.api.step_data('Hello World', retcode=1) mocks the 'Hello World' step to have failed with exit code 1.By default all simulated steps succeed, the platform is 64-bit linux, and there are no properties. The api.properties.generic() method populates some common properties for Chromium recipes.
The api passed to GenTests is confusingly NOT the same as the recipe api. It's actually an instance of recipe_test_api.py:RecipeTestApi(). This is admittedly pretty weak, and it would be great to have the test api automatically created via modules. On the flip side, the test api is much less necessary than the recipe api, so this transformation has not been designed yet.
Configs are a way for a module to expose it's “global” state in a reusable way.
A common problem in Building Things is that you end up with an inordinantly large matrix of configurations. Let's take chromium, for example. Here is a sample list of axes of configuration which chromium needs to build and test:
Obviously there are a lot of combinations of those things, but only a relatively small number of valid combinations of those things. How can we represent all the valid states while still retaining our sanity?
We begin by specifying a schema that configurations of the hello module will follow, and the config context based on it that we will add configuration items to.
# recipe_modules/hello/config.py from slave.recipe_config import config_item_context, ConfigGroup from slave.recipe_config import SimpleConfig, StaticConfig, BadConf def BaseConfig(TARGET='Bob'): # This is a schema for the 'config blobs' that the hello module deals with. return ConfigGroup( verb = SimpleConfig(str), # A config blob is not complete() until all required entries have a value. tool = SimpleConfig(str, required=True), # Generally, your schema should take a series of CAPITAL args which will be # set as StaticConfig data in the config blob. TARGET = StaticConfig(str(TARGET)), ) config_ctx = config_item_context(BaseConfig)
The BaseConfig schema is expected to return a ConfigGroup instance of some sort. All the configs that you get out of this file will be a modified version of something returned by the schema method. The arguments should have sane defaults, and should be named in ALL_CAPS (this is to avoid argument name conflicts as we'll see later).
config_ctx is the ‘context’ for all the config items in this file, and will magically become the CONFIG_CTX for the entire module. Other modules may extend this context, which we will get to later.
Finally let's define some config items themselves. A config item is a function decorated with the config_ctx, and takes a config blob as ‘c’. The config item updates the config blob, perhaps conditionally. There are many features to slave/recipe_config.py. I would recommend reading the docstrings there for all the details.
# Each of these functions is a 'config item' in the context of config_ctx. # is_root means that every config item will apply this item first. @config_ctx(is_root=True) def BASE(c): if c.TARGET == 'DarthVader': c.verb = 'Die in a fire, %s!' else: c.verb = 'Hello, %s' @config_ctx(group='tool'): # items with the same group are mutually exclusive. def super_tool(c): if c.TARGET != 'Charlie': raise BadConf('Can only use super tool for Charlie!') c.tool = 'unicorn.py' @config_ctx(group='tool'): def default_tool(c): c.tool = 'echo'
Now that we have our config, let's use it.
# recipe_modules/hello/api.py from slave import recipe_api class HelloApi(recipe_api.RecipeApi): def __init__(self, target_of_admiration): self._target = target_of_admiration def get_config_defaults(self, _config_name): return {'TARGET': self._target} def greet(self): self.m.step('Hello World', [ self.m.path.build(self.c.tool), self.c.verb % self.c.TARGET])
Note that recipe_api.RecipeApi contains all the plumbing for dealing with configs. If your module has a config, you can access its current value via self.c. The users of your module (read: recipes) will need to set this value in one way or another. Also note that c is a ‘public’ variable, which means that recipes have direct access to the configuration state by api.<modname>.c.
# recipes/hello.py DEPS = ['hello'] def RunSteps(api): api.hello.set_config('default_tool') api.hello.greet() # Greets 'target_of_admiration' or 'Bob' with echo. def GenTests(api): yield api.test('bob') yield api.test('anya') + api.properties(target_of_admiration='anya')
Note the call to set_config. This method takes the configuration name specifed, finds it in the given module ('hello' in this case), and sets api.hello.c equal to the result of invoking the named config item ('default_tool') with the default configuration (the result of calling get_config_defaults), merged over the static defaults specified by the schema.
We can also call set_config differently to get different results:
# recipes/rainbow_hello.py DEPS = ['hello'] def RunSteps(api): api.hello.set_config('super_tool', TARGET='Charlie') api.hello.greet() # Greets 'Charlie' with unicorn.py. def GenTests(api): yield api.test('charlie')
# recipes/evil_hello.py DEPS = ['hello'] def RunSteps(api): api.hello.set_config('default_tool', TARGET='DarthVader') api.hello.greet() # Causes 'DarthVader' to despair with echo def GenTests(api): yield api.test('darth')
set_config() also has one additional bit of magic. If a module (say, chromium), depends on some other modules (say, gclient), if you do api.chromium.set_config('blink'), it will apply the 'blink' config item from the chromium module, but it will also attempt to apply the 'blink' config for all the dependencies, too. This way, you can have the chromium module extend the gclient config context with a ‘blink’ config item, and then set_configs will stack across all the relevent contexts. (This has since been recognized as a design mistake)
recipe_api.RecipeApi also provides make_config and apply_config, which allow recipes more-direct access to the config items. However, set_config() is the most-preferred way to apply configurations.
Consider this recipe:
DEPS = ['step', 'path'] def RunSteps(api): step_result = api.step('Determine blue moon', [api.path['build'].join('is_blue_moon.sh')]) if step_result.retcode == 0: api.step('HARLEM SHAKE!', [api.path['build'].join('do_the_harlem_shake.sh')]) else: api.step('Boring', [api.path['build'].join('its_a_small_world.sh')]) def GenTests(api): yield api.test('harlem') + api.step_data('Determine blue moon', retcode=0) yield api.test('boring') + api.step_data('Determine blue moon', retcode=1)
See how we use step_result to get the result of the last step? The item we get back is a recipe_engine.main.StepData instance (really, just a basic object with member data). The members of this object which are guaranteed to exist are:
retcode: Pretty much what you thinkstep: The actual step json which was sent to annotator.py. Not usually useful for recipes, but it is used internally for the recipe tests framework.presentation: An object representing how the step will show up on the build page, including its exit status, links, and extra log text. This is a recipe_engine.main.StepPresentation object. See also How to change step presentation.This is pretty neat... However, it turns out that returncodes suck bigtime for communicating actual information. api.json.output() to the rescue!
DEPS = ['step', 'path', 'step_history', 'json'] def RunSteps(api): step_result = api.step( 'run tests', [api.path['build'].join('do_test_things.sh'), api.json.output()]) num_passed = step_result.json.output['num_passed'] if num_passed > 500: api.step('victory', [api.path['build'].join('do_a_dance.sh')]) elif num_passed > 200: api.step('not defeated', [api.path['build'].join('woohoo.sh')]) else: api.step('deads!', [api.path['build'].join('you_r_deads.sh')]) def GenTests(api): yield (api.test('winning') + api.step_data('run tests', api.json.output({'num_passed': 791})) yield (api.test('not_dead_yet') + api.step_data('run tests', api.json.output({'num_passed': 302})) yield (api.test('noooooo') + api.step_data('run tests', api.json.output({'num_passed': 10})))
api.json.output() returns a recipe_api.Placeholder which is meant to be added into a step command list. When the step runs, the placeholder gets rendered into some strings (in this case, like ‘/tmp/some392ra8’). When the step finishes, the Placeholder adds data to the StepData object for the step which just ran, namespaced by the module name (in this case, the ‘json’ module decided to add an ‘output’ attribute to the step_history item). I'd encourage you to take a peek at the implementation of the json module to see how this is implemented.
api.step(..., stdin=api.raw_io.input('test input'))
Also see raw_io's example.
step_result = api.step(..., stdout=api.json.output()) data = step_result.stdout # data is a parsed JSON value, such as dict
Also see json's example.
data = {'value': 1} api.step(..., stdin=api.json.input(data))
Also see json's example.
This example specifies the standard output that should be returned when a step is executed in simulation mode. This is typically used for specifying default test data in the recipe or recipe module and removes the need to specify too much test data for each test in GenTests:
api.step(..., step_test_data=api.raw_io.output('test data'))
yield ( api.test('my_test') + api.step_data( 'step_name', output=api.raw_io.output('test data')))
step_result.presentation allows modifying the appearance of a step:
step_result.presentation.logs['mylog'] = ['line1', 'line2']
Creates an extra log “mylog” under the step.
api.properties are immutable, but you can change and add new properties at the buildbot level.
step_result.presentation.properties['newprop'] = 1
This modifies the text displayed next to a step name:
step_result = api.step(...) step_result.presentation.step_text = 'Dynamic step result text'
presentaton.logs allows creating extra logs of a step run. Example:step_result.presentation.logs['mylog'] = ['line1', 'line2']
step_result.presentation.properties['newprop'] = 1
Use scripts/slave/recipes.py doc. It's super effective!
Each repo has a recipes.py entry point under recipes_path from recipes.cfg .
Execute the following commands: ./recipes.py test run ./recipes.py test train
Specifically, for tools/build repo, the commands to execute are: scripts/slave/recipes.py test run scripts/slave/recipes.py test train
*.py?Check the docstrings in *.py. <trollface text="Problem?"/>
In addition, most recipe modules have example recipes in the examples subfolder which exercises most of the code in the module for example purposes.