doc/user_guide.md

# Recipes

Recipes are a domain-specific language (embedded in python) for specifying
sequences of subprocess calls in a cross-platform and testable way.

[TOC]

## Background

Chromium uses BuildBot for its builds.  It requires master restarts to change
bot configs, which slows bot changes down.

With Recipes, most build-related things happen in scripts that run on the
slave, which means that the master does not need to be restarted in order
to change something about a build configuration.

Recipes also provide a way to unit test build scripts, by mocking commands and
recording "expectations" of what will happen when the script runs under various
conditions.  This makes it easy to verify that the scope of a change is limited.

## Intro

This README will seek to teach the ways of Recipes, so that you may do one or
more of the following:

  * Read them
  * Make new recipes
  * Fix bugs in recipes
  * Create libraries (api modules) for others to use in their recipes.

The document will build knowledge up in small steps using examples, and so it's
probably best to read the whole doc through from top to bottom once before using
it as a reference.

## Small Beginnings

**Recipes are a means to cause a series of commands to run on a machine.**

All recipes take the form of a python file whose body looks like this:

```python
DEPS = ['recipe_engine/step']

def RunSteps(api):
  api.step('Print Hello World', ['echo', 'hello', 'world'])
```

The `RunSteps` function is expected to take at least a single argument `api`
(we'll get to that in more detail later), and run a series of steps by calling
api functions.  All of these functions will eventually make calls to
`api.step()`, which is the only way to actually get anything done on the
machine.  Using python libraries with OS side-effects is prohibited to enable
testing.

For these examples we will work out of the
[tools/build](https://chromium.googlesource.com/chromium/tools/build/)
repository.

Put this in a file under `scripts/slave/recipes/hello.py`. You can then
run this recipe by calling

    $ scripts/slave/recipes.py run hello

*** promo
Note: every recipe execution (e.g. build on buildbot) emits
a step log called `run_recipe` on the `setup_build` step which provides
a precise invocation for `run_recipe.py` correlating exactly with the current
recipe invocation. This is useful to locally repro a failing build without
having to guess at the parameters to `run_recipe.py`.
***

## We should probably test as we go...

**All recipes MUST have corresponding tests, which achieve 100% code coverage.**

So, we have our recipe. Let's add a test to it.

```python
DEPS = ['recipe_engine/step']

def RunSteps(api):
  api.step('Print Hello World', ['echo', 'hello', 'world'])

def GenTests(api):
  yield api.test('basic')
```

This causes a single test case to be generated, called 'basic', which has no
input parameters.  As your recipe becomes more complex, you'll need to add more
tests to make sure that you maintain 100% code coverage.

In order to run the tests, run

    $ scripts/slave/recipes.py test train --filter hello

This will write the file `build/scripts/slave/recipes/hello.expected/basic.json`
summarizing the actions of the recipe under the boring conditions
specified by `api.test('basic')`.

    [
      {
        "cmd": [
          "echo",
          "hello",
          "world"
        ],
        "cwd": "[SLAVE_BUILD]",
        "name": "Print Hello World"
      }
    ]

## Let's do something useful

### Properties are the primary input for your recipes

In order to do something useful, we need to pull in parameters from the outside
world. There's one primary source of input for recipes, which is `properties`.

Properties are a relic from the days of BuildBot, though they have been
dressed up a bit to be more like we'll want them in the future. If you're
familiar with BuildBot, you'll probably know them as `factory_properties` and
`build_properties`. The new `properties` object is a merging of these two, and
is provided by the `properties` api module.

This is now abstracted into the PROPERTIES top level declaration in your recipe.
You declare a dictionary of properties that your recipe accepts. The recipe
engine will extract the properties your recipe cares about from all the
properties it knows about, and pass them as arguments to your RunSteps function.

Let's see an example!

```python
from recipe_engine.recipe_api import Property

DEPS = [
  'step',
  'properties',
]

PROPERTIES = {
  'target_of_admiration': Property(
    kind=str, help="Who you love and adore.", default="Chrome Infra"),
}

def RunSteps(api, target_of_admiration):
  verb = 'Hello, %s'
  if target_of_admiration == 'DarthVader':
    verb = 'Die in a fire, %s!'
  api.step('Greet Admired Individual', ['echo', verb % target_of_admiration])

def GenTests(api):
  yield api.test('basic') + api.properties(target_of_admiration='Bob')
  yield api.test('vader') + api.properties(target_of_admiration='DarthVader')
  yield api.test('infra rocks')
```

The property list is a whitelist, so if the properties provided as inputs to the
current recipe run were

```python
{
  'target_of_admiration': 'Darth Vader',
  'some_other_chill_thing': 'so_chill',
}
```

then the recipe wouldn't know about the other `some_other_chill_thing` property
at all.

Note that properties without a default are required. If you don't want a
property to be required, just add `default=None` to the definition.

Yes, elements of a test specification are combined with `+` and it's weird.

To specify property values in a local run:

    build/scripts/tools/run_recipe.py <recipe-name> opt=bob other=sally

Or, more explicitly::

    build/scripts/tools/run_recipe.py --properties-file <path/to/json>

Where `<path/to/json>` is a file containing a valid json `object` (i.e.
key:value pairs).

Note that we need to put a dependency on the 'properties' module in the DEPS
because we use it to generate our tests, even though we don't actually call
the module in our code.
See this [crbug.com/532275](bug) for more info.

### Modules

There are all sorts of helper modules.  They are found in the `recipe_modules`
directory alongside the `recipes` directory where the recipes go.

Notice the `DEPS` line in the recipe. Any modules named by string in DEPS are
'injected' into the `api` parameter that your recipe gets. If you leave them out
of DEPS, you'll get an AttributeError when you try to access them. The modules
are located primarily in `recipe_modules/`, and their name is their folder name.

There are a whole bunch of modules which provide really helpful tools. You
should go take a look at them. `scripts/slave/recipes.py` is a
pretty helpful tool. If you want to know more about properties, step and path, I
would suggest starting with `scripts/slave/recipes.py doc`, and then delving
into the helpful docstrings in those helpful modules.

## Making Modules

**Modules are for grouping functionality together and exposing it across
recipes.**

So now you feel like you're pretty good at recipes, but you want to share your
echo functionality across a couple recipes which all start the same way. To do
this, you need to add a module directory.

```
recipe_modules/
  step/
  properties/
  path/
  hello/
    __init__.py  # (Required) Contains optional `DEPS = list([other modules])`
    api.py       # (Required) Contains single required RecipeApi-derived class
    config.py    # (Optional) Contains configuration for your api
    *_config.py  # (Optional) These contain extensions to the configurations of
                 #   your dependency APIs
```

First add an `__init__.py` with DEPS:

```python
# recipe_modules/hello/__init__.py
from recipe_api import Property

DEPS = ['properties', 'step']
PROPERTIES = {
  'target_of_admiration': Property(default=None),
}
```

And your api.py should look something like:

```python
from slave import recipe_api

class HelloApi(recipe_api.RecipeApi):
  def __init__(self, target_of_admiration):
    self._target = target_of_admiration

  def greet(self, default_verb=None):
    verb = default_verb or 'Hello %s'
    if self._target == 'DarthVader':
      verb = 'Die in a fire %s!'
    self.m.step('Hello World',
                ['echo', verb % self._target])
```

Note that all the DEPS get injected into `self.m`. This logic is handled outside
of the object (i.e. not in `__init__`).

> Because dependencies are injected after module initialization, *you do not
> have access to injected modules in your APIs `__init__` method*!

And now, our refactored recipe:

```python
DEPS = ['hello']

def RunSteps(api):
  api.hello.greet()

def GenTests(api):
  yield api.test('basic') + api.properties(target_of_admiration='Bob')
  yield api.test('vader') + api.properties(target_of_admiration='DarthVader')
```

> NOTE: all of the modules are also require 100% code coverage, but you only
> need coverage from SOME recipe.

## So how do I really write those tests?

The basic form of tests is:

```python
def GenTests(api):
  yield api.test('testname') + # other stuff
```

Some modules define interfaces for specifying necessary step data; these are
injected into `api` from `DEPS` similarly to how it works for `RunSteps`.  There
are a few other methods available to `GenTests`'s `api`. Common ones include:

  * `api.properties(buildername='foo_builder')` sets properties as we have seen.
  * `api.platform('linux', 32)` sets the mock platform to 32-bit linux.
  * `api.step_data('Hello World', retcode=1)` mocks the `'Hello World'` step
  to have failed with exit code 1.

By default all simulated steps succeed, the platform is 64-bit linux, and
there are no properties.  The `api.properties.generic()` method populates some
common properties for Chromium recipes.

The `api` passed to GenTests is confusingly **NOT** the same as the recipe api.
It's actually an instance of `recipe_test_api.py:RecipeTestApi()`. This is
admittedly pretty weak, and it would be great to have the test api
automatically created via modules. On the flip side, the test api is much less
necessary than the recipe api, so this transformation has not been designed yet.

## What is that config business?

**Configs are a way for a module to expose it's "global" state in a reusable
way.**

A common problem in Building Things is that you end up with an inordinantly
large matrix of configurations. Let's take chromium, for example. Here is a
sample list of axes of configuration which chromium needs to build and test:

  * BUILD_CONFIG
  * HOST_PLATFORM
  * HOST_ARCH
  * HOST_BITS
  * TARGET_PLATFORM
  * TARGET_ARCH
  * TARGET_BITS
  * builder type (ninja? msvs? xcodebuild?)
  * compiler
  * ...

Obviously there are a lot of combinations of those things, but only a relatively
small number of *valid* combinations of those things. How can we represent all
the valid states while still retaining our sanity?

We begin by specifying a schema that configurations of the `hello` module
will follow, and the config context based on it that we will add configuration
items to.

```python
# recipe_modules/hello/config.py
from slave.recipe_config import config_item_context, ConfigGroup
from slave.recipe_config import SimpleConfig, StaticConfig, BadConf

def BaseConfig(TARGET='Bob'):
  # This is a schema for the 'config blobs' that the hello module deals with.
  return ConfigGroup(
    verb   = SimpleConfig(str),
    # A config blob is not complete() until all required entries have a value.
    tool   = SimpleConfig(str, required=True),
    # Generally, your schema should take a series of CAPITAL args which will be
    # set as StaticConfig data in the config blob.
    TARGET = StaticConfig(str(TARGET)),
  )

config_ctx = config_item_context(BaseConfig)
```

The `BaseConfig` schema is expected to return a `ConfigGroup` instance of some
sort. All the configs that you get out of this file will be a modified version
of something returned by the schema method. The arguments should have sane
defaults, and should be named in `ALL_CAPS` (this is to avoid argument name
conflicts as we'll see later).

`config_ctx` is the 'context' for all the config items in this file, and will
magically become the `CONFIG_CTX` for the entire module.  Other modules may
extend this context, which we will get to later.

Finally let's define some config items themselves. A config item is a function
decorated with the `config_ctx`, and takes a config blob as 'c'. The config item
updates the config blob, perhaps conditionally. There are many features to
`slave/recipe_config.py`. I would recommend reading the docstrings there
for all the details.

```python
# Each of these functions is a 'config item' in the context of config_ctx.

# is_root means that every config item will apply this item first.
@config_ctx(is_root=True)
def BASE(c):
  if c.TARGET == 'DarthVader':
    c.verb = 'Die in a fire, %s!'
  else:
    c.verb = 'Hello, %s'

@config_ctx(group='tool'):  # items with the same group are mutually exclusive.
def super_tool(c):
  if c.TARGET != 'Charlie':
    raise BadConf('Can only use super tool for Charlie!')
  c.tool = 'unicorn.py'

@config_ctx(group='tool'):
def default_tool(c):
  c.tool = 'echo'
```

Now that we have our config, let's use it.

```python
# recipe_modules/hello/api.py
from slave import recipe_api

class HelloApi(recipe_api.RecipeApi):
  def __init__(self, target_of_admiration):
    self._target = target_of_admiration

  def get_config_defaults(self, _config_name):
    return {'TARGET': self._target}

  def greet(self):
    self.m.step('Hello World', [
        self.m.path.build(self.c.tool), self.c.verb % self.c.TARGET])
```

Note that `recipe_api.RecipeApi` contains all the plumbing for dealing with
configs. If your module has a config, you can access its current value via
`self.c`. The users of your module (read: recipes) will need to set this value
in one way or another. Also note that c is a 'public' variable, which means that
recipes have direct access to the configuration state by `api.<modname>.c`.

```python
# recipes/hello.py
DEPS = ['hello']
def RunSteps(api):
  api.hello.set_config('default_tool')
  api.hello.greet()  # Greets 'target_of_admiration' or 'Bob' with echo.

def GenTests(api):
  yield api.test('bob')
  yield api.test('anya') + api.properties(target_of_admiration='anya')
```

Note the call to `set_config`. This method takes the configuration name
specifed, finds it in the given module (`'hello'` in this case), and sets
`api.hello.c` equal to the result of invoking the named config item
(`'default_tool'`) with the default configuration (the result of calling
`get_config_defaults`), merged over the static defaults specified by the schema.

We can also call `set_config` differently to get different results:

```python
# recipes/rainbow_hello.py
DEPS = ['hello']
def RunSteps(api):
  api.hello.set_config('super_tool', TARGET='Charlie')
  api.hello.greet()  # Greets 'Charlie' with unicorn.py.

def GenTests(api):
  yield api.test('charlie')
```

```python
# recipes/evil_hello.py
DEPS = ['hello']
def RunSteps(api):
  api.hello.set_config('default_tool', TARGET='DarthVader')
  api.hello.greet()  # Causes 'DarthVader' to despair with echo

def GenTests(api):
  yield api.test('darth')
```

`set_config()` also has one additional bit of magic. If a module (say,
`chromium`), depends on some other modules (say, `gclient`), if you do
`api.chromium.set_config('blink')`, it will apply the `'blink'` config item from
the chromium module, but it will also attempt to apply the `'blink'` config for
all the dependencies, too. This way, you can have the chromium module extend the
gclient config context with a 'blink' config item, and then `set_configs` will
stack across all the relevent contexts.  (This has since been recognized as a
design mistake)

`recipe_api.RecipeApi` also provides `make_config` and `apply_config`, which
allow recipes more-direct access to the config items. However, `set_config()` is
the most-preferred way to apply configurations.

## What about getting data back from a step?

Consider this recipe:

```python
DEPS = ['step', 'path']

def RunSteps(api):
  step_result = api.step('Determine blue moon',
      [api.path['build'].join('is_blue_moon.sh')])

  if step_result.retcode == 0:
    api.step('HARLEM SHAKE!', [api.path['build'].join('do_the_harlem_shake.sh')])
  else:
    api.step('Boring', [api.path['build'].join('its_a_small_world.sh')])

def GenTests(api):
  yield api.test('harlem') + api.step_data('Determine blue moon', retcode=0)
  yield api.test('boring') + api.step_data('Determine blue moon', retcode=1)
```

See how we use `step_result` to get the result of the last step? The item we get
back is a `recipe_engine.main.StepData` instance (really, just a basic object
with member data). The members of this object which are guaranteed to exist are:
  * `retcode`: Pretty much what you think
  * `step`: The actual step json which was sent to `annotator.py`. Not usually
    useful for recipes, but it is used internally for the recipe tests
    framework.
  * `presentation`: An object representing how the step will show up on the
    build page, including its exit status, links, and extra log text.  This is a
    `recipe_engine.main.StepPresentation` object.
    See also
    [How to change step presentation](#how-to-change-step-presentation).

This is pretty neat... However, it turns out that returncodes suck bigtime for
communicating actual information. `api.json.output()` to the rescue!

```python
DEPS = ['step', 'path', 'step_history', 'json']

def RunSteps(api):
  step_result = api.step(
      'run tests',
      [api.path['build'].join('do_test_things.sh'), api.json.output()])
  num_passed = step_result.json.output['num_passed']
  if num_passed > 500:
    api.step('victory', [api.path['build'].join('do_a_dance.sh')])
  elif num_passed > 200:
    api.step('not defeated', [api.path['build'].join('woohoo.sh')])
  else:
    api.step('deads!', [api.path['build'].join('you_r_deads.sh')])

def GenTests(api):
  yield (api.test('winning') +
         api.step_data('run tests', api.json.output({'num_passed': 791}))
  yield (api.test('not_dead_yet') +
         api.step_data('run tests', api.json.output({'num_passed': 302}))
  yield (api.test('noooooo') +
         api.step_data('run tests', api.json.output({'num_passed': 10})))
```

### How does THAT work!?

`api.json.output()` returns a `recipe_api.Placeholder` which is meant to be
added into a step command list. When the step runs, the placeholder gets
rendered into some strings (in this case, like '/tmp/some392ra8'). When the step
finishes, the Placeholder adds data to the `StepData` object for the step which
just ran, namespaced by the module name (in this case, the 'json' module decided
to add an 'output' attribute to the `step_history` item). I'd encourage you to
take a peek at the implementation of the json module to see how this is
implemented.

### Example: write to standard input of a step

```python
api.step(..., stdin=api.raw_io.input('test input'))
```

Also see [raw_io's
example](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipe_modules/raw_io/examples/full.py).

### Example: read standard output of a step as json

```python
step_result = api.step(..., stdout=api.json.output())
data = step_result.stdout
# data is a parsed JSON value, such as dict
```

Also see [json's
example](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipe_modules/json/examples/full.py).

### Example: write to standard input of a step as json

```python
data = {'value': 1}
api.step(..., stdin=api.json.input(data))
```

Also see [json's
example](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipe_modules/json/examples/full.py).

### Example: simulated step output

This example specifies the standard output that should be returned when
a step is executed in simulation mode. This is typically used for
specifying default test data in the recipe or recipe module and removes
the need to specify too much test data for each test in GenTests:

```python
api.step(..., step_test_data=api.raw_io.output('test data'))
```

### Example: simulated step output for a test case

```python
yield (
    api.test('my_test') +
    api.step_data(
        'step_name',
        output=api.raw_io.output('test data')))
```

## How to change step presentation?

`step_result.presentation` allows modifying the appearance of a step:

### Logging

```python
step_result.presentation.logs['mylog'] = ['line1', 'line2']
```

Creates an extra log "mylog" under the step.

### Setting properties

`api.properties` are immutable, but you can change and add new
properties at the buildbot level.

```python
step_result.presentation.properties['newprop'] = 1
```

### Example: step text

This modifies the text displayed next to a step name:

```python
step_result = api.step(...)
step_result.presentation.step_text = 'Dynamic step result text'
```

* `presentaton.logs` allows creating extra logs of a step run. Example:
  ```python
  step_result.presentation.logs['mylog'] = ['line1', 'line2']
  ```
* presentation.properties allows changing and adding new properties at the
  buildbot level. Example:
  ```python
  step_result.presentation.properties['newprop'] = 1
  ```

## How do I know what modules to use?

Use `scripts/slave/recipes.py doc`. It's super effective!

## How do I run those tests you were talking about?

Each repo has a recipes.py entry point under `recipes_path` from `recipes.cfg` .

Execute the following commands:
`./recipes.py test run`
`./recipes.py test train`

Specifically, for `tools/build` repo, the commands to execute are:
`scripts/slave/recipes.py test run`
`scripts/slave/recipes.py test train`

## Where's the docs on `*.py`?

Check the docstrings in `*.py`. `<trollface text="Problem?"/>`

In addition, most recipe modules have example recipes in the `examples`
subfolder which exercises most of the code in the module for example purposes.