infra/experimental/doc/luci-swarmbucket-cq.md - infra/infra.git - Git at Google

 # Detailed Description of LUCI Swarmbucket CQ Flow

 ### Gerrit -> CQ -> Buildbucket

 What to do with a CL is defined by a `cq.cfg` ([example](https://chromium.googlesource.com/infra/infra/+/master/config/cq.cfg)) file. In particular, it says what builders to trigger in what buckets. Each build is parameterized with a JSON blob. `buildset` is a string of the form `patch/gerrit/chromium-review.googlesource.com/<CL id>/<Patch id>`. We can use this form to search for builds from a CL using a query like [this](https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.search?bucket=luci.infra.try&tag=buildset%253Apatch%252Fgerrit%252Fchromium-review.googlesource.com%252F375219%252F1&_h=3&).

 Here are buildbucket builds that correspond to the CL in the results from the query above. Buildbucket field `parameters_json` contains a string that looks like this:
 ```json
 {
   "builder_name": "Infra Win Tester",
   "changes": [
     {
       "author": {
         "email": "martiniss@chromium.org"
       },
       "repo_url": "https:\/\/chromium.googlesource.com\/infra\/infra"
     }
   ],
   "properties": {
     "attempt_start_ts": 1.472083514e+15,
     "category": "cq",
     "event.change.id": "infra%2Finfra~master~I2582d437489ec9688788cbaea026d8eba7e790c3",
     "event.change.number": 375219,
     "event.change.url": "https:\/\/chromium-review.googlesource.com\/#\/c\/375219",
     "event.patchSet.ref": "refs\/changes\/19\/375219\/1",
     "gerrit": "https:\/\/chromium-review.googlesource.com",
     "master": "luci.infra.try",
     "patch_project": "infra\/infra",
     "patch_storage": "gerrit",
     "reason": "CQ",
     "revision": "HEAD"
   }
 }
 ```

 `properties` are generated in `get_try_job_properties` [here](https://chrome-internal.googlesource.com/infra/infra_internal/+/master/commit_queue/pending_manager/gerrit.py) and they all end up available from recipes, as you'll see below.

 ### Buildbucket -> Swarming

 At this point we have a Buildbucket build posted to the buildbucket that has a JSON blob of particular form as its `parameters_json` field.

 Configuration for buckets are defined in `cr-buildbucket.cfg`.
 `cq.cfg` and `cr-buildbucket.cfg` are in different locations because bucket config is same for entire repo/project, while there could be several CQs for individual branches in the repo. As an example:

 ```python
 buckets {
   name: "luci.infra.try"

   swarming {
     hostname: "chromium-swarm.appspot.com"
     url_format: "https://luci-milo.appspot.com/swarming/task/{task_id}"
     builder_defaults {
       swarming_tags: "allow_milo:1"
       dimensions: "cpu:x86-64"
       dimensions: "pool:Chrome"
       recipe {
         repository: "https://chromium.googlesource.com/infra/infra"
       }
       execution_timeout_secs: 1800  # 30 min
     }

     ...

     builders {
       category: "Infra"
       name: "Infra Win Tester"
       dimensions: "os:Windows-7-SP1"
       recipe {
         name: "infra_repo_trybot"
       }
     }

     ...
   }
 }
 ```

 [Here's](https://chromium.googlesource.com/infra/infra/+/master/appengine/cr-buildbucket/proto/project_config.proto#32) the corresponding `*.proto` file.

 Let's follow the build parameters. Looks like `parameters_json` is immediately deserialized into `parameters` field of the `Build` object. Then the service looks at the bucket config and `builder_name` parameter in `parameters` and figures out whether it needs to start a Swarming job (by searching for `Swarming.Build` messages in the bucket proto in `cr-buildbucket.cfg`).

 And we end up in `create_task_async` that looks scary. It fetches `Swarming` and `Builder` proto from corresponding `cr-buildbucket.cfg` config and calls `create_task_def_async` (that looks even scarier) to convert build `parameters` and all configs into Swarming task definition.

 The function merges `builders` proto with `builder_defaults` proto with `build.parameters['swarming']` (that is missing in our example) into something that looks like this:

 ```python
 builder {
   swarming_tags: "allow_milo:1"

   dimensions: "cpu:x86-64"
   dimensions: "pool:Chrome"
   dimensions: "os:Windows-7-SP1"

   recipe {
     name: "infra_repo_trybot"
     repository: "https://chromium.googlesource.com/infra/infra"
   }

   execution_timeout_secs: 1800
 }
 ```

 Then it sees `recipe` section and does some strange transformations to come up with something called `properties_json`. In particular, all `build.parameters['properties']` (generated by CQ at the very start) end up in `properties_json` as is. E.g. `properties_json` looks like:

 ```json
 {
   <stuff>
   "attempt_start_ts": 1.472083514e+15,
   "category": "cq",
   "event.change.id": "infra%2Finfra~master~I2582d437489ec9688788cbaea026d8eba7e790c3",
   "event.change.number": 375219,
   "event.change.url": "https:\/\/chromium-review.googlesource.com\/#\/c\/375219",
   "event.patchSet.ref": "refs\/changes\/19\/375219\/1",
   "gerrit": "https:\/\/chromium-review.googlesource.com",
   "master": "luci.infra.try",
   "patch_project": "infra\/infra",
   "patch_storage": "gerrit",
   "reason": "CQ",
   "revision": "HEAD"
 }
 ```

 Next Buildbucket fetches something called "task template" from global Buildbucket config. Here it is. It then renders this template using the data generated above. Relevant chunk of the template:

 ```json
 "command": [
   "kitchen${EXECUTABLE_SUFFIX}",
   "cook",
   "-python-path", "third_party",
   "-timestamps",
   "-repository", "$repository",
   "-revision", "$revision",
   "-recipe", "$recipe",
   "-properties", "$properties_json",
   "-logdog-host", "luci-logdog.appspot.com",
   "-logdog-project", "$project",
   "-logdog-tee",
   "-logdog-enable-annotee"
 ],
 ```

 Here
 `$repository` comes from `cr-buildbucket.cfg` ("https://chromium.googlesource.com/infra/infra" in our case)
 `$revision` comes from `build.properties['swarming']['recipe']['revision']` (empty in our case)
 `$recipe` comes from `cr-buildbucket.cfg` ("infra_repo_trybot" in our case).
 `$properties_json` is JSON blob I've described above (the one that contains data from CQ).
 `$project` is indirectly derived from `cr-buildbucket.cfg` (it's the name of the project that defined the bucket, "infra" in our case).
 So we end up with a Swarming task that invokes "kitchen" tool, passing it a JSON blob with properties. Here is the example: https://chromium-swarm.appspot.com/user/task/30d7d5a2288f9710

 ### Swarming -> Recipe engine (through kitchen)

 Kitchen is a tool that bootstraps recipe engine and its environment.
 It's source code is [here](https://chromium.googlesource.com/infra/infra/+/master/go/src/infra/tools/kitchen/).
 Let's follow the build properties.

 Kitchen reads them from the command line and appends one more property "path_config": "swarmbucket".

 It then checks out `$repository` at `$revision`. In our case "https://chromium.googlesource.com/infra/infra" at FETCH_HEAD.

 Next it prepares a command to invoke recipe engine. Reads "infra/config/recipes.cfg" from the checked out repo (in our case this file), there it finds "recipes_path" field ("recipes" in our case). It uses it to construct a path to recipes.py file. This one.

 The final command is
 python ...../recipes.py run --properties-file <path to JSON> ... <recipe>

 (In our case "<recipe>" is "infra_repo_trybot").

 `--properties-file` points to a JSON file with serialized build properties, in our case it looks like this:

 ```json
 {
   "attempt_start_ts": 1.472083514e+15,
   "blamelist": [
     "martiniss@chromium.org"
   ],
   "buildername": "Infra Win Tester",
   "category": "cq",
   "event.change.id": "infra%2Finfra~master~I2582d437489ec9688788cbaea026d8eba7e790c3",
   "event.change.number": 375219,
   "event.change.url": "https:\/\/chromium-review.googlesource.com\/#\/c\/375219",
   "event.patchSet.ref": "refs\/changes\/19\/375219\/1",
   "gerrit": "https:\/\/chromium-review.googlesource.com",
   "master": "luci.infra.try",
   "patch_project": "infra\/infra",
   "patch_storage": "gerrit",
   "path_config": "swarmbucket",
   "reason": "CQ",
   "repository": "https:\/\/chromium.googlesource.com\/infra\/infra",
   "revision": "HEAD"
 }
 ```

 As a reminder, it came directly from CQ, with few fields like `buildername`, `blamelist` and `category` added by Swarmbucket code.

 ### Recipe engine -> recipes & recipe modules

 [recipes.py](https://chromium.googlesource.com/infra/infra/+/master/recipes/recipes.py) checks out recipe\_engine code and all dependent modules (based on [recipes.cfg](https://chromium.googlesource.com/infra/infra/+/master/infra/config/recipes.cfg)) and then it passes control to the actual recipe.

 The properties are available via `api.properties` dictionary. (or `self.m.properties` if used from inside some recipe module). Documentation on properties in recipes is [here](https://github.com/luci/recipes-py/blob/master/doc/user_guide.md#properties-are-the-primary-input-for-your-recipes).

 For example `event.patchSet.ref` property (which is `refs/changes/19/375219/1` in our case, as supplied by CQ) is
 [transformed](https://cs.chromium.org/chromium/tools/depot_tools/recipe_modules/bot_update/__init__.py?q=event.patchSet.ref&l=22) into `gerrit_ref` parameter of `bot_update` recipe module.
 So `bot_update` recipe module ends up "knowing" about CL ref as `self._gerrit_ref` (and it then passes it to actual bot_update.py script).
	# Detailed Description of LUCI Swarmbucket CQ Flow

	### Gerrit -> CQ -> Buildbucket

	What to do with a CL is defined by a `cq.cfg` ([example](https://chromium.googlesource.com/infra/infra/+/master/config/cq.cfg)) file. In particular, it says what builders to trigger in what buckets. Each build is parameterized with a JSON blob. `buildset` is a string of the form `patch/gerrit/chromium-review.googlesource.com/<CL id>/<Patch id>`. We can use this form to search for builds from a CL using a query like [this](https://apis-explorer.appspot.com/apis-explorer/?base=https://cr-buildbucket.appspot.com/_ah/api#p/buildbucket/v1/buildbucket.search?bucket=luci.infra.try&tag=buildset%253Apatch%252Fgerrit%252Fchromium-review.googlesource.com%252F375219%252F1&_h=3&).

	Here are buildbucket builds that correspond to the CL in the results from the query above. Buildbucket field `parameters_json` contains a string that looks like this:
	```json
	{
	"builder_name": "Infra Win Tester",
	"changes": [
	{
	"author": {
	"email": "martiniss@chromium.org"
	},
	"repo_url": "https:\/\/chromium.googlesource.com\/infra\/infra"
	}
	],
	"properties": {
	"attempt_start_ts": 1.472083514e+15,
	"category": "cq",
	"event.change.id": "infra%2Finfra~master~I2582d437489ec9688788cbaea026d8eba7e790c3",
	"event.change.number": 375219,
	"event.change.url": "https:\/\/chromium-review.googlesource.com\/#\/c\/375219",
	"event.patchSet.ref": "refs\/changes\/19\/375219\/1",
	"gerrit": "https:\/\/chromium-review.googlesource.com",
	"master": "luci.infra.try",
	"patch_project": "infra\/infra",
	"patch_storage": "gerrit",
	"reason": "CQ",
	"revision": "HEAD"
	}
	}
	```

	`properties` are generated in `get_try_job_properties` [here](https://chrome-internal.googlesource.com/infra/infra_internal/+/master/commit_queue/pending_manager/gerrit.py) and they all end up available from recipes, as you'll see below.

	### Buildbucket -> Swarming

	At this point we have a Buildbucket build posted to the buildbucket that has a JSON blob of particular form as its `parameters_json` field.

	Configuration for buckets are defined in `cr-buildbucket.cfg`.
	`cq.cfg` and `cr-buildbucket.cfg` are in different locations because bucket config is same for entire repo/project, while there could be several CQs for individual branches in the repo. As an example:

	```python
	buckets {
	name: "luci.infra.try"

	swarming {
	hostname: "chromium-swarm.appspot.com"
	url_format: "https://luci-milo.appspot.com/swarming/task/{task_id}"
	builder_defaults {
	swarming_tags: "allow_milo:1"
	dimensions: "cpu:x86-64"
	dimensions: "pool:Chrome"
	recipe {
	repository: "https://chromium.googlesource.com/infra/infra"
	}
	execution_timeout_secs: 1800 # 30 min
	}

	...

	builders {
	category: "Infra"
	name: "Infra Win Tester"
	dimensions: "os:Windows-7-SP1"
	recipe {
	name: "infra_repo_trybot"
	}
	}

	...
	}
	}
	```

	[Here's](https://chromium.googlesource.com/infra/infra/+/master/appengine/cr-buildbucket/proto/project_config.proto#32) the corresponding `*.proto` file.

	Let's follow the build parameters. Looks like `parameters_json` is immediately deserialized into `parameters` field of the `Build` object. Then the service looks at the bucket config and `builder_name` parameter in `parameters` and figures out whether it needs to start a Swarming job (by searching for `Swarming.Build` messages in the bucket proto in `cr-buildbucket.cfg`).

	And we end up in `create_task_async` that looks scary. It fetches `Swarming` and `Builder` proto from corresponding `cr-buildbucket.cfg` config and calls `create_task_def_async` (that looks even scarier) to convert build `parameters` and all configs into Swarming task definition.

	The function merges `builders` proto with `builder_defaults` proto with `build.parameters['swarming']` (that is missing in our example) into something that looks like this:

	```python
	builder {
	swarming_tags: "allow_milo:1"

	dimensions: "cpu:x86-64"
	dimensions: "pool:Chrome"
	dimensions: "os:Windows-7-SP1"

	recipe {
	name: "infra_repo_trybot"
	repository: "https://chromium.googlesource.com/infra/infra"
	}

	execution_timeout_secs: 1800
	}
	```

	Then it sees `recipe` section and does some strange transformations to come up with something called `properties_json`. In particular, all `build.parameters['properties']` (generated by CQ at the very start) end up in `properties_json` as is. E.g. `properties_json` looks like:

	```json
	{
	<stuff>
	"attempt_start_ts": 1.472083514e+15,
	"category": "cq",
	"event.change.id": "infra%2Finfra~master~I2582d437489ec9688788cbaea026d8eba7e790c3",
	"event.change.number": 375219,
	"event.change.url": "https:\/\/chromium-review.googlesource.com\/#\/c\/375219",
	"event.patchSet.ref": "refs\/changes\/19\/375219\/1",
	"gerrit": "https:\/\/chromium-review.googlesource.com",
	"master": "luci.infra.try",
	"patch_project": "infra\/infra",
	"patch_storage": "gerrit",
	"reason": "CQ",
	"revision": "HEAD"
	}
	```

	Next Buildbucket fetches something called "task template" from global Buildbucket config. Here it is. It then renders this template using the data generated above. Relevant chunk of the template:

	```json
	"command": [
	"kitchen${EXECUTABLE_SUFFIX}",
	"cook",
	"-python-path", "third_party",
	"-timestamps",
	"-repository", "$repository",
	"-revision", "$revision",
	"-recipe", "$recipe",
	"-properties", "$properties_json",
	"-logdog-host", "luci-logdog.appspot.com",
	"-logdog-project", "$project",
	"-logdog-tee",
	"-logdog-enable-annotee"
	],
	```

	Here
	`$repository` comes from `cr-buildbucket.cfg` ("https://chromium.googlesource.com/infra/infra" in our case)
	`$revision` comes from `build.properties['swarming']['recipe']['revision']` (empty in our case)
	`$recipe` comes from `cr-buildbucket.cfg` ("infra_repo_trybot" in our case).
	`$properties_json` is JSON blob I've described above (the one that contains data from CQ).
	`$project` is indirectly derived from `cr-buildbucket.cfg` (it's the name of the project that defined the bucket, "infra" in our case).
	So we end up with a Swarming task that invokes "kitchen" tool, passing it a JSON blob with properties. Here is the example: https://chromium-swarm.appspot.com/user/task/30d7d5a2288f9710

	### Swarming -> Recipe engine (through kitchen)

	Kitchen is a tool that bootstraps recipe engine and its environment.
	It's source code is [here](https://chromium.googlesource.com/infra/infra/+/master/go/src/infra/tools/kitchen/).
	Let's follow the build properties.

	Kitchen reads them from the command line and appends one more property "path_config": "swarmbucket".

	It then checks out `$repository` at `$revision`. In our case "https://chromium.googlesource.com/infra/infra" at FETCH_HEAD.

	Next it prepares a command to invoke recipe engine. Reads "infra/config/recipes.cfg" from the checked out repo (in our case this file), there it finds "recipes_path" field ("recipes" in our case). It uses it to construct a path to recipes.py file. This one.

	The final command is
	python ...../recipes.py run --properties-file <path to JSON> ... <recipe>

	(In our case "<recipe>" is "infra_repo_trybot").

	`--properties-file` points to a JSON file with serialized build properties, in our case it looks like this:

	```json
	{
	"attempt_start_ts": 1.472083514e+15,
	"blamelist": [
	"martiniss@chromium.org"
	],
	"buildername": "Infra Win Tester",
	"category": "cq",
	"event.change.id": "infra%2Finfra~master~I2582d437489ec9688788cbaea026d8eba7e790c3",
	"event.change.number": 375219,
	"event.change.url": "https:\/\/chromium-review.googlesource.com\/#\/c\/375219",
	"event.patchSet.ref": "refs\/changes\/19\/375219\/1",
	"gerrit": "https:\/\/chromium-review.googlesource.com",
	"master": "luci.infra.try",
	"patch_project": "infra\/infra",
	"patch_storage": "gerrit",
	"path_config": "swarmbucket",
	"reason": "CQ",
	"repository": "https:\/\/chromium.googlesource.com\/infra\/infra",
	"revision": "HEAD"
	}
	```

	As a reminder, it came directly from CQ, with few fields like `buildername`, `blamelist` and `category` added by Swarmbucket code.

	### Recipe engine -> recipes & recipe modules

	[recipes.py](https://chromium.googlesource.com/infra/infra/+/master/recipes/recipes.py) checks out recipe\_engine code and all dependent modules (based on [recipes.cfg](https://chromium.googlesource.com/infra/infra/+/master/infra/config/recipes.cfg)) and then it passes control to the actual recipe.

	The properties are available via `api.properties` dictionary. (or `self.m.properties` if used from inside some recipe module). Documentation on properties in recipes is [here](https://github.com/luci/recipes-py/blob/master/doc/user_guide.md#properties-are-the-primary-input-for-your-recipes).

	For example `event.patchSet.ref` property (which is `refs/changes/19/375219/1` in our case, as supplied by CQ) is
	[transformed](https://cs.chromium.org/chromium/tools/depot_tools/recipe_modules/bot_update/__init__.py?q=event.patchSet.ref&l=22) into `gerrit_ref` parameter of `bot_update` recipe module.
	So `bot_update` recipe module ends up "knowing" about CL ref as `self._gerrit_ref` (and it then passes it to actual bot_update.py script).