This section covers common tasks for chromium developers. If you want a more full background on the chromium recipes, go to the Background section.
We're going to walkthrough an example of wanting to add a builder to the chromium.linux master.
chromium_tests recipe module: chromium_tests/chromium_linux.py .src/testing/buildbot/chromium.linux.json .builders.pyl , then you need to declare the new builder in master.cfg , assign it a recipe name and a slave pool (slaves.cfg).builders.pyl, then look at the docs and modify the corresponding file.luci-milo.cfg and luci-milo-dev.cfg in the infra/config branch of chromium/src . Existing entries in those configurations should be good examples. Your new entry should minimally include:name: a string matching either buildbot/$MASTER_NAME/$BUILDER_NAME (for buildbot bots) or buildbucket/$BUCKET_NAME/$BUILDER_NAME (for LUCI bots).
category: a string containing one or more categories separated by |.
Note: the order of builders within a console‘s configuration determines the order in which the builders appear on that console, and adjacent builders that share one or more categories will have those categories merged in the UI. As such, it’s best to list your new builder next to other builders that share its categories.
short_name: a string containing at most three characters that will be used to represent the builder on the console.
You also need to think about provisioning hardware for the new slaves. File a ticket with the label Infra-Labs (http://go/infrasys-bug is a good place to start) with information about the hardware you need, and someone on the labs team will take a look at it, and help you get the hardware you need.
Generally, you‘re going to want to add a new builder to our FYI waterfall first, before adding it to the main waterfall. This is so that we can monitor it, and make sure it’s consistently and reliably green, before making it a tree closing bot. So, before you add the bot to chromium.linux, add it to the chromium.fyi master. So, follow the instructions above, but instead of adding the builder to the chromium.linux master, add it to the chromium.fyi master.
Once everything is tested and the builder works, you must get chrome eng review to sign off on the addition to the main waterfall. Once you have that approval move the waterfall builder config to the appropriate main waterfall master (e.g. chromium.linux)
chromium_linux.py )master.chromium.linux/master_cfg.py ).Every new main waterfall bot must have corresponding trybot coverage. In addition, you can request a new trybot if you have a configuration you are interested in testing.
chromium recipe trybots currently are configured by specifying the main waterfall configuration they are supposed to emulate. This information is stored in the chromium_tests/trybots.py file. So, for example, the linux_android_rel_ng trybot is configured as follows:
...
'linux_android_rel_ng': {
'mastername': 'chromium.linux',
'buildername': 'Android Builder',
'tester': 'Android Tests',
},
...
For example, suppose we want to add a builder named new_fancy_android_device to tryserver.chromium.android, which is a copy of the builder named fancy_android_device on the main chromium.android waterfall. We would do the following
Add the builder configuration of
...
'new_fancy_android_device': {
'mastername': 'chromium.android',
'buildername': 'fancy_android_device',
},
...
Add the new trybot to the corresponding tryserver master. In this case, that's tryserver.chromium.android .
Request a master restart here .
You can then schedule tryjobs on the bot using git cl try -b new_fancy_android_device.
Coming soon...
For now, email chrome-infra@
First, please consider why you‘re doing this. If it’s needed for a test, then let the test declare needed its compile targets (this is part of the Test interface from steps.py , and when you add this test to a builder the right targets will automatically compile.
The technical way would be to modify the compile_targets key in your bot config. For example, in scripts/slave/recipe_modules/chromium_tests/chromium_linux.py , to add the compile target your_target_here:
'Linux Builder': {
...
'compile_targets': [
...
'your_target_here',
],
...
},
Add it to the JSON test spec. Example (src/testing/buildbot/chromium.linux.json):
"Linux Builder": {
"gtest_tests": [
{
"test": "base_unittests"
},
...
]
}
Changes to the JSON file can be tested on the trybots, and you can verify that they take effect and build still passes.
Generally, look at steps.py , and either use an existing class there, or add a new one.
Chromium tests are currently run on buildbot, and are organized by builders. A builder, in buildbot’s terminology, is a sequence of steps (typically, individual commands) executed by a slave, and controlled by the central master process.
Since such tightly controlled system is hard to maintain at scale, and requires human attention and frequent downtimes for master restarts on every update, Chrome Infra introduced recipes.
A recipe is a single command executed as a single buildbot builder step (called steps), which dynamically generates all the other steps. This moves the control over steps from the master to the slaves, allows for dynamic creation of steps at runtime, and, most importantly, eliminates the need for master restarts when a builder configuration changes.
Recipe-based builders have a very generic configuration on the buildbot master, and all the other specific configs live in recipes and recipe modules (more on that later).
Additional requirement is to keep continuous integration (aka waterfall) builders in sync with the tryserver (commit queue) builders.
To give recipes some context, let’s consider what a typical Chromium CL goes through, from its inception to landing into the repository.
On a typical day, Chromium CQ may check up to 400 CLs, and land over 300 of them. At peak times, CQ may land as many as 30 CLs every hour.
In order for the system to function efficiently and reliably, several important constraints are enforced.
Each of the requirements above needs a fairly complex and highly tuned system to perform each step of the verification. Therefore, Chrome Infra provides a common library of recipes implementing all of these requirements, and expects developers to use it with minimum configuration on their part.
Currently, the following components are involved in configuring a builder:
c[‘builders’] = <list of builder specs>
Each waterfall builder is specified using a dict like this (example from master.chromium.mac/master_mac_cfg.py ):
{ 'name': 'mac-rel' 'factory': m_annotator.BaseFactory( 'chromium', # name of the recipe factory_properties=None, # optional factory properties triggers=[<list of triggered builders>]), 'notify_on_missing': True, 'category': '3mac', }
Note the name of the recipe: chromium. Together with the name of the master and builder, this fully determines the builder configuration in the master. All the other details (specific steps) are configured in the recipe and recipe modules.
Similarly, a tryserver builder is specified using chromium_trybot recipe (from master.tryserver.chromium.mac/master.cfg ):
{ 'name': 'mac-rel', 'factory': m_annotator.BaseFactory('chromium_trybot'), # Share build directory [...] to save space. 'slavebuilddir': 'mac' }
Again, the recipe, master and builder names fully determine the configuration on the master side. Changing how the builder is defined can now be done without restarting the master.
If the master is configured using builders.pyl, then the above description does not apply. Please see this document instead for a description of how builders.pyl works.
Path: build/scripts/slave/recipes/chromium.py
This is a very short “glue” recipe which reads the detailed configurations, prepares the checkout, compiles the targets, and runs the tests. All the specifics are done in a shared recipe module chromium_tests.
Path: build/scripts/slave/recipes/chromium_trybot.py
Each trybot (builder) is defined in terms of the corresponding main waterfall builders. This config file is in build/scripts/slave/recipe_modules/chromium_tests/trybots.py
'mac_chromium_rel_ng': {
'mastername': 'chromium.mac',
'buildername': 'Mac Builder',
'tester': 'Mac10.8 Tests',
}
The recipe takes the corresponding compile and test configs, and adds the tryserver specific logic, such as applying a patch from a CL, retrying compile and failed tests without the patch, and failing / succeeding the build appropriately.
In particular, if compile fails both with and without a patch, the entire job fails. However, if a small portion of tests fails in the same way with and without a patch, the job succeeds (the failures are assumed not because of the CL, and are tolerable to continue the development).
This implements the requirement that try bots (CQ) are always in sync with the waterfall builders. This recipe also uses the best proven retry strategies, thus keeping CQ jobs robust and accurate.
Path: build/scripts/slave/recipe_modules/chromium_tests/api.py
Implements methods like configure_build, prepare_checkout, compile, etc. that take configuration parameters and actually run the steps.
Example config: src/testing/buildbot/chromium.mac.json
These configs live in the project repo, and define additional compile targets and specific tests to run on each builder.