This page describes in detail how the GPU bots are set up, which files affect their configuration, and how to both modify their behavior and add new bots.
Chromium‘s GPU bots, compared to the majority of the project’s test machines, are physical pieces of hardware. When end users run the Chrome browser, they are almost surely running it on a physical piece of hardware with a real graphics processor. There are some portions of the code base which simply can not be exercised by running the browser in a virtual machine, or on a software implementation of the underlying graphics libraries. The GPU bots were developed and deployed in order to cover these code paths, and avoid regressions that are otherwise inevitable in a project the size of the Chromium browser.
The GPU bots are utilized on the chromium.gpu and chromium.gpu.fyi waterfalls, and various tryservers, as described in Using the GPU Bots.
All of the physical hardware for the bots lives in the Swarming pool, and most of it in the Chrome-GPU Swarming pool. The waterfall bots are simply virtual machines which spawn Swarming tasks with the appropriate tags to get them to run on the desired GPU and operating system type. So, for example, the Win10 Release (NVIDIA) bot is actually a virtual machine which spawns all of its jobs with the Swarming parameters:
{ "gpu": "10de:1cb3-23.21.13.8816", "os": "Windows-10", "pool": "Chrome-GPU" }
Since the GPUs in the Swarming pool are mostly homogeneous, this is sufficient to target the pool of Windows 10-like NVIDIA machines. (There are a few Windows 7-like NVIDIA bots in the pool, which necessitates the OS specifier.)
Details about the bots can be found on chromium-swarm.appspot.com and by using src/tools/swarming_client/swarming.py, for example swarming.py bots. If you are authenticated with @google.com credentials you will be able to make queries of the bots and see, for example, which GPUs are available.
The waterfall bots run tests on a single GPU type in order to make it easier to see regressions or flakiness that affect only a certain type of GPU.
The tryservers like win_chromium_rel_ng which include GPU tests, on the other hand, run tests on more than one GPU type. As of this writing, the Windows tryservers ran tests on NVIDIA and AMD GPUs; the Mac tryservers ran tests on Intel and NVIDIA GPUs. The way these tryservers' tests are specified is simply by mirroring how one or more waterfall bots work. This is an inherent property of the chromium_trybot recipe, which was designed to eliminate differences in behavior between the tryservers and waterfall bots. Since the tryservers mirror waterfall bots, if the waterfall bot is working, the tryserver must almost inherently be working as well.
There are a few one-off GPU configurations on the waterfall where the tests are run locally on physical hardware, rather than via Swarming. A few examples are:
There are a couple of reasons to continue to support running tests on a specific machine: it might be too expensive to deploy the required multiple copies of said hardware, or the configuration might not be reliable enough to begin scaling it up.
Adding a new test step to the bots requires that the test run via an isolate. Isolates describe both the binary and data dependencies of an executable, and are the underpinning of how the Swarming system works. See the LUCI wiki for background on Isolates and Swarming.
template("test") template in src/testing/test.gni. See test("gl_tests") in src/gpu/BUILD.gn for an example. For a more complex example which invokes a series of scripts which finally launches the browser, see src/chrome/telemetry_gpu_test.isolate.src/testing/buildbot/gn_isolate_map.pyl that refers to your target. Find a similar target to yours in order to determine the type. The type is referenced in src/tools/mb/mb_config.pyl.At this point you can build and upload your isolate to the isolate server.
See Isolated Testing for SWEs for the most up-to-date instructions. These instructions are a copy which show how to run an isolate that's been uploaded to the isolate server on your local machine rather than on Swarming.
If cd'd into src/:
./tools/mb/mb.py isolate //out/Release [target name]./tools/mb/mb.py isolate //out/Release angle_end2end_testspython tools/swarming_client/isolate.py batcharchive -I https://isolateserver.appspot.com out/Release/[target name].isolated.gen.jsonpython tools/swarming_client/isolate.py batcharchive -I https://isolateserver.appspot.com out/Release/angle_end2end_tests.isolated.gen.jsonpython tools/swarming_client/run_isolated.py -I https://isolateserver.appspot.com -s [HASH] -- [any additional args for the isolate]See the section below on isolate server credentials.
See Adding new steps to the GPU bots for details on this process.
In the tools/build workspace:
scripts/slave/recipe_modules/chromium_tests/:linux-rel, mac-rel, and win7-rel, which run against every Chromium CL, and which mirror the behavior of bots on the chromium.gpu waterfall.linux_optional_gpu_tests_rel, mac_optional_gpu_tests_rel and win_optional_gpu_tests_rel, which are triggered manually and run some tests which can't be run on the regular Chromium try servers mainly due to lack of hardware capacity.In the chromium/src workspace:
src/tools/mb/mb_config.pylsrc/testing/buildbot/generate_buildbot_json.pychromium.gpu.json and chromium.gpu.fyi.json. It defines on which GPUs various tests run.In the infradata/config workspace (Google internal only, sorry):
Chrome-GPU Swarming pool which contains most of the specialized hardware: as of this writing, the Windows and Linux NVIDIA bots, the Windows AMD bots, and the MacBook Pros with NVIDIA and AMD GPUs. New GPU hardware should be added to this pool.This section describes various common scenarios that might arise when maintaining the GPU bots, and how they'd be addressed.
This is described in Adding new tests to the GPU bots.
The tests use virtual machines to build binaries and to trigger tests on physical hardware. VMs don't run any tests themselves. Nevertheless the OS of the VM must match the OS of the physical hardware. Android uses Linux VMs for the hosts.
If you need a Mac VM:
gpu.star and regenerate the auto-generated files.If you need a non-Mac VM, VMs are allocated using the GCE Provider APIs:
Create a CL in the infradata/config (Google internal) workspace which does the following. Git configure your user.email to @google.com if necessary. For reference, see these example CLs:
Edit gpu.star to add an entry for the new bot. Currently, the only way to limit the number of concurrent builds per bot is to limit the number of VMs associated with it. This means that each new bot requires a new prefix. Add your new entry to the correct block:
gpu_ci_bots. For example: gce_thin_trusty('linux-fyi-skiarenderer-vulkan-nvidia', 'us-east1-c') or gce_thin_win10('win10-fyi-release-amd-rx-550').gpu_try_bots block (optional GPU trybots, ANGLE trybots, etc.). For example: gce_trusty_pair('gpu-fyi-try-linux-intel-exp').Run main.star to regenerate configs/chromium-swarm/bots.cfg and ‘configs/gce-provider/vms.cfg’. Double-check your work there.
Note that previously vms.cfg had to be editted manually. Part of the difficulty was in choosing a zone. This should soon no longer be necessary per crbug.com/942301, but consult with the Chrome Infra team to find out which of the zones has available capacity.
Get this reviewed and landed. This step associates the VM or pool of VMs with the bot's name on the waterfall.
When deploying a new GPU configuration, it should be added to the chromium.gpu.fyi waterfall first. The chromium.gpu waterfall should be reserved for those GPUs which are tested on the commit queue. (Some of the bots violate this rule – namely, the Debug bots – though we should strive to eliminate these differences.) Once the new configuration is ready to be fully deployed on tryservers, bots can be added to the chromium.gpu waterfall, and the tryservers changed to mirror them.
In order to add Release and Debug waterfall bots for a new configuration, experience has shown that at least 4 physical machines are needed in the swarming pool. The reason is that the tests all run in parallel on the Swarming cluster, so the load induced on the swarming bots is higher than it would be if the tests were run strictly serially.
With these prerequisites, these are the steps to add a new (swarmed) tester bot. (Actually, pair of bots -- Release and Debug. If deploying just one or the other, ignore the other configuration.) These instructions assume that you are reusing one of the existing builders, like GPU FYI Win Builder.
Work with the Chrome Infrastructure Labs team to get the (minimum 4) physical machines added to the Swarming pool. Use chromium-swarm.appspot.com or src/tools/swarming_client/swarming.py bots to determine the PCI IDs of the GPUs in the bots. (These instructions will need to be updated for Android bots which don't have PCI buses.)
Make sure to add these new machines to the Chrome-GPU Swarming pool by creating a CL against gpu.star in the infradata/config (Google internal) workspace. Git configure your user.email to @google.com if necessary. Here is one example CL and a second example.
Run main.star to regenerate configs/chromium-swarm/bots.cfg. Double-check your work there.
Allocate new virtual machines for the bots as described in How to set up new virtual machine instances.
Create a CL in the Chromium workspace which does the following. Here's an example CL.
win to Windows-2008ServerR2-SP1 (the Win7-like flavor running in our data center). Similarly, the Win8 bots had to have a very precise OS description (Windows-2012ServerR2-SP0).src/testing/buildbot/test_suite_exceptions.pyl for references to the other bot‘s name and see if your new bot needs to be added to any exclusion lists. For example, some of the tests don’t run on certain Win bots because of missing OpenGL extensions.src/testing/buildbot/chromium.gpu.fyi.json.cr-buildbucket.cfg:luci-scheduler.cfg:luci-milo.cfg:chromium.gpu.fyi console. Look at the short names and categories and try to come up with a reasonable organization.src/tools/mb/mb_config.pyl.After the Chromium-side CL lands it will take some time for all of the configuration changes to be picked up by the system. The bot will probably be in a red or purple state, claiming that it can't find its configuration. (It might also be in an “empty” state, not running any jobs at all.)
After the Chromium-side CL lands and the bot is on the console, create a CL in the tools/build workspace which does the following. Here's an example CL.
chromium_gpu_fyi.py in scripts/slave/recipe_modules/chromium_tests/. Make sure to set the serialize_tests property to True. This is specified for waterfall bots, but not trybots, and helps avoid overloading the physical hardware. Double-check the BUILD_CONFIG and parent_buildername properties for each. They must match the Release/Debug flavor of the builder, like GPU FYI Win Builder vs. GPU FYI Win Builder (dbg).scripts/slave/recipes.py --use-bootstrap test train). This doesn‘t appear to be necessary any more, but it’s something to watch out for if your CL fails presubmit for some reason.Note that it is crucial that the bot be deployed before hooking it up in the tools/build workspace. In the new LUCI world, if the parent builder can‘t find its child testers to trigger, that’s a hard error on the parent. This will cause the builders to fail. You can and should prepare the tools/build CL in advance, but make sure it doesn‘t land until the bot’s on the console.
Let's say that you want to cause the win_chromium_rel_ng try bot to run tests on CoolNewGPUType in addition to the types it currently runs (as of this writing, NVIDIA and AMD). To do this:
tests/masters_recipes_test.py for these new testers since they aren't yet covered by try bots and are going on a non-FYI waterfall. Make sure these run green for a day or two before proceeding.win_chromium_rel_ng's bot_ids list in scripts/slave/recipe_modules/chromium_tests/trybots.py. Rerun scripts/slave/recipes.py --use-bootstrap test train.There are a lot of one-off GPU types on the chromium.gpu.fyi waterfall and sometimes a failure happens just on one type. It's helpful to just be able to send a tryjob to a particular machine. Doing so requires a specific trybot to be set up because most if not all of the existing trybots trigger tests on more than one type of GPU.
Here are the steps to set up a new trybot which runs tests just on one particular GPU type. Let's consider that we are adding a manually-triggered trybot for the Win7 NVIDIA GPUs in Release mode. We will call the new bot gpu_manual_try_win7_nvidia_rel.
Allocate new virtual machines for the bots as described in How to set up new virtual machine instances, following the “trybot” instructions.
Create a CL in the Chromium workspace which does the following. Here's an example CL.
cr-buildbucket.cfg:luci.chromium.try bucket. This is a one-liner, with “name” being “gpu_manual_try_win7_nvidia_rel” and “mixins” being the OS-appropriate mixin, in this case “win-optional-gpu-try”. (We're repurposing the existing ACLs for the “optional” GPU trybots for these manually-triggered ones.)luci-milo.cfg:luci.chromium.try and tryserver.chromium.win consoles.src/tools/mb/mb_config.pyl. Reuse the same mixin as for the optional GPU trybot; in this case, gpu_fyi_tests_release_trybot_x86.Create a CL in the tools/build workspace which does the following. Here's an example CL.
scripts/slave/recipe_modules/chromium_tests/trybots.py. Create this section after the “Optional GPU bots” section for the appropriate tryserver (tryserver.chromium.win, tryserver.chromium.mac, tryserver.chromium.linux, tryserver.chromium.android). Have the bot mirror the appropriate waterfall bot; in this case, the buildername to mirror is GPU FYI Win Builder and the tester is Win7 FYI Release (NVIDIA).tests/masters_recipes_test.py, under FAKE_BUILDERS, under the appropriate tryserver waterfall (in this case, master.tryserver.chromium.win). This is because this is a LUCI-only bot, and this test verifies the old buildbot configurations.scripts/slave/recipes.py --use-bootstrap test train). This doesn‘t appear to be necessary any more, but it’s something to watch out for if your CL fails presubmit for some reason.At this point the new trybot should automatically show up in the “Choose tryjobs” pop-up in the Gerrit UI, under the luci.chromium.try heading, because it was deployed via LUCI. It should be possible to send a CL to it.
(It should not be necessary to modify buildbucket.config as is mentioned at the bottom of the “Choose tryjobs” pop-up. Contact the chrome-infra team if this doesn't work as expected.)
Several projects (ANGLE, Dawn) run custom tests using the Chromium recipes. They use try bot bot configs that run subsets of Chromium or additional slower tests that can't be run on the main CQ.
These try bots are a little different because they mirror waterfall bots that don‘t actually exist. The waterfall bots’ specifications exist only to tell these try bots which tests to run.
Let's say that you intended to add a new such custom try bot on Windows. Call it win-myproject-rel for example. You will need to add a “fake” mirror bot for each GPU family the tests you will need to run. For a GPU type of “CoolNewGPUType” in this example you could add a “fake” bot named “MyProject GPU Win10 Release (CoolNewGPUType)”.
src/testing/buildbot/generate_buildbot_json.py to regenerate the JSON files.cr-buildbucket.cfg to add win-myproject-rel.luci-milo.cfg to include win-myproject-rel.luci-scheduler.cfg to include “MyProject GPU Win10 Release (CoolNewGPUType)”.src/tools/mb/mb_config.pyl to include win-myproject-rel.src/testing/buildbot/generate_buildbot_json.py in the list of get_bots_that_do_not_actually_exist section.tools/build workspace which does the following. Here's an example CL.chromium_gpu_fyi.py in scripts/slave/recipe_modules/chromium_tests/. You can copy a similar step.win-myproject-rel to trybots.py in the same folder. This is where you associate “MyProject GPU Win10 Release (CoolNewGPUType)” with win-myproject-rel. See the sample CL for an example.win-myproject-rel on CLs using Choose Trybots in Gerrit.Let‘s say that you want to roll out an update to the graphics drivers on one of the configurations like the Win10 NVIDIA bots. In order to verify that the new driver won’t destabilize Chromium‘s commit queue, it’s necessary to run the new driver on one of the waterfalls for a day or two to make sure the tests are reliably green before rolling out the driver update. To do this:
Make sure that all of the current Swarming jobs for this OS and GPU configuration are targeted at the “stable” version of the driver in waterfalls.pyl and mixins.pyl. Make sure that there is a “named” stable version of the driver there, which targets the _TARGETED_DRIVER_VERSIONS dictionary in bot_config.py (Google internal).
File a Build Infrastructure bug, component Infra>Labs, to have ~4 of the physical machines already in the Swarming pool upgraded to the new version of the driver.
If an “experimental” version of this bot doesn't yet exist, follow the instructions above for How to add a new tester bot to the chromium.gpu.fyi waterfall to deploy one.
Have this experimental bot target the new version of the driver in waterfalls.pyl and mixins.pyl.
Hopefully, the new machine will pass the pixel tests. If it doesn‘t, then unfortunately, it’ll be necessary to follow the instructions on updating the pixel tests to temporarily suppress the failures on this particular configuration. Keep the time window for these test suppressions as narrow as possible.
Watch the new machine for a day or two to make sure it's stable.
When it is, update bot_config.py (Google internal) to add a mapping between the new driver version and the “stable” version. For example:
_TARGETED_DRIVER_VERSIONS = {
# NVIDIA Quadro P400, Ubuntu Stable version
'10de:1cb3-384.90': 'nvidia-quadro-p400-ubuntu-stable',
# NVIDIA Quadro P400, new Ubuntu Stable version
'10de:1cb3-410.78': 'nvidia-quadro-p400-ubuntu-stable',
# ...
}
The new driver version should match the one just added for the experimental bot. Get this CL reviewed and landed.
After it lands, ask the Chrome Infrastructure Labs team to roll out the driver update across all of the similarly configured bots in the swarming pool.
If necessary, update pixel test expectations and remove the suppressions added above.
Remove the old driver version from bot_config.py, leaving the “stable” driver version pointing at the newly upgraded version.
Note that we leave the experimental bot in place. We could reclaim it, but it seems worthwhile to continuously test the “next” version of graphics drivers as well as the current stable ones.
Working with the GPU bots requires credentials to various services: the isolate server, the swarming server, and cloud storage.
To upload and download isolates you must first authenticate to the isolate server. From a Chromium checkout, run:
./src/tools/swarming_client/auth.py login --service=https://isolateserver.appspot.comThis will open a web browser to complete the authentication flow. A @google.com email address is required in order to properly authenticate.
To test your authentication, find a hash for a recent isolate. Consult the instructions on Running Binaries from the Bots Locally to find a random hash from a target like gl_tests. Then run the following:
If authentication succeeded, this will silently download a file called delete_me into the current working directory. If it failed, the script will report multiple authentication errors. In this case, use the following command to log out and then try again:
./src/tools/swarming_client/auth.py logout --service=https://isolateserver.appspot.comThe swarming server uses the same auth.py script as the isolate server. You will need to authenticate if you want to manually download the results of previous swarming jobs, trigger your own jobs, or run swarming.py reproduce to re-run a remote job on your local workstation. Follow the instructions above, replacing the service with https://chromium-swarm.appspot.com.
Authentication to Google Cloud Storage is needed for a couple of reasons: uploading pixel test results to the cloud, and potentially uploading and downloading builds as well, at least in Debug mode. Use the copy of gsutil in depot_tools/third_party/gsutil/gsutil, and follow the Google Cloud Storage instructions to authenticate. You must use your @google.com email address and be a member of the Chrome GPU team in order to receive read-write access to the appropriate cloud storage buckets. Roughly:
gsutil configAt this point you should be able to write to the cloud storage bucket.
Navigate to https://console.developers.google.com/storage/chromium-gpu-archive to view the contents of the cloud storage bucket.