As an ANGLE Sheriff. Your job is to:
If you're not an ANGLE team member, you can contact us on the public ANGLE project Google group.
Note: Please review and if needed update the wrangler schedule.
Note: If you need to suppress test failures (e.g. to keep an auto-roller unblocked), see Handling Test Failures.
Your first job is to keep the ANGLE builders green and unblocked.
There are two consoles for ANGLE continuous integration builders:
We recommend you track ANGLE build failures is via Sheriff-o-matic. Bookmark the link and check it regularly during your shift. Note: currently flaky failures show up as separate failure instances.
We expect these waterfalls to be as “green” as possible. As a wrangler please help clean out any failures by finding and reverting problematic CLs, suppressing flaky tests that can't be fixed, or finding other solutions. We aim to have zero failing builds, so follow the campsite rule and leave the waterfall cleaner than when you started your shift.
See more detailed instructions on ANGLE testing by following this link.
In addition to the CI builders, we have a console for try jobs on the ANGLE CV (change verifier):
Failures are intended on this waterfall as developers test WIP changes. You must act on any persistent flakiness or failure that causes developer drag by filing bugs, reverting CLs, or taking other action as with the CI waterfall.
If you find a failure that is unrelated to ANGLE, file a Chromium bug. Set the bug label
Hotlist-PixelWrangler. Ensure you cc the current ANGLE and Chrome GPU wranglers, which you can find by consulting build.chromium.org. For more information see Filing Chromium Bug Reports below.
You can optionally follow Chromium bugs in the
Internals>GPU>ANGLE component to be alerted to reports of ANGLE-related breakage in Chrome.
NOTE: When all builds seem to be purple or otherwise broken:
This could be a major infrastructure outage. File a high-priority bug using g.co/bugatrooper.
The GPU Pixel Wrangler is responsible for Chromium bugs. Please file Chromium issues with the Label
Hotlist-PixelWrangler for bugs outside of the ANGLE project.
IMPORTANT info to include in Chromium bug reports:
Internals>GPUfor general GPU bugs
Internals>GPU>Testingfor failures that look infrastructure-related
Internals>GPU>ANGLEfor ANGLE-related Chromium bugs
Internals>Skiafor Skia-specific bugs
parent_got_angle_revisionin both builds.
https://chromium.googlesource.com/angle/angle.git/+log/<last passing revision>..<first failing revision>
The ANGLE into Chrome auto-roller automatically updates Chrome with the latest ANGLE changes.
The ANGLE into Android auto-roller updates Android with the latest ANGLE changes. You must manually approve and land these rolls. The recommendation is to pre-approve the roll and set “Autosubmit”.
We also use additional auto-rollers to roll third party libraries into ANGLE and Chromium:
Roller health: You will be cc'ed on all rolls. Please check failed rolls to verify there is no blocking breakage.
For all rollers, you can trigger manual rolls using the dashboards to land high-priority changes. For example: Chromium-side test expectation updates or suppressions. When a roll fails, stop the roller, determine if the root cause is a problem with ANGLE or with the upstream repo, and file an issue with an appropriate next step.
To monitor the rollers during your shift, you can:
Filter. The dashboard provides the status of ANGLE related rollers. Monitor their modes and numbers.
from:(*-autoroll (Gerrit))”. This improves the signal to noise ratio of your inbox, so the important emails, ex) “
The roll is failing consistently. Time to investigate.”, can stand out.
Vulkan-deps consists of several related Vulkan dependencies: Vulkan-Tools, Vulkan-Loader, Vulkan-ValidationLayers, Vulkan-Headers and other related repos. One common source of breaks is a Vulkan Headers update, which can take a while to be integrated into other repos like the Vulkan Validation Layers. No action on your part is needed for header updates.
If a vulkan-deps AutoRoll CL triggers an failure in the
presubmit bot, in the “export targets” step, you can:
BUILD.gnif possible. See this example CL.
export_targets.py. See this example CL.
If the roll fails for a reason other than a header update or presubmit, determine the correct upstream repo and file an issue upstream. For simple compilation failures, we usually submit fixes ourselves. For more info on vulkan-deps see the README.
To handle failures on these bots:
Any large regressions should be triaged with a new ANGLE bug linked to any suspected CLs that may have caused performance to regress. If it's a known/expected regression, the bug can be closed as such. The tests are very flaky right now, so a WontFix resolution is often appropriate.