This page describes various extra details of the Skia Gold service that the GPU pixel tests use. For information on running the tests locally, see this section. For common information on triaging, modification, or general pixel wrangling, see GPU Pixel Wrangling or these sections (1, 2) of the general GPU testing documentation.
Gold is an image diff service developed by the Skia team. It was originally developed solely for Skia's usage and only supported post-submit tests, but has been picked up by other projects such as Chromium and PDFium and now supports trybots. Unlike other image diff solutions in Chromium, comparisons are done in an external service instead of locally on the testing machine.
Gold has three main advantages over the traditional local image comparison historically used by Chromium:
Gold consists of two main parts: the Gold instance/service and the
goldctl binary. A Gold instance in turn consists of two parts: a Google Storage bucket that data is uploaded to and a server running on GCE that ingests the data and provides a way to triage diffs.
goldctl simply provides a standardized way of interacting with Gold - uploading data to the correct place, retrieving baselines/golden information, etc.
In general, the following order of events occurs when running a Gold-enabled test:
goldctl, along with some information about the hardware and software configuration that the image was produced on, the test name, etc.
goldctlchecks whether the hash of the produced image is in the list of approved hashes.
goldctlexits with a non-failing return code and nothing else happens. At this point, the test is finished.
goldctluploads the image and metadata to the storage bucket and exits with a failing return code.
While this is the general order of events, there are several differences between waterfall/CI bots and trybots.
Waterfall bots are the simpler of the two bot types. There is only a single set of baselines to worry about, which is whatever baselines were approved for a git revision. Additionally, any new images that are produced on waterfalls are all lumped into the same group of “untriaged images on master”, and any images that are approved from here will immediately be added to the set of baselines for master.
Since not all waterfall bots have a trybot counterpart that can be relied upon to catch newly produced images before a CL is committed, it is likely that a change that produces new goldens on the CQ will end up making some of the waterfall bots red for a bit, particularly those on chromium.gpu.fyi. They will remain red until the new images are triaged as positive or the tests stop producing the untriaged images. So, it is best to keep an eye out for a few hours after your CL is committed for any new images from the waterfall bots that need triaging.
Trybots are a little more complicated when it comes to retrieving and approving images. First, the set of baselines that are provided when requested by a test is the union of the master baselines for the current revision and any baselines that are unique to the CL. For example, if an image with the hash
abcd is in the master baselines for
FooTest and the CL being tested has also approved an image with the hash
FooTest, then the provided baselines will contain both
When an image associated with a CL is approved, the approval only applies to that CL until the CL is merged. Once this happens, any baselines produced by the CL are automatically merged into the master baselines for whatever git revision the CL was merged as. In the above example, if the CL was merged as commit
ffff, then both
abef would be approved images on master from
You can see all currently untriaged images that are currently being produced on ToT on the GPU Gold instance's main page and currently untriaged images for a CL by substituting the Gerrit CL number into
It's possible, particularly if a test is regularly producing multiple images, for an image to be untriaged but not show up on the front page of the Gold instance (for details, see this crbug comment). To see all such images, visit this link.
If for some reason you know that a test run produced a bad image, but do not have a direct link to the failed build (e.g. you found a bad image using the untriaged non-ToT link from above), you may want to find the failed Swarming task to help debug the issue. Gold currently provides a list of CLs that were under test when a particular image was produced, but does not provide a link to the build that produced it, so the following workaround can be used.
Assuming the failure is relatively recent (within the past week or so), you can use the flakiness dashboard to help find the failed run. To do so, substitute the test name into
https://test-results.appspot.com/dashboards/flakiness_dashboard.html#showAllRuns=true&testType=pixel_skia_gold_test&tests=[test_name] and scroll through the history until you find the failed build (represented by a red square). Click on the build and follow the
Build log link. This will take you to the failed build, from which you can get to the Swarming task like normal by scrolling to the failed step and clicking on the link for the failed shard number.
If for some reason an image is not showing up in Gold but you know the hash, you can manually navigate to the page for it by filling in the correct information to
https://chrome-gpu-gold.skia.org/detail?test=[test_name]&digest=[hash]. From there, you should be able to triage it as normal.
If this happens, please also file a bug in Skia's bug tracker so that the root cause can be investigated and fixed. It's likely that you will be unable to directly edit the owner, CC list, etc. directly, in which case ping kjlubick@ with a link to the filed bug to help speed up triaging. Include as much detail as possible, such as a links to the failed swarming task and the triage link for the problematic image.
Although uncommon, changes to the Gold service and
goldctl binary may be needed. To do so, simply get a checkout of the Skia infrastructure repo and go through the same steps as a Chromium CL (
git cl upload, etc.).
The Gold service code is located in the
//golden/ directory, while
goldctl is located in
//gold-client/. Once your change is merged, you will have to either contact email@example.com to roll the service version or follow the steps in Rolling goldctl to roll the
goldctl version used by Chromium.
goldctl is available as a CIPD package and is DEPSed in as part of
gclient sync To update the binary used in Chromium, perform the following steps:
infra $ ./go/env.pyand run each of the commands it outputs to change your GOPATH and other environment variables for your terminal
deps.yamlto match the revision of your
goldctlCL (or some revision after it)
infra $ ./go/deps.py updateand copy the list of repos it updated (include this in your CL description)
If you want to make sure that
goldctl builds after the update before committing (e.g. to ensure that no extra third party dependencies were added), run the following after the
./go/deps.py update step:
infra $ ./go/deps.py install
infra $ go install go.skia.org/infra/gold-client/cmd/goldctl
infra $ which goldctlwhich should point to a binary in
infra $ goldctlto make sure it actually runs