Flake portal is the entry point to Chromium flakes and related information.
Flake Portal UI: https://analysis.chromium.org/p/chromium/flake-portal
Table of contents:
The Flakes page ranks on-going flakes from CQ and CI in last 7 days by their negative impacts. Open bugs (either filed by Flake Portal or by developers/sheriffs/other automatic tools) are linked to each flake.
Flaky tests. Any test that failed nondeterministically is a flaky test. For types of flaky tests, see below.
Flake types. Within Flakes, flake types are defined by the negative impact of a flaky run.
The Flakes page groups similar flaky tests to avoid duplications, using the following criteria:
Detected flakes are ranked by a unified score by their negative impacts.
The score for CQ flakes is calculated based on impacted CLs and weights of each flake type. The score for CI flakes is calculated based on occurrences and weights of each flake type, since CL concept is not relevant. The weights are heuritically chosen numbers which should be proportional to the negative impact of each types.
Score = Sum(CQ flake type weight * impacted CLs) + Sum(CI flake type weight * occurrences)
Tag | Example |
---|---|
binary | binary==content_browsertests matches tests in steps with content_browsertests as isolate target. |
builder | builder==win7_chromium_rel_ng matches tests that occurred in the builder win7_chromium_rel_ng. |
component | component==Blink>Accessibility matches tests whose directory's OWNERS file has Blink>Accessibility as COMPONENT. Tests in sub-components are not included. |
directory | directory==base/ matches tests whose test files are in base/ directory. |
master | master==tryserver.chromium.android matches tests that occurred in the master tryserver.chromium.android. |
parent_component | parent_component==Blink>Accessibility matches tests whose component is Blink>Accessibility or a sub-component of Blink>Accessibility. |
source | source==base/hash_unittest.cc matches tests defined in the source file base/hash_unittest.cc. |
step | step==content_browsertests (with patch) |
suite | In Flake Portal, suite is the smallest group of tests that are defined in the same file or directory, with some special cases. |
suite==GCMConnectionHandlerImplTest matches gtest GCMConnectionHandlerImplTest.* | |
suite==FullscreenVideoTest matches Java tests .FullscreenVideoTest#test. | |
suite==third_party/blink/web_tests/fast/events matches Blink layout tests fast/events/* | |
suite==webgl_conformance_tests matches Telemetry-based gpu tests gpu_tests.webgl_conformance_integration_test.* | |
test_type | test_type==content_browsertests matches tests in steps with content_browsertests as their step name if removing suffixes like “(with patch)”. |
watchlist | watchlist==accessibility matches tests whose source files match the accessibility watchlist in src/WATCHLISTS. |
When a test step fails on a CI waterfall build, Findit runs a deflake swarming task to differentiate consistent test failures and flakes. The identified flakes will then show here.
Occurrences of the same tests are aggregated and presented in following forms:
To avoid noise, a flake bug is filed on Monorail only if all the following requirements are met:
Additionally, flaky tests could be grouped so that each group can have the same bug. Flaky tests will be grouped together if:
The Report page reports weekly flakiness states of Chrome Browser project and breakdown by each crbug component.
Report is generated every Monday 12:00AM PST for data of last week.
It reports the following states:
Project view shows aggregated flakiness data and top flake components with the most flakes or highest negative impacts. To look for a particular component, please use the search field on top of the page.
Component view shows