Test apk(s) and tests to run on Android are configurable on a per Android release basis by editing webview_cts_gcs_path.json
.
{ { "<Android dessert letter>": { "arch": { "<arch1>": { "filename": "<relative path to cts_archive_dir of cts zip>", "_origin": "<CTS zip download url>", "unzip_dir": "<relative path to work directory where cts should be unzipped to>" }, "<arch2>": { "filename": "<relative path to cts_archive_dir of cts zip>", "_origin": "<CTS zip download url>", "unzip_dir": "<relative path to work directory where cts should be unzipped to>" } }, "test_runs": [ { "apk": "location of the test apk in the cts zip file", "excludes": [ { "match": "<class#testcase (wildcard supported) expression of test to skip>", "_bug_id": "<bug reference comment, optional>" } ] }, { "apk": "location of the test apk in the cts zip file", "includes": [ { "match": "<class#testcase (wildcard supported) expression of test to run>" } ] } ] } }, ... }
CTS regressions are more serious than most test failures. CTS failures block Android vendors from shipping devices and prevent the WebView team from dropping new Chrome and WebView APKs in the Android source tree. If you need to disable a test, please file a P1 crbug with ReleaseBlock-Dev in the Mobile>WebView
component.
If you must disable a test, you can add an entry to the excludes
list for the correct apk (most tests belong to CtsWebkitTestCases.apk
) under test_runs
for each OS level which is failing.
Before re-enabling tests, make sure it's actually safe to enable the test again.
aosp/oreo-cts-dev
.Re-enabling the test case is as simple as removing it from the excludes
for the relevant OS levels. Please verify this change by adding the android-webview-*
trybots (not enabled by default).
Android generally has strict backward compatibility requirements, and this extends to CTS. However, sometimes it's appropriate to change the test logic rather than restoring the old chromium behavior, such as when the test logic is responsible for flakiness or relies on an invalid assumption. Please reach out to the WebView team quickly if you think a CTS test needs to change (the failure is still considered ReleaseBlock-Dev until the test change actually lands in Android).
Any CTS changes must be backward compatible. The original WebView version which shipped on that OS version must continue to pass the revised CTS test.