Running Web Platform Tests on Android

Overview

This document provides a guide to running Web Platform Tests on Android.

For general instruction for running the Web Platform Tests, you should read Running Web Platform Tests with run_wpt_tests.py.

Initial Setup

Please follow the steps at Checking out and building Chromium for Android to setup your local environment. Once that is done, you need to build one of the following targets:

autoninja -C out/Default chrome_public_wpt        # For testing with Chrome Android
autoninja -C out/Default trichrome_webview_wpt_64 # For testing with WebView

Running the Tests

Once you have Chrome Android/WebView and chromedriver built, you can launch run_wpt_tests.py to run WPTs. You can either run the tests on an Android emulator or a real Android device.

Note: You can enable Incremental Install for Chrome Android by setting incremental_install = True in your args.gn file.

This will cause the test harness to use the build directory's incremental installation script instead of the default browser APK.

Running WPTs on an Android emulator

You will need to follow the steps in Using an Android Emulator to be ready to run the Android Emulator. Passing the --avd-config option to run_wpt_tests.py will launch an emulator that will be shut down after running the tests.

The example below runs external/wpt/badging/badge-success.https.html in Chrome Android on Android 13:

third_party/blink/tools/run_wpt_tests.py \
  -t Default \
  -p clank \
  --avd-config=tools/android/avd/proto/android_33_google_apis_x64.textpb \
  external/wpt/badging/badge-success.https.html

-t Default: Use the build in //out/Default/
-p clank: Runs the tests using Chrome for Android (clank).
--avd-config=tools/.../android_33_google_apis_x64.textpb: Runs the tests on Android 13 emulator (Google API 33).

To run the example in WebView:

$ third_party/blink/tools/run_wpt_tests.py \
  -t Default \
  -p android_webview \
  --webview-provider out/Default/apks/TrichromeWebView64.apk \
  --additional-apk out/Default/apks/TrichromeLibrary64.apk \
  --avd-config=tools/android/avd/proto/android_33_google_apis_x64.textpb \
  external/wpt/badging/badge-success.https.html

-p webview: Runs the tests using WebView.
--webview-provider out/.../TrichromeWebView64.apk: Specify TrichromeWebView64 as WebView APK.
--additional-apk out/.../TrichromeLibrary64.apk: Install TrichromeLibrary64 needed for the WebView APK after Android 10 (see WebView Channels).

Alternatively, you can launch the emulator yourself and run_wpt_tests.py will detect and connect to the emulator and run WPTs with it. This can save you the time to repeatedly launch the emulator. The commands below show how this works.

$ tools/android/avd/avd.py start \
  --avd-config=tools/android/avd/proto/android_33_google_apis_x64.textpb
$ third_party/blink/tools/run_wpt_tests.py \
  -t Default \
  -p clank \
  external/wpt/badging/badge-success.https.html

Running WPTs on a real Android device

run_wpt_tests.py should be able to work with a real device as long as the device can be found by adb devices. You will need to make sure the ABI matches and these steps are followed.

Running Tests in CQ/CI

To test Android-specific behaviors, the android-chrome-13-x64-wpt-android-specific and android-webview-13-x64-wpt-android-specific builders run the tests in android.filter and webview.filter respectively. Developers can add additional tests to the lists when necessary. A resource request might be needed depending on the case. See below for details.

To satisfy different testing requirements, WPT coverage in CQ/CI is partitioned between suites that target different //content embedders:

Suite Name	Browser Under Test	Harness	Tests Run
`android_chrome_wpt_tests`	`chrome_android`	`run_wpt_tests.py`	Tests listed in `android.filter`.
`android_webview_wpt_tests`	`android_webview`	`run_wpt_tests.py`	Tests listed in `webview.filter`.

Test expectations and Baselines

Expected failures on Chrome Android or WebView should be added to the default TestExpectations file with the modifier “Android” or “Webview” respectively.

For baselines:

Chrome Android specific baselines reside at third_party/blink/web_tests/platform/android, and fall back to third_party/blink/web_tests/platform/linux.
WebView specific baselines reside at third_party/blink/web_tests/platform/webview, and fall back to third_party/blink/web_tests/platform/linux.

To update baselines:

Trigger tryjob(s)

For Chrome Androids: trigger android-chrome-13-x64-wpt-android-specific
For WebView: trigger android-webview-13-x64-wpt-android-specific

Run the rebaseline tool after the results are ready.

Run more tests with Chrome Android or WebView

A resource request would be needed if running the additional tests increases the total test time by more than one minute. If this is the case, developers can use the bot estimator to estimate the bots needed, then follow the steps at go/i-need-hw to get the resources. The tests can be added to the filter file after the resource is deployed.

Please make sure there is no unexpected failures on the related steps before using the bot estimator, as the bot will retry those tests and that will increase the total test time.

Known Issues

For runner bugs and feature requests, please file an issue against Blink>Infra .