Running Web Platform Tests with run_wpt_tests.py

run_web_tests.py runs web tests with content shell through protocol mode. See web_tests.md for details. run_wpt_tests.py is a WebDriver-based alternative that can run web platform tests with Chrome, headless shell, Chrome Android, and WebView. This document explains how to use run_wpt_tests.py in these scenarios.

Running Web Platform Tests for Desktop Platforms

On Linux, macOS, and Windows, run_wpt_tests.py supports testing with Chrome or headless shell. Chrome is closer to the binary Google ships to end users, but is generally slower. Headless shell is a lightweight alternative that suffices for testing features implemented entirely in Blink.

Running Tests Locally

First, you will need to build the blink_tests target as you were running web tests before. This will build headless_shell, chrome, chromedriver, content_shell and all other needed binaries to run web tests and WPTs.

autoninja -C out/Default blink_tests

Once the build is done, running tests is very similar to how you would run tests with run_web_tests.py. For example, to run all tests under external/wpt/html/dom, run:

third_party/blink/tools/run_wpt_tests.py --target=Default --product=headless_shell external/wpt/html/dom

--product (or -p) selects which browser to test with. Supported values are:

• headless_shell (default if --product is not specified)
• chrome
• chrome_android (aliased as clank; see additional instructions)
• android_webview (aliased as webview; see additional instructions)

Also, consider using -v to get browser logs. It can be provided multiple times to increase verbosity.

run_wpt_tests.py --help shows a full description of run_wpt_tests.py's CLI, which resembles that of run_web_tests.py.

Running Tests in CQ/CI

To satisfy different testing requirements, WPT coverage in CQ/CI is partitioned between suites that target different //content embedders:

Suite Name	Browser Under Test	Harness	Tests Run
headless_shell_wpt_tests	headless_shell	run_wpt_tests.py	The default test suite for WPTs if not specified otherwise.
chrome_wpt_tests	chrome --headless=new	run_wpt_tests.py	Tests that depend on the //chrome layer. Can be slow, so prefer headless_shell testing if possible.
blink_wpt_tests	content_shell --run-web-tests	run_web_tests.py	Tests under internal WPTs plus any public WPTs not migrated yet.

To avoid redundant coverage, each WPT should run in exactly one suite listed above. The chrome.filter file lists tests that chrome_wpt_tests should run, and that headless_shell_wpt_tests and blink_wpt_tests should skip. content_shell.filter file lists tests that currently run in blink_wpt_tests, and will be removed once it only contains tests under wpt_internal. Tests not listed in either file run in headless_shell_wpt_tests by default.

Running tests in blink_wpt_tests is discouraged because run_web_tests.py doesn't drive tests through standard WebDriver endpoints. This can cause blink_wpt_tests results to diverge from the Chrome results published to wpt.fyi. We should generally not add new tests to content_shell.filter.

Test Expectations and Baselines

To suppress failures, run_wpt_tests.py uses the same *-expected.txt and TestExpectations files that run_web_tests.py uses.

Running webdriver tests with Chrome

wdspec tests are a subset of WPT that verifies conformance to the WebDriver specification. run_wpt_tests.py can run wdspec tests like any other WPT:

third_party/blink/tools/run_wpt_tests.py -t Default -p chrome \
  external/wpt/webdriver/tests/classic/find_element/find.py

On the bots, the webdriver_wpt_tests suite runs wdspec tests separately from the other WPT types. The linux-blink-rel builder can provide results for rebaselining.

Running Web Platform Tests on Android

See here for Android specific instructions.

Debugging Support

Per-Test Tracing

The --enable-per-test-tracing option will record trace events for each test. One use case is to record JavaScript bindings used:

1. Build blink_tests with the GN arg extended_tracing_enabled=true.

2. Run the desired tests with --enable-per-test-tracing=blink.bindings.

3. Locate trace files written to //out/$target/layout-test-results/**/*-trace.json. They're also downloadable from the results viewer:

4. Open the desired trace files in Perfetto:

Text-Based Debuggers

To interactively debug WPTs, prefix the run_wpt_tests.py command with debug_renderer to attach a debugger to a desired renderer.

For other use cases, see these debugging tips.

FAQ

• Do headless shell and Chrome support MojoJS bindings?
  • Yes. run_wpt_tests.py enables the MojoJS and MojoJSTest features and serves //out/<target>/gen/ as /gen/ in wptserve. However, in the public WPT suite, testdriver.js APIs must be backed by fully-specified testing APIs (preferably implemented with WebDriver or alternatively with MojoJS). Tests that rely on unspecified testing APIs cannot be put in WPT, but may live in chromium's own wpt_internal. See https://github.com/web-platform-tests/rfcs/issues/172 for additional discussion.

Known Issues

The wptrunner-migration hostlist tracks test results where headless shell and content shell differ. For runner bugs and feature requests, please file an issue against Blink>Infra.