run_web_tests.py runs web tests with content shell through protocol mode. See web_tests.md for details. run_wpt_tests.py is a WebDriver-based alternative that can run web platform tests with Chrome, headless shell, Chrome Android, and WebView. This document explains how to use run_wpt_tests.py in these scenarios.
On Linux, macOS, and Windows, run_wpt_tests.py supports testing with Chrome or headless shell. Chrome is closer to the binary Google ships to end users, but is generally slower. Headless shell is a lightweight alternative that suffices for testing features implemented entirely in Blink.
First, you will need to build one of the following targets to get all needed binaries:
autoninja -C out/Default chrome_wpt_tests # For testing with `chrome` autoninja -C out/Default headless_shell_wpt # For testing with `headless_shell`
Once the build is done, running tests is very similar to how you would run tests with run_web_tests.py. For example, to run all tests under external/wpt/html/dom, run:
third_party/blink/tools/run_wpt_tests.py --target=Default --product=headless_shell external/wpt/html/dom
--product (or -p) selects which browser to test with. Supported values are:
headless_shell (default if --product is not specified)chromechrome_android (aliased as clank; see additional instructions)android_webview (aliased as webview; see additional instructions)Also, consider using -v to get browser logs. It can be provided multiple times to increase verbosity.
run_wpt_tests.py --help shows a full description of run_wpt_tests.py's CLI, which resembles that of run_web_tests.py.
To satisfy different testing requirements, WPT coverage in CQ/CI is partitioned between suites that target different //content embedders:
| Suite Name | Browser Under Test | Harness | Tests Run |
|---|---|---|---|
blink_wpt_tests | content_shell --run-web-tests | run_web_tests.py | Tests that depend on web test-specific features (e.g., internal WPTs that depend on nonstandard window.internals or window.testRunner APIs). |
chrome_wpt_tests | chrome --headless=new | run_wpt_tests.py | Tests that depend on the //chrome layer. Can be slow, so prefer headless_shell testing if possible. |
headless_shell_wpt_tests | headless_shell | run_wpt_tests.py | All other tests. Most WPTs should eventually run here. |
To avoid redundant coverage, each WPT should run in exactly one suite listed above. The chrome.filter file lists tests that chrome_wpt_tests should run, and that headless_shell_wpt_tests and blink_wpt_tests should skip. headless_shell.filter works similarly. Tests not listed in either file run in blink_wpt_tests by default.
blink_wpt_tests is discouraged because run_web_tests.py doesn't drive tests through standard WebDriver endpoints. This can cause blink_wpt_tests results to diverge from the Chrome results published to wpt.fyi. You can help unblock the eventual deprecation of blink_wpt_tests by adding tests that you own to either filter file.To suppress failures, run_wpt_tests.py uses the same *-expected.txt and TestExpectations files that run_web_tests.py uses.
wdspec tests are a subset of WPT that verifies conformance to the WebDriver specification. run_wpt_tests.py can run wdspec tests like any other WPT:
third_party/blink/tools/run_wpt_tests.py -t Default -p chrome \ external/wpt/webdriver/tests/classic/find_element/find.py
On the bots, the webdriver_wpt_tests suite runs wdspec tests separately from the other WPT types. The linux-blink-rel builder can provide results for rebaselining.
See here for Android specific instructions.
To be updated.
To interactively debug WPTs, prefix the run_wpt_tests.py command with debug_renderer to attach a debugger to a desired renderer.
For other use cases, see these debugging tips.
run_wpt_tests.py enables the MojoJS and MojoJSTest features and serves //out/<target>/gen/ as /gen/ in wptserve. However, in the public WPT suite, testdriver.js APIs backed by standard WebDriver endpoints should be preferred over polyfills backed by MojoJS, which are Chromium-specific. See https://github.com/web-platform-tests/rfcs/issues/172 for additional discussion.The wptrunner-migration hostlist tracks test results where headless shell and content shell differ. For runner bugs and feature requests, please file an issue against Blink>Infra.