This page documents the experimental support for using the upstream wptrunner
tooling for running WPT tests in Chromium (vs the current approach that uses run_web_tests.py
).
It is written as a user guide. For technical details on the project, see the design doc.
run_web_tests.py
The main differences between run_web_tests.py
and wptrunner
are that:
wptrunner
runs the full chrome
binary, rather than the stripped-down content_shell
, andwptrunner
communicates with the binary via WebDriver (chromedriver
), instead of talking directly to the browser binary.These differences mean that any feature that works on upstream WPT today (e.g. print-reftests) should work in wptrunner
, but conversely features available to run_web_tests.py
(e.g. the internals
API) are not yet available to wptrunner
.
The runner script is testing/scripts/run_wpt_tests.py
. Before running the script, you must have built the necessary ninja targets:
autoninja -C out/Release wpt_tests_isolate
To run the script, you must enter the testing/scripts/
directory before executing it:
cd testing/scripts ./run_wpt_tests.py [test list]
The list of tests should be given relative to external/wpt/
, e.g. webauthn/createcredential-timeout.https.html
. Directories are also accepted. Omitting the test list will run all WPT tests.
Results from the run are placed in your build folder, in a folder called layout-test-results
(e.g. ../../out/Release/layout-test-results/
). Logs from the browser should be shown by the runner as it executes each test.
Useful flags:
-t/--target
: select which src/out/
sub-directory to use, e.g. -t Debug
. Defaults to Release
.--help
: show the help textAs of Q4 2020, an MVP of wptrunner in Chromium is being run with two customer teams: Web Payments and Web Identity. For these teams, two Linux-only bots have been brought up:
external/wpt/webauthn/
.external/wpt/{input-events, pointerevents, uievents}
, as well as external/wpt/infrastructure/testdriver/actions/
These bots run on the waterfall, but can also be run on CLs by clicking the Choose Tryjobs
button in Gerrit followed by searching for the bot name in the modal dialog that appears. One can also include the tag Cq-Include-Trybots: luci.chromium.try:linux-wpt-identity-fyi-rel
(or input) in the description for the CL, which will make the bot mandatory for that CL.
Results for the bots use the existing layout test results viewer (example).
Similar to run_web_tests.py
, wptrunner
offers the ability to add an expected status for a test or provide a baseline file that codifies what the output result should be.
By default wptrunner
will inherit expected statuses from TestExpecations
. This can currently be overridden by adding an entry to the WPTOverrideExpectations
file when wptrunner
has a different result than run_web_tests.py
. WPTOverrideExpectations
is however deprecated, and the preferred method for specifying expected results for wptrunner
is to use baseline files (which will also override a TestExpectation entry for the test).
Baseline files for wptrunner
use a different filename and format than the -expected.txt
files used by run_web_tests.py
. The ini-like baseline format is documented here, and baseline files should be placed alongside the test with an .ini
suffix:
external/wpt/folder/my-test.html external/wpt/folder/my-test-expected.txt <-- run_web_tests.py baseline external/wpt/folder/my-test.html.ini <-- wptrunner baseline
We currently do not support the full ini-like format that upstream WPT does; most notably we have chosen not to support dynamic conditionals (such as platform checks). Most .ini
baseline files in Chromium should have the form:
[my-test.html] expected: OK [First subtest name] expected: FAIL message: The failure message, e.g. assert_false didn't work [Second subtest name] expected: PASS
run_wpt_tests.py
today. In the future, we intend to allow pausing the browser after each test, and (long-term) intend to support hooking up gdb to test runs.Please file bugs and feature requests against Blink>Infra>Ecosystem
, tagging the title with [wptrunner]
.