docs/testing/run_web_platform_tests.md - chromium/src.git - Git at Google

 # Running Web Platform Tests with run_wpt_tests.py

 `run_web_tests.py` runs web tests with content shell through [protocol mode].
 See [web_tests.md](web_tests.md) for details.
 `run_wpt_tests.py` is a WebDriver-based alternative that can run [web platform
 tests] with [Chrome], [headless shell], Chrome Android, and WebView.
 This document explains how to use `run_wpt_tests.py` in these scenarios.

 [web platform tests]: web_platform_tests.md
 [Chrome]: /chrome
 [headless shell]: /headless

 [TOC]

 ## Running Web Platform Tests for Desktop Platforms

 On Linux, macOS, and Windows, `run_wpt_tests.py` supports testing with [Chrome]
 or [headless shell].
 Chrome is closer to the binary Google ships to end users, but is generally
 slower.
 Headless shell is a lightweight alternative that suffices for testing features
 implemented entirely in Blink.

 ### Running Tests Locally

 First, you will need to build the `blink_tests` target as you were running web tests
 before. This will build `headless_shell`, `chrome`, `chromedriver`, `content_shell`
 and all other needed binaries to run web tests and WPTs.

 ```bash
 autoninja -C out/Default blink_tests
 ```

 Once the build is done, running tests is very similar to how you would run
 tests with `run_web_tests.py`.
 For example, to run all tests under `external/wpt/html/dom`, run:

 ```bash
 third_party/blink/tools/run_wpt_tests.py --target=Default --product=headless_shell external/wpt/html/dom
 ```

 `--product` (or `-p`) selects which browser to test with.
 Supported values are:

 * `headless_shell` (default if `--product` is not specified)
 * `chrome`
 * `chrome_android` (aliased as `clank`; see
   [additional instructions](#Running-Web-Platform-Tests-on-Android))
 * `android_webview` (aliased as `webview`; see
   [additional instructions](#Running-Web-Platform-Tests-on-Android))

 Also, consider using `-v` to get browser logs.
 It can be provided multiple times to increase verbosity.

 `run_wpt_tests.py --help` shows a full description of `run_wpt_tests.py`'s CLI,
 which resembles that of `run_web_tests.py`.

 ### Running Tests in CQ/CI

 To satisfy different testing requirements, WPT coverage in CQ/CI is partitioned
 between suites that target different `//content` embedders:

 Suite Name | Browser Under Test | Harness | Tests Run
 --- | --- | --- | ---
 `headless_shell_wpt_tests` | `headless_shell` | `run_wpt_tests.py` | The default test suite for WPTs if not specified otherwise.
 `chrome_wpt_tests` | `chrome --headless=new` | `run_wpt_tests.py` | Tests that depend on the `//chrome` layer. Can be slow, so prefer `headless_shell` testing if possible.
 `blink_wpt_tests` | `content_shell --run-web-tests` | `run_web_tests.py` | Tests under [internal WPTs] plus any public WPTs not migrated yet.

 To avoid redundant coverage, each WPT should run in exactly one suite listed
 above.
 The [`chrome.filter`][1] file lists tests that `chrome_wpt_tests` should run,
 and that `headless_shell_wpt_tests` and `blink_wpt_tests` should skip.
 [`content_shell.filter`][2] file lists tests that currently run in `blink_wpt_tests`,
 and will be removed once it only contains tests under `wpt_internal`. Tests
 not listed in either file run in `headless_shell_wpt_tests` by default.

 *** note
 Running tests in `blink_wpt_tests` is discouraged because `run_web_tests.py`
 doesn't drive tests through standard WebDriver endpoints.
 This can cause `blink_wpt_tests` results to diverge from the Chrome results
 published to [wpt.fyi]. We should generally not add new tests to
 content_shell.filter.
 ***

 [internal WPTs]: /third_party/blink/web_tests/wpt_internal

 ### Test Expectations and Baselines

 To suppress failures, `run_wpt_tests.py` uses the [same `*-expected.txt` and
 TestExpectations files](web_test_expectations.md) that `run_web_tests.py` uses.

 ### Running webdriver tests with Chrome

 [wdspec tests] are a subset of WPT that verifies conformance to the WebDriver
 specification.
 `run_wpt_tests.py` can run wdspec tests like any other WPT:

 ```bash
 third_party/blink/tools/run_wpt_tests.py -t Default -p chrome \
   external/wpt/webdriver/tests/classic/find_element/find.py
 ```

 On the bots, the `webdriver_wpt_tests` suite runs wdspec tests separately from
 the other WPT types.
 The `linux-blink-rel` builder can provide results for rebaselining.

 [wdspec tests]: https://web-platform-tests.org/writing-tests/wdspec.html

 ## Running Web Platform Tests on Android

 See [here](./run_web_platform_tests_on_android.md) for Android specific instructions.

 ## Debugging Support

 ### Per-Test Tracing

 The `--enable-per-test-tracing` option will record [trace events] for each test.
 One use case is to record JavaScript bindings used:

 1. Build `blink_tests` with the GN arg `extended_tracing_enabled=true`.
 1. Run the desired tests with `--enable-per-test-tracing=blink.bindings`.
 1. Locate trace files written to
    `//out/$target/layout-test-results/**/*-trace.json`.
    They're also downloadable from the [results viewer]:
      ![Trace link in results viewer](images/wpt_results_viewer_trace.png)

 1. Open the desired trace files [in Perfetto](https://ui.perfetto.dev/):
      ![WPT trace opened in Perfetto](images/wpt_bindings_trace.png)

 [trace events]: https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/
 [results viewer]: web_tests_addressing_flake.md#Understanding-builder-results

 ### Text-Based Debuggers

 To interactively debug WPTs, prefix the `run_wpt_tests.py` command with
 [`debug_renderer`][debug renderer] to attach a debugger to a desired renderer.

 For other use cases, see [these debugging tips].

 [these debugging tips]: /docs/linux/debugging.md

 ## FAQ

 * Do headless shell and Chrome support MojoJS bindings?
     * Yes.
       `run_wpt_tests.py` enables the `MojoJS` and `MojoJSTest` features and
       serves `//out/<target>/gen/` as `/gen/` in wptserve.
       However, in the public WPT suite, testdriver.js APIs must be backed by
       fully-specified testing APIs (preferably implemented with WebDriver or
       alternatively with MojoJS). Tests that rely on unspecified testing APIs
       cannot be put in WPT, but may live in chromium's own wpt_internal.
       See https://github.com/web-platform-tests/rfcs/issues/172 for additional
       discussion.

 ## Known Issues

 The [`wptrunner-migration`
 hostlist](https://issues.chromium.org/hotlists/6224346) tracks test results
 where headless shell and content shell differ.
 For runner bugs and feature requests, please file [an issue against
 `Blink>Infra`](https://issues.chromium.org/issues/new?component=1456928&template=1923166).

 [protocol mode]: /content/web_test/browser/test_info_extractor.h
 [debug renderer]: /third_party/blink/tools/debug_renderer
 [wpt.fyi]: https://wpt.fyi/results/?label=experimental&label=master&aligned

 [1]: /third_party/blink/web_tests/TestLists/chrome.filter
 [2]: /third_party/blink/web_tests/TestLists/content_shell.filter
 [3]: writing_web_tests.md#Relying-on-Blink_Specific-Testing-APIs
	# Running Web Platform Tests with run_wpt_tests.py

	`run_web_tests.py` runs web tests with content shell through [protocol mode].
	See [web_tests.md](web_tests.md) for details.
	`run_wpt_tests.py` is a WebDriver-based alternative that can run [web platform
	tests] with [Chrome], [headless shell], Chrome Android, and WebView.
	This document explains how to use `run_wpt_tests.py` in these scenarios.

	[web platform tests]: web_platform_tests.md
	[Chrome]: /chrome
	[headless shell]: /headless

	[TOC]

	## Running Web Platform Tests for Desktop Platforms

	On Linux, macOS, and Windows, `run_wpt_tests.py` supports testing with [Chrome]
	or [headless shell].
	Chrome is closer to the binary Google ships to end users, but is generally
	slower.
	Headless shell is a lightweight alternative that suffices for testing features
	implemented entirely in Blink.

	### Running Tests Locally

	First, you will need to build the `blink_tests` target as you were running web tests
	before. This will build `headless_shell`, `chrome`, `chromedriver`, `content_shell`
	and all other needed binaries to run web tests and WPTs.

	```bash
	autoninja -C out/Default blink_tests
	```

	Once the build is done, running tests is very similar to how you would run
	tests with `run_web_tests.py`.
	For example, to run all tests under `external/wpt/html/dom`, run:

	```bash
	third_party/blink/tools/run_wpt_tests.py --target=Default --product=headless_shell external/wpt/html/dom
	```

	`--product` (or `-p`) selects which browser to test with.
	Supported values are:

	* `headless_shell` (default if `--product` is not specified)
	* `chrome`
	* `chrome_android` (aliased as `clank`; see
	[additional instructions](#Running-Web-Platform-Tests-on-Android))
	* `android_webview` (aliased as `webview`; see
	[additional instructions](#Running-Web-Platform-Tests-on-Android))

	Also, consider using `-v` to get browser logs.
	It can be provided multiple times to increase verbosity.

	`run_wpt_tests.py --help` shows a full description of `run_wpt_tests.py`'s CLI,
	which resembles that of `run_web_tests.py`.

	### Running Tests in CQ/CI

	To satisfy different testing requirements, WPT coverage in CQ/CI is partitioned
	between suites that target different `//content` embedders:

	Suite Name \| Browser Under Test \| Harness \| Tests Run
	--- \| --- \| --- \| ---
	`headless_shell_wpt_tests` \| `headless_shell` \| `run_wpt_tests.py` \| The default test suite for WPTs if not specified otherwise.
	`chrome_wpt_tests` \| `chrome --headless=new` \| `run_wpt_tests.py` \| Tests that depend on the `//chrome` layer. Can be slow, so prefer `headless_shell` testing if possible.
	`blink_wpt_tests` \| `content_shell --run-web-tests` \| `run_web_tests.py` \| Tests under [internal WPTs] plus any public WPTs not migrated yet.

	To avoid redundant coverage, each WPT should run in exactly one suite listed
	above.
	The [`chrome.filter`][1] file lists tests that `chrome_wpt_tests` should run,
	and that `headless_shell_wpt_tests` and `blink_wpt_tests` should skip.
	[`content_shell.filter`][2] file lists tests that currently run in `blink_wpt_tests`,
	and will be removed once it only contains tests under `wpt_internal`. Tests
	not listed in either file run in `headless_shell_wpt_tests` by default.

	*** note
	Running tests in `blink_wpt_tests` is discouraged because `run_web_tests.py`
	doesn't drive tests through standard WebDriver endpoints.
	This can cause `blink_wpt_tests` results to diverge from the Chrome results
	published to [wpt.fyi]. We should generally not add new tests to
	content_shell.filter.
	***

	[internal WPTs]: /third_party/blink/web_tests/wpt_internal

	### Test Expectations and Baselines

	To suppress failures, `run_wpt_tests.py` uses the [same `*-expected.txt` and
	TestExpectations files](web_test_expectations.md) that `run_web_tests.py` uses.

	### Running webdriver tests with Chrome

	[wdspec tests] are a subset of WPT that verifies conformance to the WebDriver
	specification.
	`run_wpt_tests.py` can run wdspec tests like any other WPT:

	```bash
	third_party/blink/tools/run_wpt_tests.py -t Default -p chrome \
	external/wpt/webdriver/tests/classic/find_element/find.py
	```

	On the bots, the `webdriver_wpt_tests` suite runs wdspec tests separately from
	the other WPT types.
	The `linux-blink-rel` builder can provide results for rebaselining.

	[wdspec tests]: https://web-platform-tests.org/writing-tests/wdspec.html

	## Running Web Platform Tests on Android

	See [here](./run_web_platform_tests_on_android.md) for Android specific instructions.

	## Debugging Support

	### Per-Test Tracing

	The `--enable-per-test-tracing` option will record [trace events] for each test.
	One use case is to record JavaScript bindings used:

	1. Build `blink_tests` with the GN arg `extended_tracing_enabled=true`.
	1. Run the desired tests with `--enable-per-test-tracing=blink.bindings`.
	1. Locate trace files written to
	`//out/$target/layout-test-results/*/-trace.json`.
	They're also downloadable from the [results viewer]:
	![Trace link in results viewer](images/wpt_results_viewer_trace.png)

	1. Open the desired trace files [in Perfetto](https://ui.perfetto.dev/):
	![WPT trace opened in Perfetto](images/wpt_bindings_trace.png)

	[trace events]: https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/
	[results viewer]: web_tests_addressing_flake.md#Understanding-builder-results

	### Text-Based Debuggers

	To interactively debug WPTs, prefix the `run_wpt_tests.py` command with
	[`debug_renderer`][debug renderer] to attach a debugger to a desired renderer.

	For other use cases, see [these debugging tips].

	[these debugging tips]: /docs/linux/debugging.md

	## FAQ

	* Do headless shell and Chrome support MojoJS bindings?
	* Yes.
	`run_wpt_tests.py` enables the `MojoJS` and `MojoJSTest` features and
	serves `//out/<target>/gen/` as `/gen/` in wptserve.
	However, in the public WPT suite, testdriver.js APIs must be backed by
	fully-specified testing APIs (preferably implemented with WebDriver or
	alternatively with MojoJS). Tests that rely on unspecified testing APIs
	cannot be put in WPT, but may live in chromium's own wpt_internal.
	See https://github.com/web-platform-tests/rfcs/issues/172 for additional
	discussion.

	## Known Issues

	The [`wptrunner-migration`
	hostlist](https://issues.chromium.org/hotlists/6224346) tracks test results
	where headless shell and content shell differ.
	For runner bugs and feature requests, please file [an issue against
	`Blink>Infra`](https://issues.chromium.org/issues/new?component=1456928&template=1923166).

	[protocol mode]: /content/web_test/browser/test_info_extractor.h
	[debug renderer]: /third_party/blink/tools/debug_renderer
	[wpt.fyi]: https://wpt.fyi/results/?label=experimental&label=master&aligned

	[1]: /third_party/blink/web_tests/TestLists/chrome.filter
	[2]: /third_party/blink/web_tests/TestLists/content_shell.filter
	[3]: writing_web_tests.md#Relying-on-Blink_Specific-Testing-APIs