Layout tests are used by Blink to test many components, including but not limited to layout and rendering. In general, layout tests involve loading pages in a test renderer (content_shell
) and comparing the rendered output or JavaScript output against an expected output file.
This document covers running and debugging existing layout tests. See the Writing Layout Tests documentation if you find yourself writing layout tests.
Before you can run the layout tests, you need to build the blink_tests
target to get content_shell
and all of the other needed binaries.
ninja -C out/Release blink_tests
On Android (layout test support currently limited to KitKat and earlier) you need to build and install content_shell_apk
instead. See also: Android Build Instructions.
ninja -C out/Default content_shell_apk adb install -r out/Default/apks/ContentShell.apk
On Mac, you probably want to strip the content_shell binary before starting the tests. If you don‘t, you’ll have 5-10 running concurrently, all stuck being examined by the OS crash reporter. This may cause other failures like timeouts where they normally don't occur.
strip ./xcodebuild/{Debug,Release}/content_shell.app/Contents/MacOS/content_shell
TODO: mention testing/xvfb.py
The test runner script is in third_party/WebKit/Tools/Scripts/run-webkit-tests
.
To specify which build directory to use (e.g. out/Default, out/Release, out/Debug) you should pass the -t
or --target
parameter. For example, to use the build in out/Default
, use:
python third_party/WebKit/Tools/Scripts/run-webkit-tests -t Default
For Android (if your build directory is out/android
):
python third_party/WebKit/Tools/Scripts/run-webkit-tests -t android --android
Tests marked as [ Skip ]
in TestExpectations won't be run at all, generally because they cause some intractable tool error. To force one of them to be run, either rename that file or specify the skipped test as the only one on the command line (see below). Read the Layout Test Expectations documentation to learn more about TestExpectations and related files.
To run only some of the tests, specify their directories or filenames as arguments to run_webkit_tests.py
relative to the layout test directory (src/third_party/WebKit/LayoutTests
). For example, to run the fast form tests, use:
Tools/Scripts/run-webkit-tests fast/forms
Or you could use the following shorthand:
Tools/Scripts/run-webkit-tests fast/fo\*
Example: To run the layout tests with a debug build of content_shell
, but only test the SVG tests and run pixel tests, you would run:
Tools/Scripts/run-webkit-tests -t Default svg
As a final quick-but-less-robust alternative, you can also just use the content_shell executable to run specific tests by using (for Windows):
out/Default/content_shell.exe --run-layout-test --no-sandbox full_test_source_path
as in:
out/Default/content_shell.exe --run-layout-test --no-sandbox \ c:/chrome/src/third_party/WebKit/LayoutTests/fast/forms/001.html
but this requires a manual diff against expected results, because the shell doesn't do it for you.
To see a complete list of arguments supported, run: run-webkit-tests --help
This script has a lot of command line flags. You can pass --help
to the script to see a full list of options. A few of the most useful options are below:
Option | Meaning |
---|---|
--debug | Run the debug build of the test shell (default is release). Equivalent to -t Debug |
--nocheck-sys-deps | Don't check system dependencies; this allows faster iteration. |
--verbose | Produce more verbose output, including a list of tests that pass. |
--no-pixel-tests | Disable the pixel-to-pixel PNG comparisons and image checksums for tests that don't call testRunner.dumpAsText() |
--reset-results | Write all generated results directly into the given directory, overwriting what's there. |
--new-baseline | Write all generated results into the most specific platform directory, overwriting what's there. Equivalent to --reset-results --add-platform-expectations |
--renderer-startup-dialog | Bring up a modal dialog before running the test, useful for attaching a debugger. |
--fully-parallel | Run tests in parallel using as many child processes as the system has cores. |
--driver-logging | Print C++ logs (LOG(WARNING), etc). |
A test succeeds when its output matches the pre-defined expected results. If any tests fail, the test script will place the actual generated results, along with a diff of the actual and expected results, into src/out/Default/layout_test_results/
, and by default launch a browser with a summary and link to the results/diffs.
The expected results for tests are in the src/third_party/WebKit/LayoutTests/platform
or alongside their respective tests.
A test that runs but produces the wrong output is marked as “failed”, one that causes the test shell to crash is marked as “crashed”, and one that takes longer than a certain amount of time to complete is aborted and marked as “timed out”. A row of dots in the script's output indicates one or more tests that passed.
The TestExpectations file (and related files) contains the list of all known layout test failures. See the Layout Test Expectations documentation for more on this.
There are two ways to run layout tests with additional command-line arguments:
Using --additional-driver-flag
:
run-webkit-tests --additional-driver-flag=--blocking-repaint
This tells the test harness to pass --blocking-repaint
to the content_shell binary.
It will also look for flag-specific expectations in LayoutTests/FlagExpectations/blocking-repaint
, if this file exists. The suppressions in this file override the main TestExpectations file.
Using a virtual test suite defined in LayoutTests/VirtualTestSuites. A virtual test suite runs a subset of layout tests under a specific path with additional flags. For example, you could test a (hypothetical) new mode for repainting using the following virtual test suite:
{ "prefix": "blocking_repaint", "base": "fast/repaint", "args": ["--blocking-repaint"], }
This will create new “virtual” tests of the form virtual/blocking_repaint/fast/repaint/...`` which correspond to the files under
LayoutTests/fast/repaintand pass
--blocking-repaint` to content_shell when they are run.
These virtual tests exist in addition to the original fast/repaint/...
tests. They can have their own expectations in TestExpectations, and their own baselines. The test harness will use the non-virtual baselines as a fallback. However, the non-virtual expectations are not inherited: if fast/repaint/foo.html
is marked [ Fail ]
, the test harness still expects virtual/blocking_repaint/fast/repaint/foo.html
to pass. If you expect the virtual test to also fail, it needs its own suppression.
The “prefix” value does not have to be unique. This is useful if you want to run multiple directories with the same flags (but see the notes below about performance). Using the same prefix for different sets of flags is not recommended.
For flags whose implementation is still in progress, virtual test suites and flag-specific expectations represent two alternative strategies for testing. Consider the following when choosing between them:
The waterfall builders and try bots will run all virtual test suites in addition to the non-virtual tests. Conversely, a flag-specific expectations file won't automatically cause the bots to test your flag - if you want bot coverage without virtual test suites, you will need to set up a dedicated bot for your flag.
Due to the above, virtual test suites incur a performance penalty for the commit queue and the continuous build infrastructure. This is exacerbated by the need to restart content_shell
whenever flags change, which limits parallelism. Therefore, you should avoid adding large numbers of virtual test suites. They are well suited to running a subset of tests that are directly related to the feature, but they don't scale to flags that make deep architectural changes that potentially impact all of the tests.
All bugs, associated with layout test failures must have the Test-Layout label. Depending on how much you know about the bug, assign the status accordingly:
When creating a new layout test bug, please set the following properties:
You can also use the Layout Test Failure template, which will pre-set these labels for you.
After the layout tests run, you should get a summary of tests that pass or fail. If something fails unexpectedly (a new regression), you will get a content_shell window with a summary of the unexpected failures. Or you might have a failing test in mind to investigate. In any case, here are some steps and tips for finding the problem.
http://localhost:8000/
and proceed from there.) The best tests describe what they‘re looking for, but not all do, and sometimes things they’re not explicitly testing are still broken. Compare it to Safari, Firefox, and IE if necessary to see if it‘s correct. If you’re still not sure, find the person who knows the most about it and ask../run_webkit_tests.py path/to/your/test.html --full-results-html
will produce a page including links to the expected result, actual result, and diff.--sources
option to run_webkit_tests.py
to see exactly which expected result it's comparing to (a file next to the test, something in platform/mac/, something in platform/chromium-win/, etc.)file:
URL.file:
or http:
) as the command argument in the Debugging section of the content_shell project Properties.--run-layout-test
, followed by the URL (file:
or http:
) to your test. More information about running layout tests in content_shell can be found here.TestShell::RunFileTest()
call in content_shell_main.cc
, or at shell->LoadURL() within RunFileTest()
in content_shell_win.cc
.To run the server manually to reproduce/debug a failure:
cd src/third_party/WebKit/Tools/Scripts run-blink-httpd start
The layout tests will be served from http://127.0.0.1:8000
. For example, to run the test LayoutTest/http/tests/serviceworker/chromium/service-worker-allowed.html
, navigate to http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html
. Some tests will behave differently if you go to 127.0.0.1 vs localhost, so use 127.0.0.1.
To kill the server, run run-blink-httpd --server stop
, or just use taskkill
or the Task Manager on Windows, and killall
or Activity Monitor on MacOS.
The test server sets up an alias to LayoutTests/resources
directory. In HTTP tests, you can access testing framework at e.g. src="/js-test-resources/js-test.js"
.
Check https://test-results.appspot.com/ to see how a test did in the most recent ~100 builds on each builder (as long as the page is being updated regularly).
A timeout will often also be a text mismatch, since the wrapper script kills the content_shell before it has a chance to finish. The exception is if the test finishes loading properly, but somehow hangs before it outputs the bit of text that tells the wrapper it's done.
Why might a test fail (or crash, or timeout) on buildbot, but pass on your local machine?
Add window.debugTest = true;
to your test code as follows:
window.debugTest = true; function test() { /* TEST CODE */ }
Do one of the following:
blink/tools/run_layout_tests.sh --additional_driver_flag='--remote-debugging-port=9222' --time-out-ms=6000000
out/Default/content_shell --remote-debugging-port=9222 --run-layout-test http://127.0.0.1:8000/path/to/test.html
Open http://localhost:9222
in a stable/beta/canary Chrome, click the single link to open the devtools with the test loaded.
You may need to replace devtools.html with inspector.html in your URL (or you can use local chrome inspection of content_shell from chrome://inspect
instead)
In the loaded devtools, set any required breakpoints and execute test()
in the console to actually start the test.
By default, text-only tests (ones that call testRunner.dumpAsText()
) produce only text results. Other tests produce both new text results and new image results (the image baseline comprises two files, -expected.png
and -expected.checksum
). So you'll need either one or three -expected.\*
files in your new baseline, depending on whether you have a text-only test or not. If you enable --no-pixel-tests
, only new text results will be produced, even for tests that do image comparisons.
cd src/third_party/WebKit Tools/Scripts/run-webkit-tests --new-baseline foo/bar/test.html
The above command will generate a new baseline for LayoutTests/foo/bar/test.html
and put the output files in the right place, e.g. LayoutTests/platform/chromium-win/LayoutTests/foo/bar/test-expected.{txt,png,checksum}
.
When you rebaseline a test, make sure your commit description explains why the test is being re-baselined. If this is a special case (i.e., something we've decided to be different with upstream), please put a README file next to the new expected output explaining the difference.
In addition to layout tests developed and run just by the Blink team, there is also a shared test suite, see web-platform-tests.
See bugs with the component Blink>Infra for issues related to Blink tools, include the layout test runner.
fast/dom/object-embed-plugin-scripting.html
and plugins/embed-attributes-setting.html
are expected to fail.