[reland] Make layout tests use default isolation (e.g. site-per-process)
This CL is 1) a reland of r604955 + 2) fixes. Original CL description
follows below.
Summary
=======
This CL makes layout tests use the default site isolation from the
platform they are run on (instead of opting out of strict site isolation
via LayoutTestContentBrowserClient::ShouldEnableStrictSiteIsolation).
Additionally, on platforms where strict site isolation is enabled,
layout tests opt into slightly stricter isolation by enabling isolation
of same-site origins used by Web Platform Tests - this ensures that
features covered by WPT also get sufficient coverage of out-of-process
iframes (OOPIFs).
After this CL, expectations for tests that differ in behavior with and
without OOPIFs are being moved from
LayoutTests/FlagExpectations/site-per-process to:
- LayoutTests/VirtualTestSuites (virtual/not-site-per-process suite)
- LayoutTests/virtual/not-site-per-process/README.md
- LayoutTests/TestExpectations and LayoutTests/NeverFixTests
("Site Isolation failures" section)
Desirability
============
The CL helps with the following:
- Focusing on testing the mode that is actually shipped to end users
- Helping ensure that newly developed features get site-per-process
coverage without having to set up a separate step (i.e. it is
sufficient to set-up a bot that runs layout tests with
--enable-features=NewFeature without also having to have a separate
test step for runing layout tests with *both*
--enable-features=NewFeature *and* --site-per-process
This CL does *not* help with reducing requirements for CQ capacity,
because we need to maintain a separate
not_site_per_process_webkit_layout_tests step to make sure that tests
pass without isolation (which is the mode Chrome ships on Android).
Also note that layout test coverage on Android is very sparse - see
https://groups.google.com/a/chromium.org/d/topic/blink-dev/SOXhTYysYkE/discussion
Preserving test coverage
========================
Most tests
----------
The CL preserves covering most layout tests with and without OOPIFs, by
relying on the fact that CQ/waterfall run layout tests on both kinds of
platforms - ones that default to strict site isolation (desktop
platforms) and ones that default to no site isolation (Android).
Tests that used to be excluded FlagExpectations/site-per-process
----------------------------------------------------------------
Around 40 tests fail when run in presence of OOPIFs. Such tests are
disabled by this CL by moving test expectations from
FlagExpectations/site-per-process into the main TestExpectations file.
The CL preserves non-OOPIF test coverage provided by the disabled tests
by introducing virtual/not-site-per-process directory which runs all
such tests with site isolation disabled. Using a virtual test suites
for preserving the test coverage relies on the ability to have separate
test expectations for these tests (i.e. relying on the fact that
disabling these tests in TestExpectations doesn't disable their
virtual/not-site-per-process equivalents).
Note that the CL keeps isolating "oopif.test" site even in
virtual/not-site-per-process suite. This site should only be used by
tests that require an OOPIF.
Preserving site-per-process-specific test expectations
------------------------------------------------------
Some tests have site-per-process-specific expectations:
- http/tests/inspector-protocol/network/security-info-on-response.js
- http/tests/inspector-protocol/network/raw-headers-for-protected-document.js
The tests above highlight that cross-origin cookies are not displayed in
site-per-process mode (a known regression tracked by
https://crbug.com/849483). This CL preserves expectations and coverage
by shuffling things around:
- old, main expectation -> android expectation
- old, site-per-process expectation -> main expectation
There is one additional test with site-per-process-specific
expectations:
- external/wpt/dom/events/EventListener-addEventListener.sub.window.js
Unlike the other 2 tests, it seems less important to preserve exact test
expectations for the case when the test fails with Site Isolation.
Therefore this test is covered by virtual/not-site-per-process test
suite and an expectation for this test is added to the main
TestExpectations.
Lost test coverage
------------------
Even with extra caution described above, some test coverage may be lost:
- Features covered by tests only on one kind of platform (e.g. disabled
on Android) are at risk of losing OOPIF or non-OOPIF coverage.
- Before this CL, site-per-process was also applied to all other
`virtual/...` test suites. After this CL,
`virtual/not-site-per-process` will not provide such coverage.
Cleaned up test expectations
----------------------------
Some additional test expectations clean-up is done, while preserving
test coverage:
- The http/tests/perf/large-inlined-script.html test has been already
present in `SlowTests` and therefore I didn't include this test in the
new `virtual/not-site-per-process` suite.
- The http/tests/devtools/network/network-datareceived.js test was
already marked as expecting a `[Failure]` in TestExpectations and
therefore I didn't include this test in the new
`virtual/not-site-per-process` suite.
- The http/tests/devtools/console-cross-origin-iframe-logging.js test
was already marked as `[Timeout]` in the old TestExpectations but for
Win only. Since I can repro a timeout on Linux (with and without site
isolation), I just extended the old expectation to all platforms and I
didn't include this test in the new `virtual/not-site-per-process`
suite. I also removed the test from SlowTests (since timeouts in the
test are not expected everywhere).
Bug: 870761, 477150
Tbr: alexmos@chromium.org
Change-Id: If3c6fbc58ccdac8bc64a81963238db13c0bff391
Reviewed-on: https://chromium-review.googlesource.com/c/1318750
Commit-Queue: Łukasz Anforowicz <lukasza@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#609379}diff --git a/testing/buildbot/test_suite_exceptions.pyl b/testing/buildbot/test_suite_exceptions.pyl
index 9b75cfd..73d293b 100644
--- a/testing/buildbot/test_suite_exceptions.pyl
+++ b/testing/buildbot/test_suite_exceptions.pyl
@@ -839,6 +839,14 @@
},
},
},
+ 'not_site_per_process_webkit_layout_tests': {
+ 'remove_from': [
+ # chromium.linux
+ # TODO(dpranke): Should we be running this step on Linux Tests (dbg)(1)?
+ 'Linux Tests (dbg)(1)',
+ 'Linux Tests (dbg)(1)(32)',
+ ],
+ },
'notification_helper_unittests': {
'remove_from': [
# The test uses WinRT, which only exists in Windows 8 or above.
@@ -921,14 +929,6 @@
},
},
},
- 'site_per_process_webkit_layout_tests': {
- 'remove_from': [
- # chromium.linux
- # TODO(dpranke): Should we be running this step on Linux Tests (dbg)(1)?
- 'Linux Tests (dbg)(1)',
- 'Linux Tests (dbg)(1)(32)',
- ],
- },
'sizes': {
'remove_from': [
'win32-dbg',
