| <!DOCTYPE html> |
| <html> |
| <head> |
| <title>CSS Regions testing strategy</title> |
| <meta http-equiv='Content-Type' content='text/html;charset=utf-8'/> |
| <!-- |
| === NOTA BENE === |
| For the three scripts below, if your spec resides on dev.w3 you can check them |
| out in the same tree and use relative links so that they'll work offline, |
| --> |
| <script src='http://www.w3.org/Tools/respec/respec-w3c-common' class='remove' async></script> |
| <script class='remove'> |
| var respecConfig = { |
| specStatus: "unofficial", |
| shortName: "css3-regions-test-strategy", |
| editors: [ |
| { |
| name: "Alan Stearns", mailto: "stearns@adobe.com", |
| company: "Adobe Systems, Inc.", companyURL: "http://www.adobe.com/" |
| }, |
| { |
| name: "Mihai Balan", mailto: "mibalan@adobe.com", |
| company: "Adobe Systems, Inc.", companyURL: "http://www.adobe.com/" |
| }, |
| ], |
| testSuiteURI: "http://test.csswg.org/suites/css3-regions/nightly-unstable/", |
| }; |
| </script> |
| <style> |
| a.bibref, |
| #references dt { |
| text-transform: uppercase; |
| } |
| </style> |
| </head> |
| <body> |
| <section id='abstract'> |
| <p> |
| This document is intended to be used as a guideline for the testing |
| activities related to the CSS Regions spec [[!css3-regions]]. Its main |
| goal is to provide an overview of the general testing areas, possible |
| caveats and testing aspects not immediately apparent from the spec. |
| Also, it provides a means of tracking the progress of the CSS Regions |
| spec testing. |
| </p> |
| <p> |
| This document is not meant to replace the spec in determining the |
| normative and non-normative assertions to be tested, but rather |
| complement it. |
| </p> |
| </section> |
| |
| <section> |
| <h2>Introduction</h2> |
| <p> |
| As CSS moved away from the monolithic development of CSS 2.1 to the |
| modular development of CSS 3, the number of proposed new features and |
| the complexity of the layout landscape have increased dramatically. |
| While this directly translates to increased flexibility and agility in |
| adopting and implementing new CSS features, it also increases the |
| complexity of testing CSS features and the need for coordinating the |
| testing efforts. Also, the need for testing coordination increases as |
| crowd-sourcing efforts like |
| <a href="http://testthewebforward.org/" target="_blank">Test the Web |
| Forward</a> present people less familiar with the processes and policies |
| of the W3C with the opportunity to contribute new tests. |
| </p> |
| <p> |
| Except when defining new behaviors or redefining old behaviors, the |
| implicit assumption for new CSS modules is that they play nicely with |
| other modules or properties defined in CSS 2.1 [[CSS21]]. As CSS |
| Regions is a spec that touches many aspects of layout, styling and CSSOM, |
| it's not unreasonable to want to test the spec against these implicit |
| assumptions, too. |
| </p> |
| <p> |
| This testing strategy document is meant to complement the CSS Regions |
| spec and the existing test suite by providing an overview of the testing |
| areas (especially the less apparent ones) and tracking the progress of |
| the testing activities against these test areas. |
| </p> |
| </section> |
| <section> |
| <h2>Goals</h2> |
| <p> |
| To ensure a comprehensive test suite with useful, high quality tests, a |
| number of goals are proposed. They range from process goals (how to |
| conduct testing) to implementation goals (how to write good tests). |
| </p> |
| <section> |
| <h3>Enabling easy test contribution</h3> |
| <p> |
| An important vector in successfully testing CSS Regions is to |
| enable easy test contributions, both from W3C partners and from |
| non-W3C members that wish to contribute. This is achieved by clearly |
| marking and explaining the areas that need testing, linked to existing |
| tests and general testing progress. |
| </p> |
| </section> |
| <section> |
| <h3>Providing guidance on testing</h3> |
| <p> |
| In order to increase the quality of the test contributions, this |
| document offers a set of guidelines for conducting testing (see |
| <a href="#approach" class="sectionRef"></a>) and a testing progress |
| tracker to increase the surface coverage of tests (see |
| <a href="#test-progress-tracking" class="sectionRef"></a>). |
| </p> |
| </section> |
| <section> |
| <h3>Creating automation-friendly tests</h3> |
| <p> |
| In terms of actual tests produced for the CSS Regions, the main goal |
| is to ensure that most tests are automatable (i.e. they're either |
| reftests or use <code>testharness.js</code>). Even where manual tests |
| are absolutely necessary they should be written so that they can be |
| easily automated – as there are on-going efforts to make |
| WebDriver [[webdriver]] automated tests a first class citized in W3C |
| testing. This means that even if a manual test requires user |
| interaction, the validation or PASS/FAIL conditions should still be |
| clear enough as to allow automatic validation if said interaction is |
| later automated. |
| </p> |
| </section> |
| </section> |
| <section> |
| <h2>Risks and mitigation</h2> |
| <p> |
| There are a number of risks associated with creating a high-quality |
| test suite for CSS Regions. The most important ones are listed below. |
| </p> |
| <section> |
| <h3>Implicit interactions with a lot of other specs (and commonly accepted |
| browser behavior)</h3> |
| <p> |
| The CSS Region spec introduces a significant change in the way layout |
| can be done and as a consequence, many of the assumptions that hold in |
| the context of CSS 2.1 must be re-validated in the context of CSS |
| Regions. In the meanwhile, a lot of new layout modules have been |
| proposed, with various degrees of implementor support and maturity. |
| Also, non-CSS specific specs introduced concepts that affect how style |
| is propagated (e.g. Shadow DOM [[shadow-dom]]) or how elements are |
| rendered and interact with their containing documents (e.g. |
| <code><iframe seamless></code> in HTML 5 [[HTML5]]). |
| </p> |
| <p> |
| All the above factors increase the testing surface and the number of |
| the possible cases that might need an explicitly specified behavior in |
| order to ensure intuitive and predictable results as well as stable |
| interaction with widely used browser features (that might not be <em>yet</em> |
| so strictly specified). |
| </p> |
| <p> |
| In terms of specifying the expected behavior, the current approach is |
| to try and specify it for specs that are final or nearly-final and to |
| just make a note of the possible interactions and unspecified behaviors |
| in the case of specs that are still in flux. In exceptional cases, a |
| new spec might be created to cater for the needs of multiple specs |
| (e.g. the CSS Fragmentation spec [[css3-break]]). |
| </p> |
| </section> |
| <section> |
| <h3>Big number of tests required</h3> |
| <div class="note"> |
| Add here the estimation of tests required produced by |
| <a href="https://github.com/w3c/web-platform-tests/tree/master/tools/coverage" target="_blank">W3C test coverage</a> |
| tool. If possible, provide a better informed guess on the number of tests. |
| </div> |
| <p> |
| Given the complexity of the spec, a big number of tests will need to |
| be created to produce a test suite that can ensure interoperability |
| between implementations. |
| </p> |
| <p> |
| In this context, the main purpose of this document is to provide |
| useful informations for creating and contributing tests in an effective |
| manner in terms of coverage and test quality. |
| </p> |
| </section> |
| </section> |
| <section> |
| <h2>Approach</h2> |
| <p> |
| As spec testing cannot be realistically separated from testing a |
| particular implementation (except for the very simple cases), the |
| approach proposed for testing is one that tries to first cover as many |
| areas as possible, instead of deep diving on a certain feature or aspect |
| of the spec first. A side benefit of this approach is that the spec |
| tests can be used at any time to gauge the level of support of a certain |
| implementation. |
| </p> |
| <p> |
| Having this <em>breadth-first</em> approach in mind, tests will be |
| created for the testing areas listed in <a href="#testing-areas" |
| class="sectionRef"></a>. Testing will be done in multiple passes, each |
| aimed at covering more specific edge-cases. |
| </p> |
| </section> |
| <section> |
| <h2>Testing areas</h2> |
| <section> |
| <h3>Explicit testing areas</h3> |
| <p> |
| These are testing areas normatively defined by the spec. They cover |
| things explicitly or implicitly defined in the CSS Regions spec. |
| Please note that while detailed, this list is not necessarily |
| exhaustive and normative behaviors may not be contained in it. |
| When in doubt, consult the CSS Regions spec or ask a question on the |
| <a href="http://lists.w3.org/Archives/Public/www-style/">mailing |
| list</a>. |
| </p> |
| <p> |
| Below is the list of explicit testing areas: |
| <ol> |
| <li> |
| Proper parsing of the CSS properties and rules, rendering |
| according to the spec. |
| <ul> |
| <li><code>flow-into</code> and <code>flow-from</code>; |
| <code>flow-into</code> should be tested both with and without |
| the <code>content</code> switch</li> |
| <li><code>region-fragment</code></li> |
| <li>fragmentation additions: <code>region-avoid</code>, |
| <code>region-always</code> – see [[!css3-break]] for |
| details</li> |
| </ul> |
| </li> |
| <li>Nested named flows and cycle detection</li> |
| <li>Multi-column regions</li> |
| <li>Regions' pseudo-elements</li> |
| <li>Region auto-sizing (regions visual formatting details)</li> |
| <li> |
| CSSOM - interfaces implemented, behavior according to the spec |
| <div class="note"> |
| <code>regionfragmentchange</code> might be removed from this level of the spec. |
| Adjust the contents of the list below accordingly. |
| </div> |
| <ul> |
| <li> |
| <code>NamedFlow</code> interface |
| </li> |
| <li> |
| <code>Region</code> interface |
| </li> |
| <li> |
| <code>regionoversetchange</code> event |
| </li> |
| <li> |
| <code>regionfragmentchange</code> event |
| </li> |
| </ul> |
| </li> |
| <li> |
| DOM additions |
| <ul> |
| <li> |
| <code>getClientRects()</code> and |
| <code>getBoundingClientRects()</code></li> |
| <li> |
| <code>offsetTop</code>, <code>offsetBottom</code>, |
| <code>offsetLeft</code>, <code>offsetRight</code> and |
| <code>offsetParent</code> |
| </li> |
| </ul> |
| </li> |
| </ol> |
| </p> |
| <div class="note"> |
| Region styling has been removed from the spec waiting for it to be defined in the page |
| styling spec. This is a reminder for adding section here related to page/region styling |
| once this gets specified. |
| </div> |
| </section> |
| <section> |
| <h3>Implicit testing areas</h3> |
| <p> |
| These are testing areas either normatively defined in other specs |
| that explicitly refer to the CSS Regions spec (e.g. [[!css3-break]]) |
| or simply not explicitly defined, but implied by various aspects of |
| the spec (e.g. processing model, CSS 2.1 compliance, etc.). |
| Please note that while detailed, this list is not necessarily |
| exhaustive and normative behaviors may not be contained in it. |
| When in doubt, consult the CSS Regions spec or ask a question on the |
| <a href="http://lists.w3.org/Archives/Public/www-style/">mailing |
| list</a>. |
| </p> |
| <p> |
| Below is the list of implicit testing areas: |
| <ol> |
| <li> |
| CSS Regions and layout modules: |
| <ul> |
| <li>Floats</li> |
| <li>Positioned elements</li> |
| <li>Intrinsic and extrinsic sizing [[css3-sizing]] and auto-sized regions</li> |
| <li>CSS Flexbox [[css3-flexbox]]</li> |
| <li>CSS Multicolumn [[css3-multicol]] content in named flows</li> |
| <li>CSS Shapes [[css-shapes-1]]</li> |
| <li>CSS Grid Layout [[css3-grid-layout]]</li> |
| </ul> |
| </li> |
| <li> |
| CSS Regions and different types of elements |
| <ul> |
| <li>List items</li> |
| <li> |
| <code><iframe></code> |
| <ul> |
| <li>fragmentation of <code><iframe></code> and |
| <code><iframe seamless></code></li> |
| </ul> |
| </li> |
| <li><code><video></code></li> |
| <li><code><canvas></code></li> |
| <li><code><table></code></li> |
| </ul> |
| </li> |
| <li>Stacking contexts</li> |
| <li> |
| Fragmentation |
| <ul> |
| <li>Named flow content fragmentation</li> |
| <li>Regions fragmentation</li> |
| </ul> |
| </li> |
| <li> |
| Dynamic content |
| <ul> |
| <li>Changing the content of the named flows or of the region chain at runtime</li> |
| <li>Interactive content: <code>contentEditable</code>, <code>designMode</code> and |
| input elements</li> |
| </ul> |
| </li> |
| <li>User input events (e.g. mouse, touch, keyboard); this includes event propagation</li> |
| <li>User action pseudo-classes (e.g. <code>:active</code>, <code>:hover</code>)</li> |
| <li>Writing modes</li> |
| <li>Transforms</li> |
| <li>Transitions and animations</li> |
| <li> |
| Shadow DOM |
| <ul> |
| <li>Regions inside shadow trees</li> |
| <li>Flowing content from shadow trees</li> |
| </ul> |
| </li> |
| <li>Content clipping and overflowing</li> |
| <li>Pseudo-elements in named flow contents</li> |
| </ol> |
| </p> |
| <div class="note"> |
| Region styling has been removed from the spec waiting for it to be defined in the page |
| styling spec. This is a reminder for adding section here related to page/region styling |
| interacting with <code><style scoped></code> once this gets specified. |
| </div> |
| </section> |
| </section> |
| <section> |
| <h2>People and responsibilities</h2> |
| <p> |
| Below is a list of people you should reach out to if you have any |
| questions related to this document or testing CSS Regions in general: |
| <ul> |
| <li>Alan Stearns – editor and test coordinator for CSS Regions spec</li> |
| <li>Rebecca Hauck – CSSWG testing owner</li> |
| </ul> |
| </p> |
| </section> |
| <section> |
| <h2>Test progress tracking</h2> |
| <p> |
| Currently test progress tracking is done via gitHub |
| <a href="https://github.com/w3c/csswg-test/issues/milestones?with_issues=yes">milestones</a> |
| and <a href="https://github.com/w3c/csswg-test/issues?milestone=9&state=open">issues</a>. |
| </p> |
| </section> |
| </body> |
| </html> |