Some Blink features cannot be automatically tested using the Web Platform. Prime examples are the APIs that require user activation (also known as a user gesture), such as Full Screen. Automated tests for these Blink features must rely on special APIs, which are only exposed in testing environments, and are therefore not available in a normal browser session.
A popular pattern used in these tests is to rely on the user to perform some manual steps in order to run the test case in a normal browser session. These tests are effectively manual tests, with additional JavaScript code that automatically performs the desired manual steps, when loaded in an environment that exposes the needed testing APIs.
Layout tests that degrade to manual tests in the absence of testing APIs have the following benefits.
Therefore, the desirability of adding a manual fallback to a test heavily depends on whether the feature under test is a Web Platform feature or a Blink-only feature, and on the developer's working style. The benefits above should be weighed against the added design effort needed to build a manual test, and the size and complexity introduced by the manual fallback.
A natural workflow for writing a layout test that gracefully degrades to a manual test is to first develop the manual test in a browser, and then add code that feature-checks for testing APIs, and uses them to automate the test's manual steps.
Manual tests should minimize the chance of user error. This implies keeping the manual steps to a minimum, and having simple and clear instructions that describe all the configuration changes and user gestures that match the effect of the Blink-specific APIs used by the test.
Below is an example of a fairly minimal test that uses a Blink-Specific API (window.eventSender
), and gracefully degrades to a manual test.
<!doctype html> <meta charset="utf-8"> <title>DOM: Event.isTrusted for UI events</title> <link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted"> <link rel="help" href="https://dom.spec.whatwg.org/#constructing-events"> <meta name="assert" content="Event.isTrusted is true for events generated by user interaction"> <script src="../../resources/testharness.js"></script> <script src="../../resources/testharnessreport.js"></script> <p>Please click on the button below.</p> <button>Click Me!</button> <script> 'use strict'; setup({ explicit_timeout: true }); promise_test(() => { const button = document.querySelector('button'); return new Promise((resolve, reject) => { const button = document.querySelector('button'); button.addEventListener('click', (event) => { resolve(event); }); if (window.eventSender) { eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop); eventSender.mouseDown(); eventSender.mouseUp(); } }).then((clickEvent) => { assert_true(clickEvent.isTrusted); }); }, 'Click generated by user interaction'); </script>
The test exhibits the following desirable features:
<link rel="help">
), because the paragraph that documents the tested feature (referenced by the primary URL) is not very informative on its own.<p>
) that tells the tester exactly what to do, and the <button>
that needs to be clicked is clearly labeled.testharness.js
by calling setup({ explicit_timeout: true });
window.eventSender
) before invoking them. The test does not automatically fail when the APIs are not present.Notice that the test is pretty heavy compared to a minimal JavaScript test that does not rely on testing APIs. Only use testing APIs when the desired testing conditions cannot be set up using Web Platform APIs.