Testing in Chromium

Web Tests with Manual Fallback

Some Blink features cannot be automatically tested using the Web Platform. Prime examples are the APIs that require user activation (also known as a user gesture), such as Full Screen. Automated tests for these Blink features must rely on special APIs, which are only exposed in testing environments, and are therefore not available in a normal browser session.

A popular pattern used in these tests is to rely on the user to perform some manual steps in order to run the test case in a normal browser session. These tests are effectively manual tests, with additional JavaScript code that automatically performs the desired manual steps, when loaded in an environment that exposes the needed testing APIs.

Motivation

Web tests that degrade to manual tests in the absence of testing APIs have the following benefits.

  • The manual test component can be debugged in a normal browser session, using the rich developer tools. Tests without a manual fallback can only be debugged in the test runner.
  • The manual tests can run in other browsers, making it easy to check whether our behavior matches other browsers.
  • The web tests can form the basis for manual tests that are contributed to web-platform-tests.

Therefore, the desirability of adding a manual fallback to a test heavily depends on whether the feature under test is a Web Platform feature or a Blink-only feature, and on the developer's working style. The benefits above should be weighed against the added design effort needed to build a manual test, and the size and complexity introduced by the manual fallback.

Development Tips

A natural workflow for writing a web test that gracefully degrades to a manual test is to first develop the manual test in a browser, and then add code that feature-checks for testing APIs, and uses them to automate the test's manual steps.

Manual tests should minimize the chance of user error. This implies keeping the manual steps to a minimum, and having simple and clear instructions that describe all the configuration changes and user gestures that match the effect of the Blink-specific APIs used by the test.

Example

Below is an example of a fairly minimal test that uses a Blink-Specific API (window.eventSender), and gracefully degrades to a manual test.

<!doctype html>
<meta charset="utf-8">
<title>DOM: Event.isTrusted for UI events</title>
<link rel="help" href="https://dom.spec.whatwg.org/#dom-event-istrusted">
<link rel="help" href="https://dom.spec.whatwg.org/#constructing-events">
<meta name="assert"
    content="Event.isTrusted is true for events generated by user interaction">
<script src="../../resources/testharness.js"></script>
<script src="../../resources/testharnessreport.js"></script>

<p>Please click on the button below.</p>
<button>Click Me!</button>

<script>
'use strict';

setup({ explicit_timeout: true });

promise_test(() => {
  const button = document.querySelector('button');
  return new Promise((resolve, reject) => {
    const button = document.querySelector('button');
    button.addEventListener('click', (event) => {
      resolve(event);
    });

    if (window.eventSender) {
      eventSender.mouseMoveTo(button.offsetLeft, button.offsetTop);
      eventSender.mouseDown();
      eventSender.mouseUp();
    }
  }).then((clickEvent) => {
    assert_true(clickEvent.isTrusted);
  });

}, 'Click generated by user interaction');

</script>

The test exhibits the following desirable features:

  • It has a second specification URL (<link rel="help">), because the paragraph that documents the tested feature (referenced by the primary URL) is not very informative on its own.
  • It links to the WHATWG Living Standard, rather than to a frozen version of the specification.
  • It contains clear instructions for manually triggering the test conditions. The test starts with a paragraph (<p>) that tells the tester exactly what to do, and the <button> that needs to be clicked is clearly labeled.
  • It disables the timeout mechanism built into testharness.js by calling setup({ explicit_timeout: true });
  • It checks for the presence of the Blink-specific testing APIs (window.eventSender) before invoking them. The test does not automatically fail when the APIs are not present.
  • It uses Promises to separate the test setup from the assertions. This is particularly helpful for manual tests that depend on a sequence of events to occur, as Promises offer a composable way to express waiting for asynchronous events that avoids callback hell.

Notice that the test is pretty heavy compared to a minimal JavaScript test that does not rely on testing APIs. Only use testing APIs when the desired testing conditions cannot be set up using Web Platform APIs.