The script gentest.sh is used to generate canvas WPT tests, found under wpt/html/canvas.
Generating tests for the canvas API has multiple advantages. It allows generating lots of tests with minimal boilerplate and configuration. In particular:
Canvas tests all have common boilerplate, like defining a whole HTML page, creating a canvas and reading back pixels. The code we care about is usually only a few lines of JavaScript. By using a test generator, we can write tests focussing on these few relevant lines, abstracting away all of the boilerplate needed to run these lines.
Canvas exists in multiple flavors (HTMLCanvasElement, OffscreenCanvas) and can run in different environments (main thread, worker). Using a code generator allows tests to be implemented only once and run then in all the flavors or environment we need test coverage for.
Canvas rendering can be affected by a large number of states. Implementations can have different code paths for different permutations of these states. For instance, simply testing that a rectangle is correctly drawn requires validating different permutations of whether the canvas has an alpha channel, whether layers are used, whether the context uses a globalAlpha, which globalCompositeOperation is used, which filters are used, whether shadows are enabled, and so on. Bugs occurring only for some specific combinations of these states have been found. A test generator allows for easily creating a large number of tests, or tests validating a large number of variant permutations, all with minimal boilerplate.
You can generate canvas tests by running wpt update-built --include canvas, or by running gentest.sh directly:
Make a python virtual environment somewhere (it doesn't matter where):
python3 -m venv venv
Enter the virtual environment:
source venv/bin/activate
This script depends on the cairocffi, jinja2 and pyyaml Python packages. You can install them using requirements_build.txt:
python3 -m pip install -r tools/ci/requirements_build.txt
Change to the directory with this script and run it:
python3 gentest.py
See WPT documentation for the current minimal Python version required. If you modify gentest.py, it‘s recommended to use that exact Python version to avoid accidentally using new Python features that aren’t be supported by that minimal version. pyenv can be used instead of the venv approach described above, to pin the html/canvas/tools folder to that exact Python version, without impacting the rest of the system. For instance:
pyenv install 3.8 cd html/canvas/tools pyenv local 3.8 python3 -m pip install -r $WPT_CHECKOUT/tools/ci/requirements_build.txt python3 gentest.py
The tests are defined in YAML files, found in wpt/html/canvas/tools/yaml. The YAML definition files consists of a sequence of dictionaries, each with at a minimum the keys name: and code:. For instance:
- name: 2d.sample.draws-red code: | ctx.fillStyle = 'red'; ctx.fillRect(0, 0, 10, 10); @assert pixel 5,5 == 255,0,0,255; - name: 2d.sample.draws-green code: | ctx.fillStyle = 'green'; ctx.fillRect(0, 0, 10, 10); @assert pixel 5,5 == 0,255,0,255;
From this configuration, the test generator would produce multiple test files and fill-in the boilerplate needed to run these JavaScript lines.
See the constants _TEST_DEFINITION_PARAMS and _GENERATED_PARAMS in the gentest.sh for a full list and description of the available parameters.
The test generator uses Jinja templates to generate the different test files it produces. The templates can be found under wpt/html/canvas/tools/templates. When rendering templates, Jinja uses a dictionary of parameters to lookup variables referred to by the template. In the test generator, this dictionary is actually the YAML dictionary defining the test itself.
Take for instance the test:
- name: 2d.sample.draws-red code: | ctx.fillStyle = 'red'; ctx.fillRect(0, 0, 10, 10); @assert pixel 5,5 == 255,0,0,255;
In the template .../templates/testharness_element.html, the title of the generated HTML is defined as:
<title>Canvas test: {{ name }}</title>
When rendering this template, Jinja looks-up the name: key from the YAML test definition, which in the example above would be 2d.sample.draws-red, producing this HTML result:
<title>Canvas test: 2d.sample.draws-red</title>
Now, more interestingly, all the parameter values in the test definition are also Jinja templates. They get rendered on demand, before being used by Jinja into another template. Since all of these use the test's YAML definition as param dictionary, test parameters can refer to each others:
- name: 2d.sample.draws-red expected_color: 255,0,0,{{ alpha }} alpha: 255 code: | ctx.fillStyle = 'red'; ctx.fillRect(0, 0, 10, 10); @assert pixel 5,5 == {{ expected_color }};
All the test parameters are also registered as templates loadable from other Jinja templates, with {% import ... %} statements for instance. This can be useful to organize the test definition and allow reuse of Jinja statements. For instance:
- name: 2d.sample.macros macros: | {% macro rgba_format(color) %} {% set r, g, b, a = color -%} rgba{{ (r, g, b, a) -}} {% endmacro %} {% macro assert_format(color) %} {% set r, g, b, a = color -%} {{- '%d,%d,%d,%d' | format(r, g, b, a * 255) -}} {% endmacro %} code: | {% import 'macros' as m %} ctx.fillStyle = '{{ m.rgba_format(color) }}'; ctx.fillRect(0, 0, 10, 10); @assert pixel 5,5 == {{ m.assert_format(color) }}; color: [64, 128, 192, 1.0]
These types of parameterization might seem strange and overkill in toy examples like these, but it's in fact really useful when using the variants: feature (more on this below).
By default, the generator produces three flavors of each tests, one for each of three different canvas types:
HTMLCanvasElement.OffscreenCanvas, used in a main thread script.OffscreenCanvas, used in a worker.HTMLCanvasElement tests get generated into the folder .../canvas/element, while the main thread and worker OffscreenCanvas tests are generated in the folder .../canvas/offscreen.
Some tests are specific to certain canvas types. The canvas types to be generated can be specified by setting the canvas_types: config to a list with one or many of the following strings:
'HtmlCanvas''OffscreenCanvas''Worker'For instance:
- name: 2d.sample.offscreen-specific canvas_types: ['OffscreenCanvas', 'Worker'] code: | assert_not_equals(canvas.convertToBlob(), null);
The test generator can generate both JavaScript tests (testharness.js), or Reftests. By default, the generator produces JavaScript tests. These are implemented with the testharness.js library. Assertions must be used to determine whether they succeed. Standard assertions provided by testharness.js can be used, like assert_true, assert_equals, etc.
Canvas tests also have access to additional assertion types and other helpers defined in wpt/html/canvas/resources/canvas-tests.js. Most of these however are private and meant to be used via macros provided by this test generator (denoted by the character “@”). Note that these “@” macros are implemented as regexp-replace, so their syntax is very strict (e.g. they don't tolerate extra whitespaces and some reserve ; as terminating character).
@assert pixel x,y == r,g,b,a;
Asserts that the color at the pixel position [x, y] exactly equals the RGBA values [r, g, b, a].
@assert pixel x,y ==~ r,g,b,a;
Asserts that the color at the pixel position [x, y] approximately equals the RGBA values [r, g, b, a], within +/- 2.
@assert pixel x,y ==~ r,g,b,a +/- t;
Asserts that the color at the pixel position [x, y] approximately equals the RGBA values [r, g, b, a], within +/- t for each individual channel.
@assert throws *_ERR code;
Shorthand for assert_throws_dom, running code and verifying that it throws a DOM exception *_ERR (e.g. INDEX_SIZE_ERR).
@assert throws *Error code;
Shorthand for assert_throws_js, running code and verifying that it throws a JavaScript exception *Error (e.g. TypeError).
@assert actual === expected;
Shorthand for assert_equals, asserting that actual is the same as expected.
@assert actual !== expected;
Shorthand for assert_not_equals, asserting that actual is different than expected.
@assert actual =~ expected;
Shorthand for assert_regexp_match, asserting that actual matches the regular expression expected.
@assert cond;
Shorthand for assert_true, but evaluating cond as a boolean by prefixing it with !!.
testharness.js allows the creation of synchronous, asynchronous or promise tests (see here for details).
To choose what test types to generate, set the test_type parameter to one of:
syncasyncpromiseFor instance, a synchronous test would use test_type: sync:
- name: 2d.sample.sync-test desc: Example synchronous test canvas_types: ['HtmlCanvas'] test_type: sync code: | assert_regexp_match(canvas.toDataURL(), /^data:/);
Given this config, the code generator would generate an HTMLCanvasElement test with the following <script> tag (not showing the rest of the HTML file).
test(t => { var canvas = document.getElementById('c'); var ctx = canvas.getContext('2d'); assert_regexp_match(canvas.toDataURL(), /^data:/); }, "Example synchronous test");
To test asynchronous code, test_type: async can be use as in:
- name: 2d.sample.async-test desc: Example asynchronous test canvas_types: ['HtmlCanvas'] test_type: async code: | canvas.toBlob(t.step_func_done(blob => { assert_greater_than(blob.size, 0); }));
async_test(t => { var canvas = document.getElementById('c'); var ctx = canvas.getContext('2d'); canvas.toBlob(t.step_func_done(blob => { assert_greater_than(blob.size, 0); })); }, "Example asynchronous test");
Promise-based APIs would use test_type: promise, for instance:
- name: 2d.sample.promise-test desc: Example promise test canvas_types: ['OffscreenCanvas'] test_type: promise code: | const blob = await canvas.convertToBlob(); assert_greater_than(blob.size, 0);
promise_test(async t => { var canvas = new OffscreenCanvas(100, 50); var ctx = canvas.getContext('2d'); const blob = await canvas.convertToBlob(); assert_greater_than(blob.size, 0); }, "Example promise test");
To maintain compatibility with old tests (until they are updated), the test generator will use a legacy test harness if the test_type is omitted. This test harness is a quirky hybrid between sync and async tests. These tests are implemented with an async_test() fixture, but the generator automatically invokes t.done() after the body, making is behave like a synchronous test. To implement actual async test, the test body must call deferTest(); and manually call t.done() to finish the test. Newer test should prefer specifying test_type and using standard testharness.js APIs instead of these canvas-specific ones.
The code generator can also be used to generate reftests. These tests are comprised of a test HTML page and a reference HTML page. Both are rendered by the test runner and the results are compared pixel by pixel. These tests do not use testharness.js and thus cannot use assertions.
To write a reftest, use a reference: key in the YAML config:
- name: 2d.sample.reftest desc: Example reftest code: | ctx.scale(2, 2); ctx.fillRect(5, 5, 15, 15); reference: | ctx.fillRect(10, 10, 30, 30);
This will produce a test file 2d.sample.reftest.html and a reference file 2d.sample.reftest-expected.html, for HTMLCanvasElement and main thread/worker OffscreenCanvas.
By default, the test will fail if the test runner sees any rendering difference between the test and reference page. If some difference is expected, fuzzy-matching can be used by using the fuzzy: config, as in:
- name: 2d.sample.reftest.fuzzy desc: Example reftest using fuzzy matching fuzzy: maxDifference=0-1; totalPixels=0-100 code: | ctx.fillStyle = 'rgba(0, 255, 0, 0.5)'; ctx.fillRect(5, 5, 10, 10); reference: | ctx.fillStyle = 'rgba(128, 255, 128, 1)'; // Should it be 127 or 128? ctx.fillRect(5, 5, 10, 10);
The code generator supports three types of reference files that can be used to validated the test results.
reference: generates a reference file using JavaScript code writing to a canvas, similarly to what the code: block of the test does. For instance:- name: 2d.sample.reftest.reference desc: Reftest comparing one canvas drawing with another. code: | ctx.scale(2, 2); ctx.fillRect(5, 5, 15, 15); reference: | ctx.fillRect(10, 10, 30, 30);
html_reference: can be useful to compare canvas rendering against HTML+CSS, or against SVG. For instance:- name: 2d.sample.reftest.html_reference desc: Example reftest using an html_reference fuzzy: maxDifference=0-1; totalPixels=0-24 code: | ctx.filter = 'blur(5px)'; ctx.fillRect(5, 5, 10, 10); html_reference: | <svg xmlns="https://www.w3.org/2000/svg" width="{{ size[0] }}" height="{{ size[1] }}"> <filter id="filter" x="-100%" y="-100%" width="300%" height="300%"> <feGaussianBlur stdDeviation="5" /> </filter> <g filter="url(#filter)"> <rect x="5" y="5" width="10" height="10"></rect> </g> </svg>
cairo_reference: produces a reftest using a reference image generated from Python code using the pycairo library. The variable cr provides a cairo.Context instance that can be used to draw the reference image. For instance:- name: 2d.sample.reftest.cairo_reference desc: Example reftest using a cairo_reference code: | ctx.fillStyle = 'rgb(255, 128, 64)'; ctx.fillRect(5, 5, 50, 30); ctx.globalCompositeOperation = 'color-dodge'; ctx.fillStyle = 'rgb(128, 200, 128)'; ctx.fillRect(15, 15, 50, 30); cairo_reference: | cr.set_source_rgb(255/255, 128/255, 64/255) cr.rectangle(5, 5, 50, 30) cr.fill() cr.set_operator(cairo.OPERATOR_COLOR_DODGE) cr.set_source_rgb(128/255, 200/255, 128/255) cr.rectangle(15, 15, 50, 30) cr.fill()
img_reference: produces a reftest using a pre-generated image file. This can be useful for trivial images (e.g. a solid color), but avoid using this for non-trivial images as it's important to be able to know how these images are generated, so we could inspect or modify them later if needed. Example:- name: 2d.sample.reftest.img_reference desc: Example reftest using an img_reference size: [100, 50] code: | ctx.fillStyle = 'rgb(0, 255, 0)'; ctx.fillRect(0, 0, {{ size[0] }}, {{ size[1] }}); img_reference: /images/green-100x50.png
As explained in the WPT documentation, the test runner screenshots reftests after the page is loaded and pending paints are completed. To support asynchronous calls or promises, we must let the test runner know when the test is done. This can be done by using test_type: promise, for instance:
- name: 2d.sample.reftest.promise desc: Example of a reftest using promises test_type: promise code: | ctx.beginLayer(); ctx.fillRect(5, 5, 10, 10); // Checks that layers survive frame boundaries. await new Promise(resolve => requestAnimationFrame(resolve)); ctx.endLayer(); reference: ctx.fillRect(5, 5, 10, 10);
Test parameterization is a very useful tool, allowing a test to be exercised on a number of different inputs with minimal boilerplate. The test generator supports this via the variants: parameter. Let's begin by showing a motivating example. Say you would like to compare canvas rendering with SVG. You might want to implement these two tests:
- name: 2d.compare-canvas-and-svg.draws-red color: 'red' code: | ctx.fillStyle = '{{ color }}'; ctx.fillRect(0, 0, 10, 10); html_reference: | <svg xmlns="https://www.w3.org/2000/svg" width="{{ size[0] }}" height="{{ size[1] }}"> <rect x="0" y="0" width="10" height="10" fill="{{ color }}"/> </svg> - name: 2d.compare-canvas-and-svg.draws-green color: 'green' code: | ctx.fillStyle = '{{ color }}'; ctx.fillRect(0, 0, 10, 10); html_reference: | <svg xmlns="https://www.w3.org/2000/svg" width="{{ size[0] }}" height="{{ size[1] }}"> <rect x="0" y="0" width="10" height="10" fill="{{ color }}"/> </svg>
This works, but there's a lot of duplication and it would not scale for testing with a larger number of inputs. You could cut down on the duplication by using YAML & anchors and * aliases, for instance:
- name: 2d.compare-canvas-and-svg.draws-red fill_style: 'red' code: &draw-rect-with-canvas | ctx.fillStyle = '{{ fill_style }}'; ctx.fillRect(0, 0, 10, 10); html_reference: &draw-rect-with-svg | <svg xmlns="https://www.w3.org/2000/svg" width="{{ size[0] }}" height="{{ size[1] }}"> <rect x="0" y="0" width="10" height="10" fill="{{ fill_style }}"/> </svg> - name: 2d.compare-canvas-and-svg.draws-green fill_style: 'green' code: *draw-rect-with-canvas html_reference: *draw-rect-with-svg
This however adds indirections making the test hard to read. Test must still be duplicated, but they now mostly contain boilerplate and this would still not scale for a larger number of test inputs. With variants, you could write the test like this:
- name: 2d.compare-canvas-and-svg code: | ctx.fillStyle = '{{ fill_style }}'; ctx.fillRect(0, 0, 10, 10); html_reference: | <svg xmlns="https://www.w3.org/2000/svg" width="{{ size[0] }}" height="{{ size[1] }}"> <rect x="0" y="0" width="10" height="10" fill="{{ fill_style }}"/> </svg> variants: - draws-red: {fill_style: red} draws-green: {fill_style: green}
Now, this has minimal boilerplate and easily scales to many more test inputs.
Variant are defined as a list of dictionaries of dictionaries. In the above examples, the - before draws-red: denotes a list item, which is a dictionary with two keys: draws-red and draws-green. That dictionary defines a variant dimension. A test file will be generated for each items in this dictionary. The draws-red and draws-green strings are the variant names, which are appended to the generated test name. The value associated with these keys (for instance {fill_style: red}) are parameters that get merged over the base test parameters for that test instance.
If the variants: list has more than one dictionary, the test generator will generate tests with the cross product of all variant dimensions. For instance, this config:
- name: 2d.grid code: ctx.{{ variant_names[0] }} = '{{ color }}'; variants: - fillStyle: shadowColor: - red: {color: '#F00'} blue: {color: '#00F'}
Will generate:
2d.grid.fillStyle.red, with code: ctx.fillStyle = '#F00';2d.grid.fillStyle.blue, with code: ctx.fillStyle = '#00F';2d.grid.shadowColor.red, with code: ctx.shadowColor = '#F00';2d.grid.shadowColor.blue, with code: ctx.shadowColor = '#00F';This last example uses the variable {{ variant_names[0] }} to avoid having to write fillStyle and shadowColor twice, once in the variant name and once on the variant params. variant_names is a special variable which holds a list of the variant names for that particular test instance. When generating the test 2d.grid.fillStyle.red for instance, variant_names would be set to ['fillStyle', 'red'].
Variants are really useful to easily validate many test inputs, but this can lead to a large number of test files being generated. For instance, the 2d canvas compositing pipeline is affected by these states (and more):
fillRect, drawImage, CanvasPattern.1.0 or non 1.0.globalCompositeOperationIn practice, we have found bugs in the canvas what only happen for very specific combinations of states, like, using a specific globalCompositeOperation, with transforms and shadows enabled, but only when drawImage is used. We might want be thorough and test as many combinations of these states as we can, but testing all permutations would require $322622223 = 7488$ tests. Putting all of these in different files would be unmanageable. Comes variant grids to the rescue! If we generated test files each testing all composite operations in a grid, we'd only need a much more reasonable $322222*3 = 288$ files.
To generate a variant grid, use the variants_layout: config. This has to be a list of the same lengths as the variants: list (as long as there are variant dimensions). For each dimension, variants_layout: must hold either the string multi_files or single_file. Variant dimensions marked as single_file will be expanded in the same file. For instance:
- name: grid-example variants: - A1: A2: - B1: B2: - C1: C2: - D1: D2: - E1: E2: variants_layout: - single_file - multi_files - single_file - multi_files - single_file
Because this test has 2 multi_files dimension with two variants each, 4 files would be generated:
Then, the 3 single_file dimensions would produce $222 = 8$ tests in each of these files. For JavaScript tests, each of these test would be generated in sequence, each with their own test(), async_test() or promise_test() fixture. Reftest on the other hand would produce a 2x2x2 grid, as follows:
A1.C1.E1 A2.C1.E1 A1.C2.E1 A2.C2.E1 A1.C1.E2 A2.C1.E2 A1.C2.E2 A2.C2.E2