There are several types of dump accessibility tests:
tree tests to test accessible trees;node tests to test a single accessible node;script tests to run a script and compare its output to expected results;event tests to test accessible events.All these tests are backed by the same engine and linked with the same idea: a test generates an output and then the output gets compared to expected results. Such approach has a great benefit: the tests can be rebaselined easily by making a test to generate expectations itself.
Tree tests are designed to test accessible tree. It loads an HTML file, waits for it to load, then dumps the accessible tree. The dumped tree is compared to an expectation file. The tests are driven by DumpAccessibilityTreeTest testing class.
Node tests are used to run a test for a single node, for example, to check a specific property. The test loads an HTML file, waits for it to load, then dump a single accessible node for a DOM element whose id or class attribute is test. There is no support for multiple “test” nodes and the output will be for the first match located. The tests are driven by DumpAccessibilityNodeTest testing class.
Script tests are used to run a script and test its output against expectations. The tests is driven by DumpAccessibilityScriptTest testing class.
Event tests tests use a similar format but the events are dumped after the document finishes loading, and an optional go() function runs. See more on this below.
Each test is parameterized to run multiple times. Most platforms dump in the “blink” format (the internal data), and again in a “native” (platform-specific) format. The Windows platform has two native formats, “uia” and “ia2”, so the test is run three times. The test name indicates which test pass was run, e.g., DumpAccessibilityTreeTest.TestName/blink. (Note: for easier identification, the Windows, Mac, Linux, and Android platforms rename the “native” pass to “ia2”, “mac”, “linux” and “android”, respectively.)
The test output is a compact text representation of the accessible node(s) for that format, and it should be familiar if you‘re familiar with the accessibility protocol on that platform, but it’s not in a standardized format - it's just a text dump, meant to be compared to expected output.
The exact output can be filtered so it only dumps the specific attributes you care about for a specific test.
Once the output has been generated, it compares the output to an expectation file in the same directory. If an expectation file for that test for that platform is present, it must match exactly or the test fails. If no expectation file is present, the test passes. Most tests don't have expectations on all platforms.
Most of the tests are hosted by content_browsertests testsuite.
autoninja -C out/Debug content_browsertests out/Debug/content_browsertests --gtest_filter="All/DumpAccessibility*"
PDF accessibility tests though are running under browsertest testsuite.
autoninja -C out/Default browser_tests out/Default/browser_tests --gtest_filter="PDFExtensionAccessibilityTreeDumpTest*"
foo.html -- a file to be testedfoo-expected-[platform].txt -- a file to be testedfoo-node-expected-[platform].txt - a a file containing a node test expectationsSupported platforms are:
android -- expected Android AccessibilityNodeInfo outputauralinux -- expected Linux ATK outputauralinux-xenial -- expected Linux ATK output (Version Specific Expected File)blink -- representation of internal accessibility treeblink-cros -- representation of internal accessibility tree (Version Specific Expected File for Chrome OS and Lacros)mac -- expected Mac NSAccessibility outputwin -- expected Win IAccessible/IAccessible2 outputuia-win -- expected Win UIA outputuia-win7 -- expected Win7 UIA output (Version Specific Expected File)Note, a single HTML test files can be used both for tree tests and node tests. In this case the expectations files for node tests portion will have an expectations-file qualifier of -node inserted immediately before expected. Thus for foo.html, there could be:
foo-expected-mac.txt -- expected Mac NSAccessibility output for the entire accessibility treefoo-node-expected-mac.txt -- expected Mac NSAccessibility output for just the node in foo.html whose class is test# are ignored#<skip then the test passes. This can be used to indicate desired output with a link to a bug, or as a way to temporarily disable a test during refactoring.UIA sometimes differs between windows 7 and later versions of Windows. To account for these differences, the UIA accessibility tree formatter will look for a version specific expected file first: foo-expected-uia-win7.txt. If the version specific expected file does not exist, the normal expected file will be used instead: “foo-expected-uia-win.txt”. There is no concept of version specific filters.
In the case of Linux, the tests are run on several LTS releases of Ubuntu:
In many cases the expected results for foo.html will be the same for all versions of Ubuntu, in which case foo-expected-auralinux.txt is all that is needed. However, if the foo.html test passes on the Linux release build (“linux-rel”), but fails on “linux-xenial-rel”, you will need an additional foo-expected-auralinux-xenial.txt file.
At the present time there is no version-specific support for Bionic Beaver, which is the current version run on “linux-rel”.
The need for a version-specific expectations file on Chrome OS / Lacros is extremely rare. However, there can be occasional differences in the internal accessibility tree. For instance, the SVG g element is always included in order to support select-to-speak functionality. If foo.html has a foo-expected-blink.txt file which works on all platforms except the Chrome OS and Lacros bots, create foo-expected-blink-cros.txt.
Directives allow you to control test flow and test output. The directives are defined inside the first comment block in the test's input file, one directive per line. For example, in the case of an HTML file the directives are located in between <!-- and -->, in the case of a PDF file the directives are preceding by % character designating a comment.
Directives have format of @directive_name:directive_value. Directives can be spawned over multiple lines:
@directive_name: directive_value directive_value
Certain directives are platform dependent. If so, then such directives are prefixed by a platform name:
@WIN- applied to Windows platform, MSAA/IAccessible2 APIs;@UIA-WIN- applied to UIA on Windows;@MAC- applied to Mac platform, NSAccessibility API;@BLINK- applied to Chromium engine;@ANDROID- applied to Android platform;@AURALINUX- applied to Linux platform, ATK API.By default only some attributes of nodes in the accessibility tree, or events fired (when running event tests), are output. This is to keep the tests robust and not prone to failure when unrelated changes affect the accessibility tree in unimportant ways.
You can use these filter types to match the attributes and/or attribute values you want included in the output.
-ALLOW filter means to include the attribute having non empty values;-ALLOW-EMPTY filter means to include the attribute even if its value is empty;-DENY filter means to exclude an attribute.Filter directives are platform-dependent (see above).
Filters can contain simple wildcards (*) only, they're not regular expressions. Examples:
@WIN-ALLOW:name will output the name attribute on Windows@WIN-ALLOW:name='Foo' will only output the name attribute if it exactly matches ‘Foo’.@WIN-DENY:name='X* will skip outputting any name that begins with the letter X.@WIN-ALLOW:* to dump all attributes, useful for debuggin a test.Note: Mac platform is supported only.
Script tests provide platform dependent -SCRIPT directive to indicate a script to run. For example:
@MAC-SCRIPT: input.AXName
to dump accessible name of an accessible node for a DOM element having input DOM id on Mac platform. You can also use :LINE_NUM syntax to indicate an accessible object, where LINE_NUM is index of a line where the accessible object is placed in the formatted tree. However you should avoid using :LINE_NUM in a test as it may break the test automatic rebaseling.
You can put multiple instructions under the same @MAC-SCRIPT directive, for example:
@MAC-SCRIPT: input.AXRole input.AXName
Calls can be chained. For example:
input.AXFocusableAncestor.AXRole
Note: The .AXAttribute will dump the accessible attribute for the node only if the attribute is supported for that node.
To test for the support of the attribute in mac accessibility API, you can see if the attribute is included in the accessibilityAttribute names using has(). For example, the following will tell you whether the attribute AXInvalid is supported on an accessible node, regardless of whether the attribute has been provided by the web author.
@MAC-SCRIPT: input.accessibilityAttributeNames.has(AXInvalid)
Parameterized attributes are also supported. For example:
paragraph.AXTextMarkerForIndex(0)
AXTextMarker is serialized as a {:LINE_NUM, offset, direction} triple, for example: textarea.AXPreviousWordStartTextMarkerForTextMarker({:3, 3, down})
AXTextMarkerRange is serialized as a dictionary: {anchor: TEXT_MARKER, focus: TEXT_MARKER}.
You can also retrieve anchor and focus text markers from a text marker range, for example:
p.AXTextMarkerRangeForUIElement(p).anchor or p.AXTextMarkerRangeForUIElement(p).focus
You can also use array operator[] to refer to an array element at a given index, for example paragraph.AXChildren[0] will refer to the first child of the paragraph.
To set a settable attribute you can assign a value to the attribute. For example:
textarea_range:= textarea.AXTextMarkerRangeForUIElement(textarea) textarea.AXSelectedTextMarkerRange = textarea_range
To pass a SEL as argument, you need to use the “@SEL:” prefix. For example:
@SCRIPT: slider.isAccessibilitySelectorAllowed(@SEL:setAccessibilityValue:)
You can use the wait for instruction to wait for a specific event before the script scontinues. For example:
@SCRIPT: button.AXPerformAction(AXPress) wait for AXFocusedUIElementChanged
will trigger AXPress action on a button and will wait for AXFocusedUIElementChanged event. You can also be more specific if you want to and provide the event target. For example: wait for AXFocusedUIElementChanged on AXButton
You can use press instruction to simulate key events. The instruction accepts a single parameter which could be a character or a key name (as specified in http://www.w3.org/TR/DOM-Level-3-Events-key/). For example,
@SCRIPT: press Enter wait for AXValueChanged
You can use print tree to print a snapshot of an accessible tree. For example,
@SCRIPT: print tree
Normally the system waits for the document to finish loading before running the test. You can tune the behavior up by the following directives.
Instructs to not wait for document load for url defined by the directive.
If you do not expect an iframe or object to load, (e.g. testing fallback), you can use the @NO-LOAD-EXPECTED: to cause the test to not wait for that frame to finish loading. For example the test would not wait for a url containing “broken.jpg” to load: @NO-LOAD-EXPECTED:broken.jpg <object data="./broken.jpg">Fallback</object
Delays a test unitl a string defined by the directive is present in the dump.
Occasionally you may need to write a dump tree test that makes some changes to the document before it runs the test. In that case you can use a special @WAIT-FOR: directive. It should be in an HTML comment, just like @ALLOW-WIN: directives. The WAIT-FOR directive just specifies a text substring that should be present in the dump when the document is ready. The system will keep blocking until that text appears.
You can add as many @WAIT-FOR: directives as you want, the test won't finish until all strings appear.
Delays a test until a string returned by a script defined by the directive is present in the dump.
You may also want to execute script and then capture a dump. Rather than use setTimeout and @WAIT-FOR:, consider using the @EXECUTE-AND-WAIT-FOR: directive. This directive expects a javascript function that returns a string to wait for. If a string is not returned, the tree dumper will not wait. @EXECUTE-AND-WAIT-FOR: directives are executed in order, after the document is ready and all @WAIT-FOR: strings have been found. Example: @EXECUTE-AND-WAIT-FOR: foo()
Invokes default action on an accessible object defined by the directive.
To skip dumping a particular element, add @NO_DUMP to a property that will be exposed as an ax::mojom::StringAttribute. For example <div class="@NO_DUMP"></div>.
To skip dumping all children of a particular element, add @NO_CHILDREN_DUMP to a property that will be exposed as an ax::mojom::StringAttribute. For example <div class="@NO_CHILDREN_DUMP"></div>.
Note that setting the aria-label value to @NO_DUMP or @NO_CHILDREN_DUMP is not guaranteed to work due to certain roles no longer supporting author- provided naming in ARIA 1.2.
To load an iframe from a different site, forcing it into a different process, use /cross-site/HOSTNAME/ in the url. For example: <iframe src="cross-site/1.com/accessibility/html/frame.html"></iframe>
If you want to populate the expectation file directly rather than typing it or copying-and-pasting it, first make sure the file exists (it can be empty), then run the test with the --generate-accessibility-test-expectations argument. For example:
out/Debug/content_browsertests \
--generate-accessibility-test-expectations \
--gtest_filter="All/DumpAccessibilityTreeTest.AccessibilityAriaAtomic/*"
This will replace the -expected-*.txt file with the current output. It's a great way to rebaseline a bunch of tests after making a change. Please manually check the diff, of course!
The * is a wildcard and will match any substring, in this case all platforms. To run on a single platform, replace the wildcard, e.g.:
--gtest_filter="All/DumpAccessibilityTreeTest.AccessibilityAriaAtomic/linux"
To rebaseline all OSes at once, use:
tools/accessibility/rebase_dump_accessibility_tree_test.py
For more information, see the detailed help with:
out/Debug/content_browsertests --gtest_help
Note: For Android, generated expectations will replace the existing files on the test device. For example, if running on an emulator, for an ARIA test called my-test.html, the generated output can be found:
/storage/emulated/0/chromium_tests_root/content/test/
data/accessibility/aria/my-test-expected-android.txt
If you are adding a new test file remember to add a corresponding test case in:
content/browser/accessibility/dump_accessibility_events_browsertest.cc; orcontent/browser/accessibility/dump_accessibility_tree_browsertest.ccIf you are adding a new events test, remember to add a corresponding test case for Android, see more info below.
These tests are similar to DumpAccessibilityTreeTest tests in that they first load an HTML document, then dump something, then compare the output to an expectation file. The difference is that what's dumped is accessibility events that are fired.
To write a test for accessibility events, your document must contain a JavaScript function called go(). This function will be called when the document is loaded (or when the @WAIT_FOR directive passes), and any subsequent events will be dumped. Filters apply to events just like in tree dumps.
After calling go(), the system asks the page to generate a sentinel accessibility event - one you‘re unlikely to generate in your test. It uses that event to know when to “stop” dumping events. There isn’t currently a way to test events that occur after some delay, just ones that happen as a direct result of calling go().
Windows will “translate” some IA2 events to UIA, and it is not possible to turn this feature off. Therefore as our UIA behavior is in addition to IA2, we will receive duplicated events for Focus, MenuOpened and MenuClosed.
The Android DumpAccessibilityEventsTests tests work differently than the other platforms and are driven by the Java-side code. The tests all reside in the WebContentsAccessibilityEventsTest.java class. The tests are controlled from the Java code so that they can leverage the full accessibility suite and test the AccessibilityEvents that are sent to downstream services. For this to work, when adding a new events test, you must include a test line in the Java class.
Example: If you are adding a new events test, “example-test.html”, you would first create the html file as normal (content/test/data/accessibility/event/example-test.html), and add the test to the existing dump_accessibility_events_browsertests.cc:
IN_PROC_BROWSER_TEST_P(DumpAccessibilityEventsTest, AccessibilityEventsExampleTest) {
RunEventTest(FILE_PATH_LITERAL("example-test.html"));
}
To include this test on Android, you would add a similar block to the WebContentsAccessibilityEventsTest.java class:
@Test
@SmallTest
public void test_exampleTest() {
performTest("example-test.html", "example-test-expected-android.txt");
}
Some tests on Android won't produce any events. For these you do not need to create an empty file, but can instead make the test line:
performTest("example-test.html", EMPTY_EXPECTATIONS_FILE);
The easiest approach is to use the above line, run the tests, and if it fails, the error message will give you the exact text to add to the -expected-android.txt file. The -expected-android.txt file should go in the same directory as the others (content/test/data/accessibility/event).
A PRESUBMIT check will give a non-blocking warning if you are adding, renaming, or deleting an events test without a corresponding change for Android.