The WebAppProvider system has very wide action space and testing state interactions between all of the subsystems is very difficult. The goal of this piece is to accept:
This is done by downloading data from a google sheet, processing it, and saving the results in the output/ folder.
See the design doc for more information and links.
This README covers how the spreadsheet & tests are read to compute coverage information and tell the script runner how to modify the tests to get the respective coverage. For information about how to implement actions in the framework (or how the framework implementation works), see the testing framework implementation README.md.
Related:
A primitive test operation, or test building block, that can be used to create coverage tests. The integration test framework may or may not support an action on a given platform. Actions can fall into one of three types:
Actions also can be fully, partially, or not supported on a given platform. This information is used when generating the tests to run & coverage report for a given platform. To make parsing easier, actions are always snake_case.
A state-changing action is expected to change the state chrome or the web app provider system.
Examples: navigate_browser(SiteA), switch_incognito_profile, sync_turn_off, set_app_badge
Some actions are classified as “state check” actions, which means they do not change any state and only inspect the state of the system. In graph representations, state check actions are not given nodes, and instead live under the last non-state-check action.
All actions that start with check_ are considered state-check actions.
Examples: check_app_list_empty, check_install_icon_shown, check_platform_shortcut_exists(SiteA), check_tab_created
When creating tests, there emerged a common scenario where a given action could be applied to multiple different sites. For example, the “navigate the browser to an installable site” action was useful if “site” could be customized.
The simplest possible mode system to solve this:
To allow for future de-parsing of modes (when generating C++ tests), modes will always be CapitalCase.
To help with testing scenarios like outlined above, an action can be defined that references or ‘turns into’ a set of non-parameterized actions. For example, an action install_windowed can be created and reference the set of actions install_omnibox_icon, install_menu_option, install_create_shortcut_windowed, add_policy_app_windowed_shortcuts, and add_policy_app_windowed_no_shortcuts. When a test case includes this action, it will generate multiple tests in which the parameterized action is replaced with the non-parameterized action.
A sequence of actions used to test the WebAppProvider system. A test that can be run by the test framework must not have any “parameterized” actions, as these are supposed to be used to generate multiple tests.
This is the set of tests that, if all executed, should provide full test coverage for the WebAppProvider system. They currently live in this sheet as “unprocessed”.
Processed tests go through the following steps from the unprocessed version in the sheet:
Some tests are going to be platform-specific. For example, all tests that involve “locally installing” an app are only applicable on Windows/Mac/Linux, as ChromeOS automatically locally installs all apps from sync. Because of this, tests must be able to specify which platforms they should be run on. This is done by specifying the platforms each test applies to in a column on the spreadsheet.
Actions are the basic building blocks of integration tests. A test is a sequence of actions. Each action has a name that must be a valid C++ identifier.
Actions are defined (and can be modified) in this sheet. Tests are defined (and can be modified) in this sheet.
Actions are the building blocks of tests.
To help making test writing less repetitive, actions are described as templates in the actions spreadsheet. Action templates specify actions while avoiding rote repetition. Each action template has a name (the action base name). Each action template supports a mode, which takes values from a predefined list associated with the template. Parameter values must also be valid C++ identifiers.
An action template without modes specifies one action whose name matches the template. For example, the check_tab_created template generates the check_tab_created action.
An action template with a mode that can take N values specifies N actions, whose names are the concatenations of the template name and the corresponding value name, separated by an underscore (_). For example, the clear_app_badge template generates the clear_app_badge_SiteA and clear_app_badge_SiteB actions.
The templates also support parameterizing an action, which causes any test that uses the action to be expanded into multiple tests, one per specified output action. Modes will carry over into the output action, and if an output action doesn't support a given mode then that parameterization is simply excluded during test generation.
All templates with modes can mark one of their mode values as the default value. This value is used to magically convert template names to action names.
Human-friendly action names are a slight variation of the canonical names above.
Actions generated by mode-less templates have the same human-friendly name as their (canonical?) name.
Actions generated by parametrized templates use parenthesis to separate the template name from the value name. For example, the actions generated by the clear_app_badge template have the human-friendly names clear_app_badge(SiteA) and clear_app_badge(SiteB).
The template name can be used as the human-friendly name of the action generated by the template with the default value. For example, clear_app_badge is a shorter human-friendly name equivalent to clear_app_badge(SiteA).
Tests are created specifying actions.
The mindset for test creation and organization is to really exhaustively check every possible string of user actions. The framework will automatically combine tests that are the same except for state check actions. They are currently organized by:
Each test can have at most one state check action as the last action.
One way to enumerate tests is to think about affected-by action edges. These are a pair of two actions, where the first action affects the second. For example, the action set_app_badge will affect the action check_app_badge_has_value. Or, uninstall_from_app_list will affect check_platform_shortcut_exists. There is often then different setup states that would effect these actions. Once these edges are identified, tests can be created around them.
A test should be created that does the bare minimum necessary to set up the test before testing the primary state change action and then checking the state.
The framework is designed to be able to collapse tests that contain common non-‘state-check’ actions, so adding a new test does not always mean that a whole new test will be run by the framework. Sometimes it only adds a few extra state-check actions in an existing test.
If a new test needs a new action implemented, it will only be used in the generated tests if it is added to the actions sheet, and it is specifically marked as supported or partially supported. Then the script will print out the relevant browsertest to add to the relevant file.
The action also needs to be implemented by the testing framework. See the Testing Framework Implementation README.md for more info about how to do that.
To tell the script that an action is supported by the testing framework (on a given platform), modify the framework_supported_actions.csv file, and use the following emojis to specify coverage for an action on a given platform:
The script reads this file to determine what tests to generate.
The test data is hosted in this spreadsheet. To download the latest copy of the data, run the included script:
./chrome/test/webapps/download_data_from_sheet.py
This will download the data from the sheet into csv files in the data/ directory:
actions.csv This describes all actions that can be used in the required coverage tests (processed or unprocessed).coverage_required.csv This is the full list of all tests needed to fully cover the Web App system. The first column specifies the platforms for testing, and the test starts on the fifth column.Required test changes are printed and coverage files are written by running:
chrome/test/webapps/generate_framework_tests_and_coverage.py
This uses the files in chrome/test/webapps/data and existing browsertests on the system (see custom_partitions and default_partitions in generate_framework_tests_and_coverage.py) to:
stdout all detected changes needed to browsertests.The script is not smart enough to automatically add/remove/move tests to keep complexity to a minimum. Instead, it prints out the tests that need to be added or removed to have the tests match what it expects. It assumes:
TestPartitionDescriptions in generate_framework_tests_and_coverage.py._mac, _win, etc) are only run on those platformsThis process doesn't modify the browsertest files so any test disabling done by sheriffs can remain. The script runner is thus expected to make the requested changes manually. In the rare case that a test is moving between files (if we are enabling a test on a new platform, for example), then the script runner should be careful to copy any sheriff changes to the browsertest as well.
tsv files in chrome/test/webapps/coverageThese are the processed required coverage tests with markers per action to allow a conditional formatter (like the one here) to highlight what was and was not covered by the testing framework.
To view the directed graphs that are generated to process the test and coverage data, the --graphs switch can be specified:
chrome/test/webapps/generate_framework_tests_and_coverage.py --graphs
This will generate:
coverage_required_graph.dot - The graph of all of the required test coverage. Green nodes are actions explicitly listed in the coverage list, and orange nodes specify partial coverage paths.framework_test_graph_<platform>.dot - The graph that is now tested by the generated framework tests for the given platform, including partial coverage.The graphviz library can be used to view these graphs. An install-free online version is here.
To help debug or explore further, please see the graph_cli_tool.py script which includes a number of command line utilities to process the various files.
Both this file and the generate_framework_tests_and_coverage.py file support the -v option to print out informational logging.