| <style> | |
| .note::before { | |
| content: 'Note: '; | |
| font-variant: small-caps; | |
| font-style: italic; | |
| } | |
| .doc h1 { | |
| margin: 0; | |
| } | |
| </style> | |
| # Testing WebUI Pages | |
| [TOC] | |
| Chromium WebUI pages should be tested using Mocha test suites. | |
| [Mocha](https://mochajs.org) is a flexible, lightweight test framework with | |
| strong async support. With Mocha, we can write small unit tests for individual | |
| features of components instead of grouping everything into a monolithic browser | |
| test. | |
| There are a few pieces of Chromium WebUI infrastructure designed to make | |
| setting up WebUI Mocha browser tests easier: | |
| * `build_webui_tests` is a gn build rule that preprocesses test `.ts` files, | |
| compiles them to JavaScript, and places them in a `.grdp` file. This file | |
| can then be included in the WebUI test `.grd` so that the files can be | |
| served at runtime. This rule is further documented at | |
| [WebUI Build Configuration](./webui_build_configuration.md) | |
| * `WebUIMochaBrowserTest` is a test base class that WebUI tests should | |
| inherit from. It handles browser test setup and teardown, injects Mocha, | |
| loads the `chrome://` page being tested, and imports the test files. This | |
| class also ensures code coverage metrics are collected when JS code | |
| coverage is enabled. The API for this class is documented in its | |
| [header | |
| file](https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/base/web_ui_mocha_browser_test.h) | |
| Note: For the default test loading logic in `WebUIMochaBrowserTest` to work, | |
| `test_loader.html`, `test_loader_util.js` and `test_loader.js` should be | |
| registered with the production Web UI's data source. This can be done manually, | |
| or (most commonly) can be done by calling `SetupWebUIDataSource()`. If using | |
| the test loader is not feasible for a given test - for example, test relies on | |
| loading a specific URL such that loading `chrome://foo-bar/test_loader.html` | |
| would break the test - see [Unusual Test Setups](#unusual-test-setups) | |
| ### Adding the BrowserTest | |
| UIs should add a `.cc` file with the definition of their BrowserTest class in | |
| `chrome/test/data/webui/my_webui_dir_name`. The directory structure | |
| of `chrome/test/data/webui` should mirror that of | |
| `chrome/browser/resources/` where the WebUI TypeScript/HTML code being tested | |
| resides. The BrowserTest class defined in the `.cc` file should inherit from | |
| `WebUIMochaBrowserTest`. | |
| Example: | |
| ```c++ | |
| // chrome/test/data/webui/foo_bar/foo_bar_browsertest.cc | |
| #include "base/test/scoped_feature_list.h" | |
| #include "chrome/common/webui_url_constants.h" | |
| #include "chrome/test/base/web_ui_mocha_browser_test.h" | |
| #include "content/public/test/browser_test.h" | |
| class FooBarBrowserTest : public WebUIMochaBrowserTest { | |
| protected: | |
| FooBarBrowserTest() { | |
| set_test_loader_host(chrome::kChromeUIFooBarHost); | |
| } | |
| private: | |
| // Necessary if you have features (or the UI itself) behind a feature flag. | |
| base::test::ScopedFeatureList scoped_feature_list_{foo_bar::kSomeNewFeature}; | |
| }; | |
| ``` | |
| Note: If testing a `chrome-untrusted://` UI, also call: | |
| ```c++ | |
| set_test_loader_scheme(content::kChromeUIUntrustedScheme); | |
| ``` | |
| in the constructor. | |
| After defining the class, create one or more `IN_PROC_BROWSER_TEST_F` | |
| functions. Each should run one or more Mocha tests. | |
| ``` c++ | |
| typedef FooBarBrowserTest FooBarAppTest; | |
| IN_PROC_BROWSER_TEST_F(FooBarAppTest, All) { | |
| // Run all Mocha test suites found in app_test | |
| RunTest("foo_bar/app_test.js", "mocha.run();"); | |
| } | |
| ``` | |
| Should you write a large number of tests in a single file (e.g., for | |
| one particular custom element), it may be necessary to run different suites or | |
| even individual tests from that same file in separate `IN_PROC_BROWSER_TEST_F` | |
| invocations, to avoid having one very long test that is at risk of timing out. | |
| Example: | |
| ```c++ | |
| // This class defines a convenience method to run one Mocha test suite in the | |
| // search_box_test.js file at a time. | |
| class FooBarSearchBoxTest: public FooBarBrowserTest { | |
| protected: | |
| void RunTestSuite(const std::string& suiteName) { | |
| FooBarBrowserTest::RunTest( | |
| "foo_bar/search_box_test.js", | |
| base::StringPrintf("runMochaSuite('%s');", suiteName.c_str())); | |
| } | |
| }; | |
| IN_PROC_BROWSER_TEST_F(FooBarSearchBoxTest, SearchNonEmpty) { | |
| RunTestSuite("SearchNonEmpty"); | |
| } | |
| IN_PROC_BROWSER_TEST_F(FooBarSearchBoxTest, SearchEmpty) { | |
| RunTestCase("SearchEmpty"); | |
| } | |
| // This class defines a convenience method to run one Mocha test from the | |
| // FooBarChildDialogTest suite at a time. | |
| class FooBarChildDialogTest: public FooBarBrowserTest | |
| protected: | |
| void RunTestCase(const std::string& testCase) { | |
| FooBarBrowserTest::RunTest( | |
| "foo_bar/child_dialog_test.js", | |
| base::StringPrintf("runMochaTest('FooBarChildDialogTest', '%s');", | |
| testCase.c_str())); | |
| } | |
| }; | |
| IN_PROC_BROWSER_TEST_F(FooBarChildDialogTest, OpenDialog) { | |
| RunTestCase("OpenDialog"); | |
| } | |
| IN_PROC_BROWSER_TEST_F(FooBarChildDialogTest, ClickButtons) { | |
| RunTestCase("ClickButtons"); | |
| } | |
| ``` | |
| ### Adding the Mocha test files | |
| Mocha test files should be added alongside the `.cc` file in the same | |
| directory. Each test file should contain one or more Mocha test suites. Often | |
| each major custom element of a UI will have its own Mocha test file, as will | |
| many of the data classes. | |
| A build time check exists as part of `build_webui_tests()` to ensure Mocha test | |
| files are actually hooked up to a C++ test file. Mocha test files must end with | |
| the `\_test.ts` suffix. Any other TS supporting files that don't actually hold | |
| Mocha test cases should not finish with this suffix otherwise the automated | |
| check will trigger erroneously. | |
| `WebUIMochaBrowserTest` sets up a `WebUIDataSource` to serve all the test files | |
| from `chrome://webui-test` (or `chrome-untrusted://webui-test`, if configured | |
| via `set_test_loader_scheme`). | |
| ```ts | |
| // chrome/test/data/webui/foo_bar/search_box_test.ts | |
| import 'chrome://foo-bar/foo_bar.js'; | |
| import type {FooBarSearchBoxElement} from 'chrome://foo-bar/foo_bar.js'; | |
| import {assertEquals, assertFalse, assertTrue} from 'chrome://webui-test/chai_assert.js'; | |
| import {eventToPromise} from 'chrome://webui-test/test_util.js'; | |
| suite('SearchNonEmpty', function() { | |
| let searchBox: FooBarSearchBoxElement; | |
| setup(function() { | |
| document.body.innerHTML = window.trustedTypes!.emptyHTML; | |
| searchBox = document.createElement('foo-bar-search-box'); | |
| document.body.appendChild(searchBox); | |
| // More setup goes here | |
| }); | |
| test('search abc', async function () { | |
| const search = searchBox.shadowRoot!.querySelector('input'); | |
| assertTrue(!!search); | |
| search.value = 'abc'; | |
| await eventToPromise('search-change', searchBox); | |
| // Do some assertions | |
| }); | |
| test('search def', async function () => { | |
| // etc | |
| }); | |
| }); | |
| suite('SearchEmpty', function() { | |
| let searchBox: FooBarSearchBoxElement; | |
| setup(function() { | |
| document.body.innerHTML = window.trustedTypes!.emptyHTML; | |
| const searchBox = document.createElement('foo-bar-search-box'); | |
| document.body.appendChild(searchBox); | |
| // Different setup goes here | |
| }); | |
| test('search abc when empty', async() { | |
| const search = searchBox.shadowRoot!.querySelector('input'); | |
| assertTrue(!!search); | |
| search.value = 'abc'; | |
| await eventToPromise('search-change', searchBox); | |
| // Do some assertions | |
| }); | |
| }); | |
| ``` | |
| ### Setting up the build | |
| To build your tests as part of the `browser_tests` target, add a BUILD.gn file | |
| in your folder: | |
| ```python | |
| # chrome/test/data/webui/foo_bar/BUILD.gn | |
| import("../build_webui_tests.gni") | |
| build_webui_tests("build") { | |
| cc_test_files = [ "foo_bar_browsertest.cc" ] | |
| # Test files go here | |
| files = [ | |
| "app_test.ts", | |
| "child_dialog_test.ts", | |
| "search_box_test.ts", | |
| ] | |
| # Path mapping the WebUI host URL to the path where the WebUI's compiled | |
| # TypeScript files are placed by the TS compiler. | |
| ts_path_mappings = | |
| [ "chrome://foo-bar/*|" + | |
| rebase_path("$root_gen_dir/chrome/browser/resources/foo_bar/tsc/*", | |
| target_gen_dir) ] | |
| ts_deps = [ | |
| "//chrome/browser/resources/foo_bar:build_ts", | |
| ] | |
| } | |
| ``` | |
| Note: If testing a `chrome-untrusted://` UI, also set the | |
| `is_chrome_untrusted` parameter for `build_webui_tests()` to `true`, to allow | |
| importing shared test files from `chrome-untrusted://webui-test/` paths. | |
| You then need to hook up the targets generated by this file, and list your | |
| browsertest.cc source file, in `chrome/test/data/webui/BUILD.gn`: | |
| ```python | |
| # chrome/test/data/webui/BUILD.gn | |
| source_set("browser_tests") { | |
| testonly = true | |
| sources = [ | |
| # Lots of files | |
| # Big list, add your .cc file in alphabetical order | |
| "foo_bar/foo_bar_browsertest.cc", | |
| # Lots more files | |
| ] | |
| # Conditionally added sources (e.g., platform-specific) | |
| } | |
| # Further down in the file... | |
| generate_grd("build_grd") { | |
| testonly = true | |
| grd_prefix = "webui_test" | |
| output_files_base_dir = "test/data/grit" | |
| out_grd = "$target_gen_dir/resources.grd" | |
| deps = [ | |
| ":build_chai_grdp", | |
| ":build_mocha_grdp", | |
| # Lots of other deps here | |
| # Add your build_grdp target: | |
| "foo_bar:build_grdp", | |
| # Lots more other deps | |
| ] | |
| grdp_files = [ | |
| # Big list of grdp files | |
| # Add your grdp in alphabetical order | |
| "$target_gen_dir/foo_bar/resources.grdp", | |
| # Lots more grdp files | |
| ] | |
| # Conditionally added (e.g. platform-specific) grdps and deps go here | |
| } | |
| ``` | |
| ### Unusual Test Setups | |
| Some UIs may need to rely on loading a particular URL, such that using the | |
| default `chrome://foo-bar/test_loader.html` URL that is used to dynamically | |
| import the test modules may not work. Other UIs need to do additional setup | |
| on the C++ side of the test, and simply want to run a test on a specified | |
| WebContents instance. These cases are accommodated by additional methods in | |
| `WebUIMochaBrowserTest`. A few brief examples are provided below. | |
| Example: loading a different URL | |
| ```c++ | |
| IN_PROC_BROWSER_TEST_F(FooBarBrowserTest, NonExistentUrl) { | |
| // Invoking the test from a non existent URL chrome://foo-bar/does-not-exist/. | |
| set_test_loader_host( | |
| std::string(chrome::kChromeFooBarHost) + "/does-not-exist"); | |
| RunTestWithoutTestLoader("foo_bar/non_existent_url_test.js", "mocha.run()"); | |
| } | |
| ``` | |
| Example: Advanced setup required in C++ | |
| ```c++ | |
| IN_PROC_BROWSER_TEST_F(FooDialogBrowserTest, MyTest) { | |
| ui_test_utils::NavigateToURLWithDisposition( | |
| browser(), GURL(chrome::kSomeOtherBaseUrl), | |
| WindowOpenDisposition::CURRENT_TAB, | |
| ui_test_utils::BROWSER_TEST_WAIT_FOR_LOAD_STOP); | |
| FooDialog* my_dialog = FooDialog::ShowConstrained( | |
| kDummyParam, | |
| browser()->tab_strip_model()->GetActiveWebContents(), | |
| browser()->window()->GetNativeWindow()); | |
| content::WebContents* web_contents = my_dialog->webui_->GetWebContents(); | |
| ASSERT_TRUE(RunTestOnWebContents( | |
| web_contents, "foo_dialog/foo_dialog_test.js", "mocha.run()", | |
| /*skip_test_loader=*/true)); | |
| } | |
| ``` | |
| ### Running tests | |
| You can build the `browser_tests` target and run your WebUI tests just like | |
| other browser tests. You may filter tests by class or test name (the arguments | |
| to `IN_PROC_BROWSER_TEST_F`) via `--gtest_filter`: | |
| ```sh | |
| autoninja -C out/Release browser_tests | |
| ./out/Release/browser_tests --gtest_filter=FooBarSearchBoxTest.Empty | |
| ``` | |
| ### Debugging tests | |
| You can run a subset of the Mocha test cases, by using `test.only()` instead of | |
| `test()`. For example: | |
| ```ts | |
| test.only('MyTestcase', function() {...}); | |
| ``` | |
| You can skip a subset of the Mocha test cases, by using `test.skip()` instead of | |
| `test()`. For example: | |
| ```ts | |
| test.skip('MyTestcase', function() {...}); | |
| ``` | |
| When authoring a new test case or when investigating existing test failures, you | |
| will often have a need to step through the code in the DevTools debugger, to | |
| figure out what is happening. Use the `launchDebugger` helper function to do | |
| that. Detailed instructions on how to use it exist in the source code | |
| documentation [here](https://source.chromium.org/chromium/chromium/src/+/main:chrome/test/data/webui/test_util.ts;l=107-144;drc=b1866df4398a971088ba287d4c7efe704f6bc4b1). | |
| ### Disabling tests | |
| Tests can be disabled at the GTest (C++) level or the Mocha (TypeScript) | |
| level. Disabling at the Mocha level is preferred as it allows for more | |
| granular control (skipping individual test cases rather than entire suites) | |
| and provides better visibility into which specific tests are failing. If the | |
| entire suite is flaky or failing, then disabling the GTest might be | |
| appropriate. | |
| If your team receives a flaky test bug and you notice that the whole C++ GTest | |
| suite is disabled, before deflaking you should first land a CL to disable the | |
| specific test cases that are causing problems. After that lands, you can | |
| investigate the flakes or failures. This will reduce the chance that other | |
| tests in the suite start failing while they are disabled. | |
| #### Determining which test cases are failing | |
| Test cases can fail or entire suites can time out if they take too long to run. | |
| You can expand the "Summary" that appears below the "Failure Reason" and | |
| search for the cases marked "failed" or the last test case marked "started" if | |
| there is a timeout. Here is an example where the "RenamesBookmark" case failed. | |
| ``` | |
| [ RUN ] SidePanelBookmarksAppTest.General2 | |
| [25749:440407:0512/171449.559368:INFO:components/enterprise/browser/controller/chrome_browser_cloud_management_controller.cc:239] No machine level policy manager exists. | |
| Trying to load the allocator multiple times. This is *not* supported. | |
| [25749:440407:0512/171449.854681:INFO:CONSOLE:35] "start", source: chrome://webui-test/mocha_adapter_simple.js (35) | |
| ... | |
| [25749:440407:0512/171450.076260:INFO:CONSOLE:44] " started: General Part2 SetsExpandedSearchResultDescription", source: chrome://webui-test/mocha_adapter_simple.js (44) | |
| [25749:440407:0512/171450.101585:INFO:CONSOLE:47] " passed: General Part2 SetsExpandedSearchResultDescription", source: chrome://webui-test/mocha_adapter_simple.js (47) | |
| [25749:440407:0512/171450.102566:INFO:CONSOLE:44] " started: General Part2 RenamesBookmark", source: chrome://webui-test/mocha_adapter_simple.js (44) | |
| [25749:440407:0512/171450.114796:INFO:CONSOLE:58] " failed: General Part2 RenamesBookmark | |
| AssertionError: expected false to be true | |
| at assertTrue (chrome://webui-test/chai_assert.js:14:12) | |
| at Context.<anonymous> (chrome://webui-test/side_panel/bookmarks/power_bookmarks_app_test.js:587:13)", source: chrome://webui-test/mocha_adapter_simple.js (58) | |
| [25749:440407:0512/171450.115013:INFO:CONSOLE:44] " started: General Part2 BlursRenameInput", source: chrome://webui-test/mocha_adapter_simple.js (44) | |
| [25749:440407:0512/171450.132939:INFO:CONSOLE:47] " passed: General Part2 BlursRenameInput", source: chrome://webui-test/mocha_adapter_simple.js (47) | |
| ... | |
| [25749:440407:0512/171450.385904:INFO:CONSOLE:62] "end: 18/19 ok", source: chrome://webui-test/mocha_adapter_simple.js (62) | |
| [25749:440407:0512/171450.385944:INFO:CONSOLE:64] "TEST all complete, status=FAIL, duration=531ms", source: chrome://webui-test/mocha_adapter_simple.js (64) | |
| ``` | |
| Individual test cases also appear as an additional test result. See the | |
| [below section](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/webui/testing_webui.md#reporting-webui-test-results-to-luci) | |
| on how test cases show up in LUCI. | |
| If there is a timeout, it is helpful to look at a few of the failures to see if | |
| the same test case is always timing out or different tests (especially ones | |
| near the end) are flaking. If different cases are flaking each run, the entire | |
| suite will likely need to be disabled while an owner of the tests splits it | |
| into two test suites. If the same case is always timing out, then that can be | |
| disabled. | |
| #### Disabling in Mocha (Preferred) | |
| To disable a single test case for all platforms, use `test.skip()` instead of | |
| `test()`: | |
| ```ts | |
| // TODO(crbug.com/511960512): Flaky test. | |
| test.skip('MyTestcase', function() {...}); | |
| ``` | |
| To disable an entire suite, use `suite.skip()` instead of `suite()`: | |
| ```ts | |
| // TODO(crbug.com/511960512): Flaky test. | |
| suite.skip('MySuite', function() {...}); | |
| ``` | |
| To disable a test for a specific platform, use `<if expr>`: | |
| ```ts | |
| // TODO(crbug.com/511960512): Flaky test. | |
| // <if expr="not is_macosx"> | |
| test('LogsBookmarkCountMetric', async () => { | |
| ... | |
| } | |
| // </if> | |
| ``` | |
| #### Disabling in GTest (Discouraged) | |
| Disabling a C++ GTest suite using the `DISABLED_` prefix is discouraged. There | |
| are often dozens of individual test cases under a single C++ GTest suite. | |
| Disabling the whole suite when a single test is failing results in decreased | |
| test coverage and a higher change that regressions could be introduced. | |
| Disabling a Mocha test for a specific bot configuration (e.g. MSAN or ASAN) | |
| isn't supported. If you encounter a failure, weigh the benefits of disabling | |
| all tests for that bot configuration versus a specific test for all | |
| configurations. If the configuration is only run for a single OS, you could | |
| disable the test case for just that OS using `<if expr>` syntax described | |
| above. | |
| Here is how to disable a test in GTest if that is required: | |
| ```c++ | |
| // TODO(crbug.com/511960512): Flaky test. | |
| IN_PROC_BROWSER_TEST_F(SidePanelBookmarksAppTest, DISABLED_General2) { | |
| SidePanelBookmarksTest::RunTest( | |
| "side_panel/bookmarks/power_bookmarks_app_test.js", | |
| "runMochaSuite('General Part2');"); | |
| } | |
| ``` | |
| A presubmit check enforced in `chrome/test/data/webui/` will trigger if | |
| `DISABLED_` is added to C++ test files. This can be bypassed by adding | |
| `SKIP_DISABLING_SUITES_CHECK: <reason>` to the CL description if disabling at the | |
| C++ level is truly necessary (e.g., the entire test suite is flaky or failing). | |
| ### Reporting WebUI test results to LUCI | |
| WebUI test results are automatically reported to LUCI. No special handling is | |
| required of WebUI test authors. | |
| A single GTest typically reports a single test result. A [new test reporting | |
| mechanism was | |
| added](https://chromium-review.googlesource.com/c/chromium/src/+/6358478) to | |
| support reporting multiple results from a single GTest. Using this mechanism, | |
| each Mocha JS test can have its own entry in LUCI (see screenshot). The | |
| integration is accomplished by calling base::AddSubTestResult in | |
| WebUIMochaBrowserTest. | |
|  | |
| ### Common errors | |
| Tests that rely on focus, blur, or other input-related events need to be added | |
| to the interactive_ui_tests build target rather than the browser_tests target. | |
| browser_tests are run in parallel, and the window running the test will | |
| gain/lose activation at unpredictable times. | |
| Tests that touch production logic that depends on onFocus, onBlur or similar | |
| events also needs to be added to interactive_ui_tests |