Normally, Android Instrumentation tests finish all Activities between tests. In Chrome we go even further and run a single test per Instrumentation invocation by default, so the entire process also gets killed and restarted between tests.
Test batching groups multiple tests into the same Instrumentation invocation, and disables the Activity finishing between tests, so you have full control over what setup/teardown happens between your tests and can reset to a known-good state without restarting Activities, etc. which can save multiple seconds per test.
Add the @Batch annotation to the test class, and ensure that each test within the chosen batch doesn't leave behind state that could cause other tests in the batch to fail.
All tests in a batch can be run locally using the -A test filter. eg:
out/<dir>/bin/run_chrome_public_test_apk -A Batch=UnitTests
Each test suite is likely to require careful consideration for how to batch the tests. For some tests batching won’t be as useful (tests that test Activity startup, for example), and tests that test process startup shouldn’t be batched at all. For most tests, you’ll want to pick a known starting state for each test, and ensure each test resets to that state.
If a few tests within a larger batched suite cannot be batched (eg. it tests process initialization), you may add the @RequiresRestart annotation, which will exclude that test from the batch.
Tests that belong in this category are tests that are effectively unit tests. They may be written as instrumentation tests rather than junit tests for a variety of reasons such as needing to use real Android APIs, or needing to use the native library.
Batching Unit Test style tests is usually fairly simple (example). It requires adding the "@Batch(Batch.UNIT_TESTS)” annotation, and ensuring no global state, like test overrides, persists across tests. Unit Tests should also not start the browser process, but may load the native library. Note that even with Batched tests, the test fixture (the class) is recreated for each test.
Note that since the browser isn't initialized for unit tests, if you would like to take advantage of feature annotations in your test you will have to use Features#JUnitProcessor instead of Features#InstrumentationProcessor.
This batching type is typically for larger and more complex test suites, and will run the suite in its own batch. This will reduce the complexity of managing and leaking state from these tests as you only have to think about tests within the suite. For smaller and less complex test suites, see Custom below.
If you use different @Features annotations on test methods, you can use the @Batch.SplitByFeature annotation to run tests with different features in separate batches.
This batching type is best for smaller and less complex test suites, that require browser initialization, or something else that prevents them from being unit tests. Custom batches allow you to pay the process startup cost once per batch instead of once per test suite. To put multiple test suites into the same batch, you will have to use a shared custom batch name (example). When batching across suites you’ll want to use something like BlankCTATabInitialStateRule to persist static state (like the Activity) between test suites and perform any necessary state cleanup between tests.
Note that there is an inherent tradeoff here between batch size and debuggability - the larger your batch, the harder it will be to diagnose one test causing a different test to fail/flake. I would recommend grouping tests semantically to make it easier to understand relationships between the tests and which shared state is relevant.
Example command to run all of the tests in a custom batch:
./tools/autotest.py -C out/Debug BluetoothChooserDialogTest \ --gtest_filter="*" -A Batch=device_dialog
./tools/autotest.py -C out/Debug ExternalNavigationHandlerTest \ --gtest_filter="*#testOrdinaryIncognitoUri:*#testChromeReferrer"