ANGLE Test Harness

The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It features:

  • splitting a test set into shards
  • catching and reporting crashes and timeouts
  • outputting to the Chromium JSON test results format
  • multi-process execution

Command-Line Arguments

The ANGLE test harness accepts all standard GoogleTest arguments. The harness also accepts the following additional command-line arguments:

  • --shard-count and --shard-index control the test sharding
  • --bot-mode enables multi-process execution and test batching
  • --batch-size limits the number of tests to run in each batch
  • --batch-timeout limits the amount of time spent in each batch
  • --max-processes limits the number of simuntaneous processes
  • --test-timeout limits the amount of time spent in each test
  • --results-file specifies a location for the JSON test result output
  • --results-directory specifies a directory to write test results to
  • --filter-file allows passing a larget gtest_filter via a file

Implementation Notes

  • The test harness only requires angle_common and angle_util.
  • It does not depend on any Chromium browser code. This allows us to compile on other non-Clang platforms.
  • It uses rapidjson to read and write JSON files.
  • Timeouts are detected via a watchdog thread.
  • Crashes are handled via ANGLE's test crash handling code.
  • Currently it does not entirely support Android or Fuchsia.
  • Test execution is not currently deterministic in multi-process mode.
  • We capture stdout to output test failure reasons.

See the source code for more details: TestSuite.h and TestSuite.cpp.

Potential Areas of Improvement

  • Deterministic test execution.
  • Using sockets to communicate with test children. Similar to dEQP's test harness.
  • Closer integration with ANGLE's test expectations and system config libraries.
  • Supporting a GoogleTest-free integration.