tree: cb4f1a68a8dce6ba0ff76cee40f8185a144083da [path history] [tgz]
  1. stable_test_filters/
  2. BUILD.gn
  3. OWNERS
  4. PRESUBMIT.py
  5. README.md
  6. android.asan.breakpad_unittests.filter
  7. android.asan.content_browsertests.filter
  8. android.asan.unit_tests.filter
  9. android.emulator.base_unittests.filter
  10. android.emulator.blink_unittests.filter
  11. android.emulator.cc_unittests.filter
  12. android.emulator.gl_tests.filter
  13. android.emulator.net_unittests.filter
  14. android.emulator.sandbox_linux_unittests.filter
  15. android.emulator.viz_unittests.filter
  16. android.emulator.webview_instrumentation_test_apk.filter
  17. android.emulator_10.net_unittests.filter
  18. android.emulator_11.chrome_public_test_apk.filter
  19. android.emulator_11.content_shell_test_apk.filter
  20. android.emulator_11.crashpad_tests.filter
  21. android.emulator_11.gl_tests.filter
  22. android.emulator_11.net_unittests.filter
  23. android.emulator_11.viz_unittests.filter
  24. android.emulator_11_12.cc_unittests.filter
  25. android.emulator_11_12.media_unittests.filter
  26. android.emulator_12.crashpad_tests.filter
  27. android.emulator_12.device_unittests.filter
  28. android.emulator_12.gl_tests.filter
  29. android.emulator_12.gl_unittests.filter
  30. android.emulator_12.sandbox_linux_unittests.filter
  31. android.emulator_m.chrome_public_test_apk.filter
  32. android.emulator_m.content_browsertests.filter
  33. android.emulator_m.content_shell_test_apk.filter
  34. android.emulator_m.media_unittests.filter
  35. android.emulator_p.chrome_public_test_apk.filter
  36. android.emulator_p.content_browsertests.filter
  37. android.emulator_p.media_unittests.filter
  38. android.marshmallow_tablet_tester.content_browsertests.filter
  39. android.pie_arm64_rel.chrome_public_test_apk.filter
  40. android.pie_arm64_rel.gl_tests.filter
  41. bfcache.unit_tests.filter
  42. cast-linux.content_browsertests.filter
  43. chromeos.base_unittests.filter
  44. chromeos.browser_tests.filter
  45. chromeos.media_unittests.filter
  46. chromeos.msan.browser_tests.oobe_negative.filter
  47. chromeos.msan.browser_tests.oobe_positive.filter
  48. chromeos.ozone_unittests.filter
  49. chromeos.services_unittests.filter
  50. chromeos.unit_tests.filter
  51. chromium.webrtc.fyi.android.tests.dbg.content_browsertests.filter
  52. code_coverage.browser_tests.filter
  53. e2e.sync_integration_tests.filter
  54. fuchsia.base_perftests.filter
  55. fuchsia.blink_unittests.filter
  56. fuchsia.components_unittests.filter
  57. fuchsia.content_browsertests.filter
  58. fuchsia.content_unittests.filter
  59. fuchsia.debug.cc_unittests.filter
  60. fuchsia.debug.compositor_unittests.filter
  61. fuchsia.debug.media_unittests.filter
  62. fuchsia.debug.snapshot_unittests.filter
  63. fuchsia.debug.views_unittests.filter
  64. fuchsia.debug.viz_unittests.filter
  65. fuchsia.headless_browsertests.filter
  66. fuchsia.lsan.accessibility_unittests.filter
  67. fuchsia.lsan.base_unittests.filter
  68. fuchsia.lsan.content_unittests.filter
  69. fuchsia.lsan.extensions_unittests.filter
  70. fuchsia.lsan.gin_unittests.filter
  71. fuchsia.lsan.media_unittests.filter
  72. fuchsia.lsan.net_unittests.filter
  73. fuchsia.lsan.perfetto_unittests.filter
  74. fuchsia.lsan.skia_unittests.filter
  75. fuchsia.lsan.views_unittests.filter
  76. fuchsia.net_perftests.filter
  77. fuchsia.net_unittests.filter
  78. fuchsia.services_unittests.filter
  79. fuchsia.storage_unittests.filter
  80. fuchsia.ui_base_unittests.filter
  81. fuchsia.unit_tests.filter
  82. fuchsia.views_examples_unittests.filter
  83. fuchsia.views_unittests.filter
  84. fuchsia.viz_unittests.filter
  85. gpu.linux.skiarenderer_dawn_cc_unittests.filter
  86. gpu.linux.skiarenderer_dawn_viz_unittests.filter
  87. gpu.skiarenderer_dawn_cc_unittests.filter
  88. gpu.skiarenderer_vulkan_blink_web_tests.filter
  89. gpu.skiarenderer_vulkan_content_browsertests.filter
  90. linux-chromeos.browser_tests.require_lacros.filter
  91. linux-lacros.browser_tests.filter
  92. linux-lacros.components_unittests.filter
  93. linux-lacros.content_browsertests.filter
  94. linux-lacros.interactive_ui_tests.filter
  95. linux-lacros.lacros_chrome_browsertests.filter
  96. linux-lacros.lacros_chrome_browsertests.skew.filter
  97. linux-lacros.sync_integration_tests.filter
  98. linux-lacros.unit_tests.filter
  99. linux_trusty_rel.browser_tests.filter
  100. mac.mac-rel.browser_tests.filter
  101. mac.mac11-arm64-rel.browser_tests.filter
  102. ozone-linux.interactive_ui_tests_wayland.filter
  103. ozone-linux.unit_tests_wayland.filter
  104. ozone-linux.wayland_browser_tests.filter
  105. ozone-linux.wayland_views_unittests.filter
  106. pixel_browser_tests.filter
  107. site_isolation_android.content_browsertests.filter
  108. vulkan.content_browsertests.filter
  109. webrtc.content_unittests.filter
  110. webrtc_functional.browser_tests.filter
  111. win_backuprefptr_fyi.browser_tests.filter
buildbot/filters/README.md

Test filter files

Summary

This directory contains files that list tests that are not yet ready to run in a particular mode. For example - the cast-linux.content_browsertests.filter file lists tests that should be excluded when running content_browsertests on the Cast device or bot (e.g. on 'Cast Linux).

File syntax

Contents of test filter files follow the syntax below:

  • Empty lines are ignored.
  • Any text after ‘#’ on each line is treated as comments and ignored.
  • All other lines specify a single test name pattern.
    • Negative patterns: Patterns prefixed with the ‘-’ character specify tests to exclude from a test run.
    • Positive patterns: All other patterns specify tests to include in a test run.

A test will be executed if and only if it matches any of the positive patterns but does not match any of the negative patterns (please see documentation of --gtest_filter for more details and examples).

Please see [https://crbug.com/587527] for discussion how “positive” and “negative” test patterns should be combined in presence of both the --gtest_filter and the --test-launcher-filter-file command line flags.

Example test filter file for excluding a set of tests:

# crbug.com/417518: Get tests working w/ --site-per-process
-BrowserTest.OtherRedirectsDontForkProcess
-ChromeRenderProcessHostTest.*
-ReferrerPolicyTest.HttpsRedirect

# crbug.com/448592: Get extension browsertests working w/ --site-per-process
-IsolatedAppTest.CookieIsolation
-IsolatedAppTest.CrossProcessClientRedirect
-IsolatedAppTest.IsolatedAppProcessModel
-IsolatedAppTest.SubresourceCookieIsolation

Usage

When running tests on desktop platforms, the test filter file can be specified using --test-launcher-filter-file command line flag. Example test invocation (single filter file):

$ out/dbg/content_browsertests \
    --test-launcher-filter-file=testing/buildbot/filters/foo.content_browsertests.filter

Example test invocation (multiple filter files, separated by ‘;’):

$ out/dbg/content_browsertests \
    --test-launcher-filter-file=testing/buildbot/filters/foo.content_browsertests.filter;\
                                testing/buildbot/filters/foo.chromeos.content_browsertests.filter

When running tests on Android, the test filter file can also be specified using --test-launcher-filter-file command line flag. Example test invocation:

$ out/android/bin/run_content_browsertests \
    --test-launcher-filter-file=testing/buildbot/filters/foo.content_browsertests.filter

Multiple filter files

We are in the process of unifying the way multiple filter files are passed across all test runners.

Though our standard for passing multiple filter files is to pass each one in its own flag, we still support ‘;’ for passing multiple filter files on Android for legacy reasons and haven‘t implemented multiple filter flags in Gtest. Gtest’s current problematic behavior leads to users passing 2 flags, but only the latter being respected.

Gtest

Multiple filter files should be separated by a ‘;’, passed with one flag.

Web tests and Android

Multiple filter files should be passed with multiple filter flags.

Applicability

Test filter files described here are currently only supported for gtest-based tests.

For excluding layout tests when running with a particular command line flag, see third_party/WebKit/LayoutTests/FlagExpectations/README.txt.

Adding new test filter files

Please use the following conventions when naming the new file:

  • Please include the name of the test executable (e.g. content_browsertests).
  • Please use .filter suffix.
  • Feel free to add other relevant things into the file name (e.g. the mode the file applies to - for example site-per-process).

When adding a new filter file, you will need to:

  • Update //testing/buildbot/filters/BUILD.gn.
  • Add //testing/buildbot/filters:foo_filters to the appropriate test target.
  • Add '--test-launcher-filter-file=../../testing/buildbot/filters/foo.filter' to the desired test suite(s) in test_suites.pyl.
  • Run testing/buildbot/generate_buildbot_json.py to update .json files.