commit aaf86394e29f65af257720d5e952125615fab5fb
author Will Yeager <wyeager@google.com> Mon Mar 25 15:14:28 2024
committer Chromium LUCI CQ <chromium-scoped@luci-project-accounts.iam.gserviceaccount.com> Mon Mar 25 15:14:28 2024
tree 7971e79cc08bd4061858eedac150e85da48ea3e8
parent c6318aa5793fdedcdb5fcd84e044238017fe8989

Replace use of otool in bling test runner with -enumerate-tests

Doc: go/replace-otool-with-enum-tests

This CL was originally reviewed as a set of stacked changes, with the base here:
https://crrev.com/c/5279816
In the interest of simplifying the revert process, should anything go wrong when it starts taking effect on the main CQ, I've condensed all 4 of those CLs into this one.

This CL replaces otool with the new xcodebuild -enumerate-tests CLI. otool was previously being used from |Runner.resolve_test_cases| to list the EG tests present in the test app binary. That is now being accomplished when initializing either a SimulatorParallelTestRunner or DeviceXcodeTestRunner in the functions:
|SimulatorParallelTestRunner.fetch_test_names| and |SimulatorParallelTestRunner.resolve_test_cases|.

|Runner.resolve_test_cases| was also previously handling filtering non-EG tests based on arguments passed to the test runner, like --gtest_filter. That work is now being done when parsing CLI args.

Disabled tests also required special handling. Outputting disabled tests is one thing that otool had an easier time with than -enumerate-tests. In chromium we disable tests by prefixing the test method name with "DISABLED_" rather than doing things the "Apple approved" way of marking the test as disabled through Xcode which I think would store that information in the .xcodeproj file or a scheme. This means that -enumerate-tests can't find the disabled tests. To work around this limitation I've applied the same logic that we use for running only FLAKY_ prefixed test methods. This finds all disabled tests present in the test app binary, so in the case that the test suite is run across multiple swarming shards we should only bothering outputting them on the first shard.

The CL also contains a few refactors that are not necessarily related to replacing otool, but felt like clear improvements and touched related code.
- consolidating the 3 duplicate _xctest_path functions from test_apps.py into xcode_util.xctest_path
- more clearly checking for the condition which raises ExcessShardsError and improving the exception message
- adds python typing hints to a few new/updated functions

Bug: 326957933,1480192,982435,1077277
Change-Id: If2e63376a6935cd8c4f0245bb814103319a75f07
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5386304
Commit-Queue: Will Yeager <wyeager@google.com>
Reviewed-by: Zhaoyang Li <zhaoyangli@chromium.org>
Reviewed-by: Yue She <yueshe@google.com>
Reviewed-by: Lindsay Pasricha <lindsayw@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1277652}