pgo: Run per story benchmarks on android

Each telemetry benchmark has one or more stories associated with it. PGO
on Android works by running an entire benchmark and all its associated
stories and then copying over the files in
/data/data/<package_name>/cache/pgo_profiles/*.profraw and merging them
into a .profdata file. The .profdata files from various benchmark runs
are then merged into a final profile.profdata file which is used in
PGO-optimized builds.

The telemetry platform code for Android starts the Android browser at
the beginning of each story and sets up it profile. During profile setup
all the files (except those in lib/) in /data/data/<package_name>/* are
deleted. After each story run completes, the PGO profile data associated
with that story run are saved to
/data/data/<package_name>/cache/pgo_profiles/*.profraw files. This means
that any subsequent stories start by clearing the PGO profile data
collected during the preceding story run. Running any benchmark with
more than one story results in only the PGO profile data from the last
story in that benchmark being kept at the end of the benchmark.

In order to obtain more comprehensive PGO profile coverage, this CL (on
Android only) splits each benchmark by its stories, so that each story
gets its own explicit benchmark run, independent from the others. This
also has the benefit of enabling per-story repeats which in turn greatly
reduces flakiness associated with any one of the stories failing causing
the entire benchmark to fail. In principle this has the same effect for
a multi-story benchmark as it does for a multi-benchmark suite. See
https://crrev.com/c/5680092 for an example of improving from 13% success
rate to 96% success rate just by adding repeats.

This CL also addresses flakiness in PGO profile generation and merging
on the LLVM side. Either profile merging would fail explicitly with an
error code, or it would succeed but print out a warning to stderr that
it produced an invalid profile. This CL ensures that the benchmark or
story is repeated in both failure modes, as otherwise the failure
propagates to the final merge of up to 16 *.profdata files, and that
final merge will fail if any of the 16 *.profdata files are invalid
individually.

This CL also adds a passthrough for the -q flag to run_benchmark as that
greatly reduces the noisiness of run_benchmark for successful runs.
This way devs can focus on failure cases instead of length output from
successful runs.

This CL extends handling for Temporal PGO to all `llvm-profdata merge`
commands. This codepath will be tested and extended in a later CL.

The new --print-only-runnable flag for run_benchmark is used to collect
the list of stories to run individually.

Bug: 352333007
Change-Id: Ia5f839bc79bcc534c4df38b4366550bf00129229
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/5718532
Auto-Submit: Peter Wen <wnwen@chromium.org>
Commit-Queue: Peter Wen <wnwen@chromium.org>
Reviewed-by: Andrew Grieve <agrieve@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1329121}
1 file changed
tree: b3b17650b1cbf9923a055bc20a2f3b1853043245
  1. android_webview/
  2. apps/
  3. ash/
  4. base/
  5. build/
  6. build_overrides/
  7. buildtools/
  8. cc/
  9. chrome/
  10. chromecast/
  11. chromeos/
  12. codelabs/
  13. components/
  14. content/
  15. courgette/
  16. crypto/
  17. dbus/
  18. device/
  19. docs/
  20. extensions/
  21. fuchsia_web/
  22. gin/
  23. google_apis/
  24. google_update/
  25. gpu/
  26. headless/
  27. infra/
  28. ios/
  29. ipc/
  30. media/
  31. mojo/
  32. native_client_sdk/
  33. net/
  34. pdf/
  35. ppapi/
  36. printing/
  37. remoting/
  38. rlz/
  39. sandbox/
  40. services/
  41. skia/
  42. sql/
  43. storage/
  44. styleguide/
  45. testing/
  46. third_party/
  47. tools/
  48. ui/
  49. url/
  50. webkit/
  51. .clang-format
  52. .clang-tidy
  53. .clangd
  54. .eslintrc.js
  55. .git-blame-ignore-revs
  56. .gitallowed
  57. .gitattributes
  58. .gitignore
  59. .gitmodules
  60. .gn
  61. .mailmap
  62. .rustfmt.toml
  63. .vpython3
  64. .yapfignore
  65. ATL_OWNERS
  66. AUTHORS
  67. BUILD.gn
  68. CODE_OF_CONDUCT.md
  69. codereview.settings
  70. CPPLINT.cfg
  71. DEPS
  72. DIR_METADATA
  73. LICENSE
  74. LICENSE.chromium_os
  75. OWNERS
  76. PRESUBMIT.py
  77. PRESUBMIT_test.py
  78. PRESUBMIT_test_mocks.py
  79. README.md
  80. WATCHLISTS
README.md

Logo Chromium

Chromium is an open-source browser project that aims to build a safer, faster, and more stable way for all users to experience the web.

The project's web site is https://www.chromium.org.

To check out the source code locally, don't use git clone! Instead, follow the instructions on how to get the code.

Documentation in the source is rooted in docs/README.md.

Learn how to Get Around the Chromium Source Code Directory Structure.

For historical reasons, there are some small top level directories. Now the guidance is that new top level directories are for product (e.g. Chrome, Android WebView, Ash). Even if these products have multiple executables, the code should be in subdirectories of the product.

If you found a bug, please file it at https://crbug.com/new.