commit 7226f9435afe250b14abe22d33dcc2ba72bb4116
author Ned Nguyen <nednguyen@google.com> Thu Dec 06 17:24:03 2018
committer Commit Bot <commit-bot@chromium.org> Thu Dec 06 17:24:03 2018
tree 3e8ad3f7278deb7f02d1fd00f830d5b568e9caf4
parent 7b27d290eab4ed6d50a84f1ddf959e82039afeff

Improve error messaging of perf_data_generator when benchmarks scheduled state is incorrect

For Telemetry or non Telemetry benchmarks, these two states generally need to match:
1) The states of all benchmarks that are tracked BenchmarkMetadata objects. This
will results in benchmarks being generated in https://bit.ly/chrome-benchmarks
list.
2) The states of all benchmarks that are scheduled, either directly (for
non Telemetry benchmarks) as a 'script' in src-side json, or indirectly through
benchmark shard map files.

The only exception to this is when users add a new Telemetry benchmark but need
to defer the job of scheduling it to Telemetry/Chrome Client Infrastructure team
member.

This CL improves perf_data_generator check to make it clear which cases of
inconsistent states are happening & what to do about each of them.

For example, assuming a user rename blink_perf.paint benchmark to blink_perf.paint2

Without this CL, the error message is:

AssertionError: Please update NON_TELEMETRY_BENCHMARKS as below:
Remove blink_perf.paint2 from NON_TELEMETRY_BENCHMARKS
Add blink_perf.paint to NON_TELEMETRY_BENCHMARKS

With this CL, the error message is:

* Telemetry benchmark blink_perf.paint2 exists but is not scheduled to
run. Rename it to UNSCHEDULED_blink_perf.paint2, then file a crbug
against Telemetry and ChromeClient Infrastructure team to schedule the
benchmark on the perf waterfall.

* Telemetry benchmark blink_perf.paint no longer exists but is
scheduled. File a bug against Telemetry and/or Chrome Client
Infrastructure team to remove the corresponding benchmark class and
deschedule the benchmark on the perf waterfall. After that, you can
safely remove the benchmark's dependency code, e.g: stories, WPR
archives, metrics,..


Bug: 911828
Change-Id: Ia9e9746391eb1e9932c8e0619a25e3273a77e9c5
Reviewed-on: https://chromium-review.googlesource.com/c/1363681
Commit-Queue: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Emily Hanley <eyaich@chromium.org>
Cr-Commit-Position: refs/heads/master@{#614396}