commit 35f8e8c2e537e9c86d3ba4b4f29dd81925b75309
author Bruce Dawson <brucedawson@chromium.org> Tue Nov 29 06:00:47 2022
committer Chromium LUCI CQ <chromium-scoped@luci-project-accounts.iam.gserviceaccount.com> Tue Nov 29 06:00:47 2022
tree bfed0ea59e64ee084e21b1a8e8bc00c7eb5bd94b
parent 0a2b8d0b9b14d2abbf0dca750e51dde3e0a355ab

Make CheckWprShaFiles ~60x faster

CheckWprShaFiles is the slowest presubmit on the presubmit --all bot:
  https://ci.chromium.org/p/chromium/builders/ci/linux-presubmit

It takes 420+ s whereas the second slowest takes more like 90 s. The
slowness is because of calling cloud_storage.Exists() individually for
500+ files. The network and CPU latency is non-trivial (including the
cost of starting vpython3 for each request). It turns out that most of
these files are all in the same storage bucket and checking their status
can be optimized by calling cloud_storage.ListFiles() once and then
checking to see if the names are in the results. However this doesn't
work for all files because some buckets have enough files that calling
cloud_storage.ListFiles() literally takes hours to run.

With a little special casing of faster versus slow buckets the number
of cloud_storage calls drops by over 500, thus giving a speedup of
roughly 60x (from ~1,000 s to ~16 s on my Linux development machine).

This also updates a comment that was missed when landing
crrev.com/c/4058427.

Bug: 1393724
Change-Id: I9b26bf722628012a109a3111c376f2593d8b8e77
Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4060580
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Brian Sheedy <bsheedy@chromium.org>
Cr-Commit-Position: refs/heads/main@{#1076674}