commit ee9cb9809bd878bad9d03c5a6befdcfa4d60209b
author Robert Iannucci <iannucci@chromium.org> Fri Jan 17 19:54:14 2025
committer LUCI CQ <infra-scoped@luci-project-accounts.iam.gserviceaccount.com> Fri Jan 17 19:54:14 2025
tree 37620c3072e7528bc90585190d04637c95c282af
parent 534ba3a709022ce0ebd75078dd43de2c969b542e

[cipd] Add per-RPC timeout to inner gcs.Reader.ReadAt call.

We've spotted this Reader implementation taking up to 20 minutes
without timing out while reading 64MB of data. When this happens,
the CIPD verification backend takes an extraordinary amount of time
(e.g. greater than an hour), leading to client side timeouts.

We're not certain what the root underlying cause is, but we are sure
that these RPCs should never be allowed to take this long, and we
see evidence that retrying can lead to successful (and reasonably
fast) RPC responses.

This CL computes a timeout for the RPC based on the theoretical
amount of time it SHOULD take to transfer the requested chunk (plus
an extra 500ms of headroom). If the timeout hits, it's treated as
any other transient error and the chunk will be retried.

We're hoping that this will be enough to allow the verification
backend to make forward progress, even in the event of apparently
arbitrarily slow responses from GCS.

R=bryner, fancl, vadimsh

Bug: 40202444, 361711863
Change-Id: I4c768dea4d07694c242124bcf14d642c4dcc2367
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/6181407
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Chenlin Fan <fancl@chromium.org>
Reviewed-by: Brian Ryner <bryner@google.com>