Use "UPLOADED" to determine what artifacts are available on Google Storage
This change determines whether a file is available on Google Storage
(GS) by looking up the filenames listed in the "UPLOADED" in the same
bucket.
Currently the dev server uses "gsutils ls" to determine whether a file
is available on GS. This approach is flaky because of the following two
reasons.
1. GS provides strong consistenty for objects reads/writes, but only
(non-time bound) eventual consistency for bucket listing performed by
"ls". This means that the the dev server may be waiting for an object to
show up in the listing indefinitely.
2. GS does not guarantee causal consistency, which means that the dev
server may see the new bucket listing once but get an older listing
afterwards. This is caused by the multiple replicas used in distributed
caching system in GS.
Instead of relying on "gsutils ls", I have patched cbuildbot to maintain
a manifest of uploaded artifacts and store the manifest (named UPLOADED)
in the same bucket. This change allows the dev server to poll UPLOADED
for the available artifacts and to check whether a specific artifact is
available (e.g. the full payload) by using a pattern to match names in
UPLOADED.
BUG=chromium-os:32361
TEST=unittest + local dev server
Change-Id: Ic43a5be20840fb9f70a65ca848fe22f2dc701a3d
Reviewed-on: https://gerrit.chromium.org/gerrit/30573
Commit-Ready: Yu-Ju Hong <yjhong@chromium.org>
Reviewed-by: Yu-Ju Hong <yjhong@chromium.org>
Tested-by: Yu-Ju Hong <yjhong@chromium.org>
2 files changed
