commit d920ebdcf70a4bd9dbd52e185d703632676e5589
author Vadim Shtayura <vadimsh@chromium.org> Thu Jan 06 23:14:09 2022
committer LUCI CQ <infra-scoped@luci-project-accounts.iam.gserviceaccount.com> Thu Jan 06 23:14:09 2022
tree fec1671d2a12009061dbc8c8f41ca8eed3cc2ce5
parent 88ff990418bfdd72446d446910acf8cec670c7ae

[auth] Shard deflated AuthDB when storing it in datastore.

This affects GAE v1 apps only (GAE v2 apps don't use datastore
for AuthDB).

AuthDBDeflated quickly approaches 1MB datastore limit. When it is
hit, auth group updates will stop propagating. In this CL we now
split AuthDBDeflated blob into multiple separate entities to avoid
hitting the limit.

There are some nuances related to the rollout and rollbacks: old
code doesn't know about ShardIDs field and can't use it. As soon as
we *actually* start splitting AuthDBDeflated into shards, old code
versions will stop working with errors like:

  failed to update instance of *authdb.SnapshotDB ::
  {"error":"gae: cannot load field \"ShardIDs\" into a
  \"authdbimpl.Snapshot\": no such struct field (and 1
  other error)"}

It means if we need to rollback a service to a version that
doesn't understand sharded AuthDB, we are screwed.

For that reason this CL should be deployed and stabilized
*everywhere* before we actually start splitting AuthDB into
shards, because once we have sharded AuthDB in the datastore,
there's no way to rollback to old code.

Unfortunately we have little control over AuthDB size. The only
thing we can control is the sharding threshold and per-shard sizes.
This CL sets them to 1020 KiB. The hard datastore limit is 1024 KiB.
The current AuthDB size is 962 KiB. So we have 58 KiB of room to grow
before rollbacks may become problematic.

R=cjacomet@google.com
BUG=1284778

Change-Id: Ia2c180ac914d91c024dce774816268ff80874a24
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/3369024
Reviewed-by: Camilo Jacomet <cjacomet@google.com>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>