commit a4f2cae8c809ce14def29093b55e74ee6d62cb79
author Vadim Shtayura <vadimsh@chromium.org> Mon Mar 04 19:53:08 2024
committer LUCI CQ <infra-scoped@luci-project-accounts.iam.gserviceaccount.com> Mon Mar 04 19:53:08 2024
tree b153c9996cd0c641446ab06730473eccc994be75
parent f130d7db4d8c99353be51a4f86332c566361ae43

[swarming] Store task text output in a file to avoid OOMs.

Avoid keeping task's stdout in memory as much as possible. Instead
write it to a file and read this file back when necessary. Note that
when using `-task-output-stdout json` we still need to load the
entirety of task's stdout in memory in order to put it into JSON
(because we need to make sure it is properly escaped as a JSON string
and we currently can't do this in a streaming fashion and this is
likely an overkill anyway).

As an alternative, there's a new `-task-output-stdout dir:<path>`
mode. When this flag is passed, tasks's text output is stored in
`<path>/<task-ID>.txt` files. `<path>` defaults to `-output-dir`
(i.e. when fetching isolated outputs somewhere, passing
`-task-output-stdout dir` will also fetch text outputs at the same
directory).

There's one other minor observable change: when using
`-task-output-stdout console` tasks' output is written into
os.Stdout as is, without extra UTF-8 validation (that way we just
copy bytes from one file into another, which doesn't require much
memory). It should be fine, since this is how Swarming CLI worked
until few recent commits that introduced UTF-8 validation.

Likewise, files produced in `-task-output-stdout dir:<path>` mode
contain raw Swarming text output with potentially invalid UTF-8.

We still do UTF-8 validation when writing to JSON though (because
the protobuf library wants valid UTF-8).

Increase default fetch concurrency now that each fetch eats
less memory.

R=chanli@chromium.org, randymaldonado@google.com
BUG=40276579

Change-Id: I586e178f601dbdf7f53e34387cc3d64cb0c681cd
Reviewed-on: https://chromium-review.googlesource.com/c/infra/luci/luci-go/+/5336479
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>