bsdiff: support for lazy reading from extent files in bspatch
Previously, bspatch used to read the whole content of a file to be
patched into a memory buffer, only to copy this data while generating
the new file content, which is written to a second buffer. The said
read buffer entails an unnecessary memory allocation requirement,
especially since it's being indexed almost linearly. This behavior was
further extended to support extents (i.e. a list of <offset, length>
pairs), which are used extensively during Chrome OS updates.
This change introduces extent files, which let users open files through
a layer of extents as ordinary glibc file handles. This in turn allows
us to easily convert reads from a memory buffer into direct reads from
an extent file. Extent files are buffered on the outer level (done for
us by glibc), but otherwise use a system file descriptor for the
underlying I/O; this proved to be the most efficient combination when
applying actual update payloads. Since we are reading a single byte at
a time using fread(2), and since the program is decidedly
single-threaded, we shift to using _unlocked variants, which appear to
reduce the total update time significantly without otherwise affecting
memory/CPU footprint.
We expect this to cut bspatch's memory usage by nearly one half. Note
that in general it is possible to use the same abstraction for
implementing direct writing to the target file; however, due to the way
we implement delta updates, there is risk that such writes might clobber
the read data, and so further support is needed to mark safe operations
(i.e. no read/write dependencies) as such.
This CL obsoletes the previous ebuild patch for handling extent
arguments, which is therefore removed. It also (i) gets rid of logic for
special handling of /dev/fd filenames, which is deemed redundant; (ii)
fixes the Makefile (via a separate patch) and changes the ebuild to use
it, for uniformity; (iii) updates the ebuild to EAPI version 4; (iv)
sets -Wall -Werror and fixes eliminates a warning due to bsdiff.c; (v)
enhances man pages for both bsdiff/bspatch.
BUG=chromium:229705
TEST=Passes update engine unittests with new bspatch
TEST=delta payload with BSDIFF operations updates correctly on x86-alex
Change-Id: I4bb4afa42e43279048093e7a7f0ef96406b0c9e0
Reviewed-on: https://gerrit.chromium.org/gerrit/49595
Reviewed-by: Gilad Arnold <garnold@chromium.org>
Tested-by: Gilad Arnold <garnold@chromium.org>
Commit-Queue: Gilad Arnold <garnold@chromium.org>
9 files changed
