vp8: Do not zero-initialize the encoder token buffer

vp8_alloc_compressor_data() allocates cpi->tok for the worst case of
mb_rows * mb_cols * 384 tokens (16 bytes each), e.g. ~59 MB at
1920x1280 and ~200 MB at 3840x2160. It used vpx_calloc(), which memsets
the whole buffer, so every page became resident at encoder creation even
though a frame typically uses only a small fraction of it.

The buffer never needs zeroing: tokens are written during tokenization,
and the bitstream packer only reads back the [start, stop) range of each
macroblock row (or [tok, tok + tok_count) in the single-thread path).
Switch to vpx_malloc() with an explicit size overflow check.

At 1920x1280, 4 Mbps, peak RSS of a single-threaded vpxenc encode drops
from ~98 MB to ~51 MB, and the RSS right after encoder init drops from
~63 MB to ~7 MB. The output bitstreams are bit-identical, and
MemorySanitizer reports no uninitialized reads with 1 and 4 threads.
With multiple threads each macroblock row starts writing at its own
offset in the buffer, so with transparent huge pages enabled most of
the buffer still becomes resident; without THP the 4-thread peak RSS
also drops from ~98 MB to ~47 MB.

This was seen in a container OOM where the 4 GB limit was hit inside
vp8_alloc_compressor_data() while creating a VP8 encoder through
OpenCV's VideoWriter.

TAG=agy

Change-Id: I045f338d40af02651b175ada8446cb4f92311afc
1 file changed