vp8/{ratectrl,onyx_if}: fix some signed integer overflows

in calculations involving bitrate in encode_frame_to_data_rate() and
vp8_compute_frame_size_bounds()

note this isn't exhaustive, it's just the result of a vpxenc run with:
-w 800 -h 480 --cpu-used=8 --rt --target-bitrate=1400000000

Bug: b/151945689
Change-Id: I3a4f878046fcf80e87482761588c977c283ae917
2 files changed