vp9_ratectrl: fix some signed integer overflows

in calculations involving bitrate in vp9_rc_postencode_update() and
calc_pframe_target_size_one_pass_vbr()

note this isn't exhaustive, it's just the result of a vpxenc run with:
-w 800 -h 480 --cpu-used=8 --rt --target-bitrate=1400000000

Bug: b/151945689
Change-Id: I941a77340fd44b09fc965dd182d7aeab9f1f3da0
1 file changed