vp9_rdopt: fix integer sanitizer warnings

implicit conversion from type 'int' of value -9 (32-bit, signed) to type
'uint8_t' (aka 'unsigned char') changed the value to 247 (8-bit, unsigned)

BUG=webm:1615

Change-Id: Ic2254ef4312f349ee38ec6e12a56b2cd5714b101
1 file changed