vp9 encoder: fix integer overflows

fixing integer overflow with 16K content and enabling the test

Bug: webm:1750
Fixed: webm:1750
Change-Id: I76eebd915bcae55bc755613251a98e1716dea4c0
5 files changed