vp8,update_extracfg: add missing setjmp

Fixes an assert from a `CHECK_MEM_ERROR` when using
`VP8E_SET_SCALEMODE`. The additional change modifying the branch
structure is to avoid a `-Wclobbered` warning from
`update_error_state()` when the function is inlined.

Bug: 506017558
Change-Id: Ibd190161251d6bbbd4076beb43c6e4e6745045c8