Skip numpy tests when numpy is not available (#298)

Make it possible to successfully run tests without numpy installed.
This package is becoming quite hard profile, and being able to test it
without building numpy first would be a nice feature.