Keywords: This testing document describes how to run GLBench, TearTest, Piglit, WebGLConformance and other tests and benchmarks on ChromeOS and ChromiumOS devices.
Python sources are under src/third_party/autotest/files/client/site_tests
drwxr-xr-x-graphics_GLAPIChecktree | historydrwxr-xr-x-graphics_GLBenchtree | historydrwxr-xr-x-graphics_O3DSeleniumtree | historydrwxr-xr-x-graphics_SanAngelestree | historydrwxr-xr-x-graphics_TearTesttree | historydrwxr-xr-x-graphics_WebGLConformancetree | historydrwxr-xr-x-graphics_WindowManagerGraphicsCapturetree | history
Binary dependencies are under src/third_party/autotest/files/client/deps
drwxr-xr-x-glbench-imagestree | historydrwxr-xr-x-glbenchtree | historydrwxr-xr-x-piglittree | history
The ebuild system knows if a target knows either OpenGL or OpenGL ES, or even both. If only pre-compiled binaries are used there should not be much of a problem with running tests. When editing tests after using the cros_workon script the default system behavior is to recompile the binaries once more without invoking the ebuild system. Unfortunately this means USE=“opengles” flags are not honored. Instead the easy way is to specify the proper switch via am environment variable GRAPHICS_BACKEND=OPENGLES./run_remote_tests.sh that gets directly passed to the Makefile. This is somewhat hacky. To use the ebuild system to generate the binaries one has to do a painfully slow USE=opengles emerge-tegra2_seaboard chromeos-base/autotest followed by ./run_remote_tests.sh --use_emerged [...]. Don't forget the --use_emerged flag as it is not default.
-- GRAPHICS_BACKEND=OPENGL or OPENGLES
runs 25 seconds
sample results
for OPENGL creates output below and parses some samples in python to see if they are there and if SUCCEED: run to the end is there
GL_VERSION = 3.2.0 NVIDIA 195.36.24
GL_EXTENSIONS = GL_ARB_color_buffer_float GL_ARB_compatibility GL_ARB_copy_buffer GL_ARB_depth_buffer_float GL_ARB_depth_clamp GL_ARB_depth_texture GL_ARB_draw_buffers GL_ARB_draw_elements_base_vertex [...] NV-GLX NV-CONTROL Generic
Event Extension SHAPE MIT-SHM XInputExtension XTEST BIG-REQUESTS SYNC XKEYBOARD XC-MISC SECURITY XINERAMA XFIXES RENDER RANDR XINERAMA Composite DAMAGE
SUCCEED: run to the end
(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.71.194 graphics_GLAPIChec****k --use_emerged
Initiating first contact with remote host
[...]
INFO : Test results:
graphics_GLAPICheck PASS
graphics_GLAPICheck/graphics_GLAPICheck PASS
Total PASS: 2/2 (100%)
No crashes detected during testing.
Elapsed time: 0m25s
draws green/ref/purple/fractal rectangles
runs about 5 minutes
This benchmark executes glbench, a graphics benchmark designed to time how long
various graphic intensive activities take, which includes measuring:
fill rate
blended
opaque
-Z reject rate
-triangle rate
no cull
half cull (half triangles backface culled)
full cull (mix of back face and degenerates)
blend rate
texture fetch
nearest
bilinear
trilinear
compute
vertex shader
pixel shader
*fragement shader to test ddx and ddy
attribute fetch
color depth stencil test
*state change
texture upload
read back
does MD5 checksums of some but not all images
-- deps/glbench/src/checksums
-- can be re-generated with -save option
(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGLES ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.27 graphics_GLBench --use_emerged
[...]
INFO : Test results:
graphics_GLBench PASS
graphics_GLBench/graphics_GLBench PASS
1280x768_fps_compositing 11.18
1280x768_fps_no_fill_compositing 41.96
mpixels_sec_clear_color 490.74
mpixels_sec_clear_colordepth 323.88
mpixels_sec_clear_colordepthstencil 281.45
mpixels_sec_clear_depth 969.71
mpixels_sec_clear_depthstencil 647.34
mpixels_sec_fill_solid 285.1
mpixels_sec_fill_solid_blended 128.55
mpixels_sec_fill_solid_depth_neq 973.2
mpixels_sec_fill_solid_depth_never 1189.08
mpixels_sec_fill_tex_bilinear 149.56
mpixels_sec_fill_tex_nearest 203.7
mpixels_sec_fill_tex_trilinear_linear_01 164.63
mpixels_sec_fill_tex_trilinear_linear_04 nan
mpixels_sec_fill_tex_trilinear_linear_05 nan
mpixels_sec_fill_tex_trilinear_nearest_05 nan
mpixels_sec_pixel_read 6.72
mpixels_sec_pixel_read_2 6.72
mpixels_sec_pixel_read_3 6.74
mpixels_sec_varyings_shader_1 270.68
mpixels_sec_varyings_shader_2 144.33
mpixels_sec_varyings_shader_4 73.57
mpixels_sec_varyings_shader_8 24.29
mpixels_sec_yuv_shader_1 nan
mpixels_sec_yuv_shader_2 47.82
mpixels_sec_yuv_shader_3 93.36
mpixels_sec_yuv_shader_4 136.15
mtexel_sec_texture_update_teximage2d_1024 149.85
mtexel_sec_texture_update_teximage2d_128 80.92
mtexel_sec_texture_update_teximage2d_1536 146.05
mtexel_sec_texture_update_teximage2d_2048 149.47
mtexel_sec_texture_update_teximage2d_256 132.51
mtexel_sec_texture_update_teximage2d_32 10.29
mtexel_sec_texture_update_teximage2d_512 142.05
mtexel_sec_texture_update_teximage2d_768 147.56
mtexel_sec_texture_update_texsubimage2d_1024 150.94
mtexel_sec_texture_update_texsubimage2d_128 81.59
mtexel_sec_texture_update_texsubimage2d_1536 147.2
mtexel_sec_texture_update_texsubimage2d_2048 149.03
mtexel_sec_texture_update_texsubimage2d_256 130.59
mtexel_sec_texture_update_texsubimage2d_32 10.09
mtexel_sec_texture_update_texsubimage2d_512 113.54
mtexel_sec_texture_update_texsubimage2d_768 150.0
mtri_sec_triangle_setup 30.34
mtri_sec_triangle_setup_all_culled 49.89
mtri_sec_triangle_setup_half_culled 44.91
mvtx_sec_attribute_fetch_shader 149.51
mvtx_sec_attribute_fetch_shader_2_attr 149.34
mvtx_sec_attribute_fetch_shader_4_attr 141.36
mvtx_sec_attribute_fetch_shader_8_attr 106.71
us_swap_swap 19464.19
Total PASS: 2/2 (100%)
No crashes detected during testing.
Elapsed time: 8m58s
Details stored under /tmp/run_remote_tests.purz
The “nan” in this example are passed because of knownbad output images. For more more details and failures look at
(cros-chroot) ihf@ql ~/trunk/src/scripts $ more /tmp/run_remote_tests.purz/graphics_GLBench/graphics_GLBench/summary.txt
Sometimes when site_tests/suite_HWQual/control.auto is run a constraint is violated and failure **mtri_sec_triangle_setup >= 10 **is thrown. This is due to some x86 boards doing vertex shading in the CPU. This exception is considered harmless until a waiver system is developed.
tries to run O3D tests inside of chrome
failed to run and is known to be broken
tests OpenGL implementation. Does not currently work on OpenGL ES only systems like Tegra2
about 1000-2000 targeted subtests
usually a few hundred failures
mostly care about counts of passes/failures over time like perf test
runs about 12-18 minutes, but initial copying dep-piglit.tar.bz2 takes 5-10 minutes that seem like a hang. Need to fix this.
generates html page with detailed results in
/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html
./piglit-summary-html.py summary/compare results/baseline.results results/current.results
can be build locally from piglit.tar.gz via cmake .;make
the numbers of passes does fluctuate by about 10 over time, so some intermittency
about 10 test binaries crash and are blacklisted to not be reported
sometimes (1 in about 2 weeks of test runs) fbo_depth_sample_compare takes gpu/system with it when it crashes
2011-05-04T11:06:02.995237-07:00 localhost kernel: [ 75.994363] fbo-depth-sampl[4250]: segfault at 0 ip (null) sp 7fc7abac error 14 in card0[75f97000+4000] 2011-05-04T11:06:03.040387-07:00 localhost crash_reporter[4251]: Received crash notification for fbo-depth-sample-compare[4250] sig 11 (developer build - not testing - always dumping) 2011-05-04T11:06:03.234936-07:00 localhost crash_reporter[4251]: Stored minidump to /var/spool/crash/fbo_depth_sample_compare.20110504.110603.4250.dmp 2011-05-04T11:06:03.235566-07:00 localhost crash_reporter[4251]: Leaving core file at /var/spool/crash/fbo_depth_sample_compare.20110504.110603.4250.core due to developer image 2011-05-04T11:06:17.742377-07:00 localhost kernel: [ 90.742080] [drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung 2011-05-04T11:06:17.742439-07:00 localhost kernel: [ 90.742100] render error detected, EIR: 0x00000000 2011-05-04T11:06:17.742471-07:00 localhost kernel: [ 90.742168] [drm:i915_do_wait_request] *ERROR* i915_do_wait_request returns -5 (awaiting 29946 at 29941) 2011-05-04T11:06:18.330692-07:00 localhost crash_reporter[4328]: Received crash notification for Xorg[3498] sig 6 (developer build - not testing - always dumping)
(cros-chroot) ihf@ql ~/trunk/src/scripts $ ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_Piglit** --use_emerged**
[...]
INFO : Test results:
graphics_Piglit PASS
graphics_Piglit/graphics_Piglit PASS
count_subtests_fail 239
count_subtests_pass 1040
count_subtests_skip 687
count_subtests_warn 14
Total PASS: 2/2 (100%)
No crashes detected during testing.
Elapsed time: 13m31s
Details stored under /tmp/run_remote_tests.8gjT
Point browser to
/tmp/run_remote_tests.*/graphics_Piglit/graphics_Piglit/cros-driver/html/index.html
This test runs the San Angeles Observation GPU benchmark. This benchmark uses
a minimal and portable framework to generate a small demo program. It exercises
basic features of OpenGL like vertex arrays, color arrays, and lighting. It
also uses objects generated using procedural algorithms.
This test is a benchmark. It will fail if it fails to complete.
(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_SanAngeles** --use_emerged**
[...]
INFO : Test results:
graphics_SanAngeles PASS
graphics_SanAngeles/graphics_SanAngeles PASS
frames_per_sec_rate_san_angeles 2.1
Total PASS: 2/2 (100%)
No crashes detected during testing.
Elapsed time: 2m17s
Details stored under /tmp/run_remote_tests.vDwN
copies glbench as dependency over
runs deps/glbench/teartest
This test will fail if there is tearing in the two vertical lines that are
scrolling horizontally.
This is a semi-automated test that displays vertical lines scrolling
horizontally and asks the user if tearing was observed. Three variants are
available:
* using uniform update. This tests that glSwapInterval function performs as
expected.
* using full texture update. This tests that CPU-GPU interaction is properly
synchronized in the driver.
* using pixmap to texture extension. This tests that pixmap to texture
extension is properly synchronized.
(cros-chroot) ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_TearTest** --use_emerged**
[...] - some user interaction
INFO : Test results:
graphics_TearTest PASS
graphics_TearTest/graphics_TearTest PASS
Total PASS: 2/2 (100%)
No crashes detected during testing.
Elapsed time: 1m26s
Details stored under /tmp/run_remote_tests.SHxS
This test runs the WebGL conformance tests: http://www.khronos.org/webgl/wiki/Testing/Conformance
currently uses release version 1.0.0
opens copy of above website in chrome and executes the html page with hundreds of targeted tests
all suite tests must pass, but there is the ability to waive failures in the python script
javascript is patched to report individual results
suite is currently blocked in autotest regression runs by hangs/crashes in browser
Doesn‘t compile any code, so no GRAPHICS_BACKEND needed (but no harm in using it). When the browser doesn’t crash/hang it should look like this.
(cros-chroot) ihf@ql ~/trunk/src/scripts $ ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.80 graphics_WebGLConformance/control
[...]
INFO : Test results:
graphics_WebGLConformance PASS
graphics_WebGLConformance/graphics_WebGLConformance PASS
count_tests_fail 352
count_tests_pass 5064
count_tests_timeout 2
waived_url_000 conformance/tex-image-and-sub-image-2d-with-array-buffer-view.html : 192 failures (192 waived)
waived_url_001 conformance/tex-image-with-format-and-type.html : 12 failures (12 waived)
waived_url_002 conformance/texture-npot.html : 12 failures (12 waived)
waived_url_003 conformance/glsl-conformance.html : 1 failures (1 waived)
waived_url_004 conformance/tex-image-and-sub-image-2d-with-image.html : 8 failures (8 waived)
waived_url_005 conformance/copy-tex-image-and-sub-image-2d.html : 34 failures (34 waived)
waived_url_006 conformance/gl-clear.html : 4 failures (4 waived)
waived_url_007 conformance/more/functions/readPixelsBadArgs.html : 1 failures (1 waived)
waived_url_008 conformance/more/conformance/webGLArrays.html : 1 failures (1 waived)
waived_url_009 conformance/gl-teximage.html : 46 failures (46 waived)
waived_url_010 conformance/texture-active-bind.html : 4 failures (4 waived)
waived_url_011 conformance/read-pixels-test.html : 3 failures (3 waived)
waived_url_012 conformance/gl-object-get-calls.html : 2 failures (2 waived)
waived_url_013 conformance/point-size.html : 1 failures (1 waived)
waived_url_014 conformance/texture-formats-test.html : 4 failures (4 waived)
waived_url_015 conformance/texture-complete.html : 1 failures (1 waived)
waived_url_016 conformance/tex-image-and-sub-image-2d-with-video.html : 8 failures (8 waived)
waived_url_017 conformance/context-lost-restored.html : 2 failures (2 waived)
waived_url_018 conformance/tex-image-and-sub-image-2d-with-image-data.html : 16 failures (16 waived)
Total PASS: 2/2 (100%)
Crashes detected during testing:
chrome sig 6
graphics_WebGLConformance/graphics_WebGLConformance
Total unique crashes: 1
Details stored under /tmp/run_remote_tests.iumX
This test verifies the window manager can capture graphics from applications.
This test fails if application screen shots cannot capture the screen output.
draws less than 10 full screen color frames of a dot/sphere
copies dep-glbench.tar.bz2 as dependency over
runs deps/glbench/windowmanagertest
converts/resizes images to 100x100 pixels
runs perceptualdiff to compare with reference images
runs about 30 seconds
fairly stable test
ihf@ql ~/trunk/src/scripts $ GRAPHICS_BACKEND=OPENGL ./run_remote_tests.sh --board=${BOARD} --remote=172.22.70.69 graphics_WindowManagerGraphicsCapture** --use_emerged**
[...]
INFO : Test results:
graphics_WindowManagerGraphicsCapture PASS
graphics_WindowManagerGraphicsCapture/graphics_WindowManagerGraphicsCapture PASS
Total PASS: 2/2 (100%)
No crashes detected during testing.
Elapsed time: 0m36s
Details stored under /tmp/run_remote_tests.Lb4p