Google Git
Sign in
chromium / angle / angle / refs/heads/chromium/5593 / . / src / tests / perf_tests
tree: 1eae5a6dee7399d93af497b926cb5a1f1ea37852 [path history] [tgz]
  1. third_party/
  2. ANGLEPerfTest.cpp
  3. ANGLEPerfTest.h
  4. ANGLEPerfTestArgs.cpp
  5. ANGLEPerfTestArgs.h
  6. AstcDecompressorPerf.cpp
  7. BindingPerf.cpp
  8. BitSetIteratorPerf.cpp
  9. BlitFramebufferPerf.cpp
  10. BufferSubData.cpp
  11. ClearPerf.cpp
  12. CompilerPerf.cpp
  13. DispatchComputePerf.cpp
  14. DrawCallPerf.cpp
  15. DrawCallPerfParams.cpp
  16. DrawCallPerfParams.h
  17. DrawElementsPerf.cpp
  18. DynamicPromotionPerfTest.cpp
  19. EGLInitializePerf.cpp
  20. EGLMakeCurrentPerf.cpp
  21. FramebufferAttachmentPerfTest.cpp
  22. GenerateMipmapPerf.cpp
  23. glmark2Benchmark.cpp
  24. ImagelessFramebufferPerfTest.cpp
  25. IndexConversionPerf.cpp
  26. IndexDataManagerTest.cpp
  27. InstancingPerf.cpp
  28. InterleavedAttributeData.cpp
  29. LinkProgramPerfTest.cpp
  30. MapBufferRange.cpp
  31. MultisampledRenderToTexturePerf.cpp
  32. MultisampledSwapchainResolve.cpp
  33. MultiviewPerf.cpp
  34. PointSprites.cpp
  35. PreRotationPerf.cpp
  36. ProgramPipelineObjectPerfTest.cpp
  37. README.md
  38. ResultPerf.cpp
  39. TextureSampling.cpp
  40. TexturesPerf.cpp
  41. TextureUploadPerf.cpp
  42. TracePerfTest.cpp
  43. UniformsPerf.cpp
  44. VertexArrayPerfTest.cpp
  45. VulkanBarriersPerf.cpp
  46. VulkanCommandBufferPerf.cpp
  47. VulkanPipelineCachePerf.cpp
src/tests/perf_tests/README.md

ANGLE Performance Tests

angle_perftests is a standalone microbenchmark testing suite that contains tests for the OpenGL API. angle_trace_tests is a suite to run captures traces for correctness and performance. Because the traces contain confidential data, they are not publicly available. For more details on ANGLE's tracer please see the docs.

The tests currently run on the Chromium ANGLE infrastructure and report results to the Chromium perf dashboard. Please refer to thepublic dashboard docs for help

Running the Tests

You can follow the usual instructions to check out and build ANGLE. Build the angle_perftests or angle_trace_tests targets. Note that all test scores are higher-is-better. You should also ensure is_debug=false in your build. Running with angle_assert_always_on or debug validation enabled is not recommended.

Variance can be a problem when benchmarking. We have a test harness to run a tests repeatedly to find a lower variance measurement. See src/tests/run_perf_tests.py.

To use the script first build angle_perftests or angle_trace_tests, set your working directory your build directory, and invoke the run_perf_tests.py script. Use --test-suite to specify your test suite, and --filter to specify a test filter.

Choosing the Test to Run

You can choose individual tests to run with --gtest_filter=*TestName*. To select a particular ANGLE back-end, add the name of the back-end to the test filter. For example: DrawCallPerfBenchmark.Run/gl or DrawCallPerfBenchmark.Run/d3d11. Many tests have sub-tests that run slightly different code paths. You might need to experiment to find the right sub-test and its name.

Null/No-op Configurations

ANGLE implements a no-op driver for OpenGL, D3D11 and Vulkan. To run on these configurations use the gl_null, d3d11_null or vulkan_null test configurations. These null drivers will not do any GPU work. They will skip the driver entirely. These null configs are useful for diagnosing performance overhead in ANGLE code.

Command-line Arguments

Several command-line arguments control how the tests run:

  • --one-frame-only: Runs tests once and quickly exits. Used as a quick smoke test.
  • --enable-trace: Write a JSON event log that can be loaded in Chrome.
  • --trace-file file: Name of the JSON event log for --enable-trace.
  • --calibration: Prints the number of steps a test runs in a fixed time. Used by perf_test_runner.py.
  • --steps-per-trial x: Fixed number of steps to run for each test trial.
  • --max-steps-performed x: Upper maximum on total number of steps for the entire test run.
  • --render-test-output-dir=dir: Directory to store test artifacts (including screenshots but unlike --screenshot-dir, dir here is always a local directory regardless of platform and --save-screenshots isn't implied).
  • --verbose: Print extra timing information.
  • --warmup-trials x: Number of times to warm up the test before starting timing. Defaults to 3.
  • --warmup-steps x: Maximum number of steps for the warmup loops. Defaults to unlimited. Set to -1 to use a trace's frame count.
  • --no-warmup: Skip warming up the tests. Equivalent to --warmup-steps 0.
  • --calibration-time: Run each test calibration step in a fixed time. Defaults to 1 second.
  • --trial-time x or --max-trial-time x: Run each test trial under this max time. Defaults to 10 seconds.
  • --fixed-test-time x: Run the tests until this much time has elapsed.
  • --fixed-test-time-with-warmup x: Start with a warmup trial, then run the tests until this much time has elapsed.
  • --trials: Number of times to repeat testing. Defaults to 3.
  • --no-finish: Don't call glFinish after each test trial.
  • --validation: Enable serialization validation in the trace tests. Normally used with SwiftShader and retracing.
  • --perf-counters: Additional performance counters to include in the result output. Separate multiple entries with colons: ‘:’.

The command line arguments implementations are located in ANGLEPerfTestArgs.cpp.

Test Breakdown

Microbenchmarks

  • DrawCallPerfBenchmark: Runs a tight loop around DrawArarys calls.
    • validation_only: Skips all rendering.
    • render_to_texture: Render to a user Framebuffer instead of the default FBO.
    • vbo_change: Applies a Vertex Array change between each draw.
    • tex_change: Applies a Texture change between each draw.
  • UniformsBenchmark: Tests performance of updating various uniforms counts followed by a DrawArrays call.
    • vec4: Tests vec4 Uniforms.
    • matrix: Tests using Matrix uniforms instead of vec4.
    • multiprogram: Tests switching Programs between updates and draws.
    • repeating: Skip the update of uniforms before each draw call.
  • DrawElementsPerfBenchmark: Similar to DrawCallPerfBenchmark but for indexed DrawElements calls.
  • BindingsBenchmark: Tests Buffer binding performance. Does no draw call operations.
    • 100_objects_allocated_every_iteration: Tests repeated glBindBuffer with new buffers allocated each iteration.
    • 100_objects_allocated_at_initialization: Tests repeated glBindBuffer the same objects each iteration.
  • TexSubImageBenchmark: Tests glTexSubImage update performance.
  • BufferSubDataBenchmark: Tests glBufferSubData update performance.
  • TextureSamplingBenchmark: Tests Texture sampling performance.
  • TextureBenchmark: Tests Texture state change performance.
  • LinkProgramBenchmark: Tests performance of glLinkProgram.
  • glmark2: Runs the glmark2 benchmark.

Many other tests can be found that have documentation in their classes.

Trace Tests

Trace tests take command line arguments that pick the run configuration:

  • --use-gl=native: Runs the tests against the default system GLES implementation instad of your local ANGLE.
  • --use-angle=backend: Picks an ANGLE back-end. e.g. vulkan, d3d11, d3d9, gl, gles, metal, or swiftshader. Vulkan is the default.
  • --offscreen: Run with an offscreen surface instead of swapping every frame.
  • --vsync: Run with vsync enabled, and measure CPU and GPU work insead of wall clock time.
  • --minimize-gpu-work: Modify API calls so that GPU work is reduced to minimum.
  • --screenshot-dir dir: Directory to store test screenshots. Implies --save-screenshots. On Android this directory is on device, not local (see also --render-test-output-dir). Only implemented in TracePerfTest.
  • --save-screenshots: Save screenshots. Only implemented in TracePerfTest.
  • --screenshot-frame <frame>: Which frame to capture a screenshot of. Defaults to first frame (1). Only implemented in TracePerfTest.

For example, for an endless run with no warmup on swiftshader, run:

angle_trace_tests --gtest_filter=TraceTest.trex_200 --use-angle=swiftshader --steps 1000000 --no-warmup

Understanding the Metrics

  • cpu_time: Amount of CPU time consumed by an iteration of the test. This is backed by GetProcessTimes on Windows, getrusage on Linux/Android, and zx_object_get_info on Fuchsia.
    • This value may sometimes be larger than wall_time. That is because we are summing up the time on all CPU threads for the test.
  • wall_time: Wall time taken to run a single iteration, calculated by dividing the total wall clock time by the number of test iterations.
    • For trace tests, each rendered frame is an iteration.
  • gpu_time: Estimated GPU elapsed time per test iteration. We compute the estimate using GLES timestamp queries at the beginning and ending of each test loop.
    • For trace tests, this metric is only enabled in vsync mode.