Description

This script is to run any version of Khronos WebGL conformance test on various OSes (like Android, ChromeOS, Linux, MacOS and Windows) with various browsers (like Chrome, Edge, FireFox, Safari, etc.). Results will be compared with expectations and a final report will be generated.
WebDriver backs the main logic of test automation in this script.

Supported Configurations

Target OS means the OS you run test on, while host OS is the place you run this script. They are same most of time, while Android is the only known exception now.

[1] means the configuration has been tested.

Setup

Android, Linux, MacOS and Windows

• Install Python
  Both Python 2 and 3 are supported, and you may download it from https://www.python.org/downloads/.
• Install Python Selenium package
  pip install selenium
  Note on Windows, pip resides in <python_dir>/Scripts.
• Download Android platform tools (Android only)
  You may download tools as below according to your system, and please be sure to put them in your PATH environment.
  https://dl.google.com/android/repository/platform-tools-latest-darwin.zip
  https://dl.google.com/android/repository/platform-tools-latest-linux.zip
  https://dl.google.com/android/repository/platform-tools-latest-windows.zip
  
• Download this script
  Put it in <work_dir>
  
• Download webdriver
  You may put them under <work_dir>/webdriver/<os_name>, which is the default place for them. Otherwise, you have to designate the path of webdriver with option --webdriver-path.
  <os_name>: android, linux, mac or win.
  Webdriver download links:
  Chrome (chromedriver(.exe)): https://sites.google.com/a/chromium.org/chromedriver
  Edge (MicrosoftWebDriver.exe): https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver
  FireFox (geckodriver(.exe)): https://github.com/mozilla/geckodriver/releases
  Safari (safaridriver): already included as /usr/bin/safaridriver
• Execute script
  python conformance.py
  Type --help for more information
• Check report
  After test, you may find report in <work_dir>/result/<timestamp>.html
  <timestamp>: The datetime you run the test in format %Y%m%d%H%M%S, e.g., 20170403235901

ChromeOS

First, a test image is required as the script relies on telemetry. Then you just need to copy the script to your ChromeOS and execute it as others, including Python, webdriver binary, etc., just work out of the box.

Supported Features

• Multiple Android devices
  You may connect multiple Android devices with your host machine, and use --android-device-id to designate the exact device you will test on.
• Multiple GPUs
  Multiple GPUs can be installed on same device. Typically, you may have one discrete GPU and one integrated GPU in this scenario. The choice among them can be quite flexible. For example, on MacOS, you may run one application with discrete GPU, while running another application with integrated GPU at the same time. The script will try to check some info from browser at runtime to see which GPU it uses actually. The info of GPU in usage can be very important for the tests. For example, it's important to know how many of the expectations can be applied in current tests.
• Crash handling
  It's often to see some GPU driver issues crash the browser. To run the whole test suite in a batch, the capability to recover from crash is critical. However, the crash handling can be very complex, due to different browsers under very different situations.
  Currently, some simple but effective crash handling was added, which was verified to be very useful for tests at least with Chrome.
• Resume from last tests
  We can't always guarantee the tests to be finished smoothly, especially when many crashes are unexpected. The script will record the progress (<work_dir>/log/resume) in details so that you may resume from it next time.
• Automatic retry
  Sometimes, a test case can be flaky under an abnormal context, and a clean retest can mute this false alarm. A simple retry mechanism is brought for this sake.
• Expectation as the baseline
  Sophiscated expectations regarding to OS, GPU and browser can be set so that you can always have a clear idea on improvements and regressions.
• Test with only a subset of all cases
  You may designate a folder or a specific case for testing using option --suite.
• Extra browser options
  You may pass extra browser options to test script. An intuitive usage of this is to live behind the proxy.
• Top time consuming cases
  Top time consuming cases will also be listed in final report, which can help to find some performance issue.
• OpenGL ES
  Sometimes, you want to test against OpenGL ES instead of OpenGL on Linux, and option --gles is your friend here.
• Self-build Mesa driver
  On Linux, Mesa driver can be used on the fly, which means you may run the system with system graphics stack, while running browser solely with your self-build Mesa driver. Option --mesa-dir can be used for this sake.

TODO Features

• Python 3 support
  
• More support of host_os, target_os and browser combinations, especially for Safari
• The design of expectations
• Get more GPU, OS, browser info
• log_path of geckodriver
• Run with multiple frames (?frame=x in url)
  This might not be an important feature.