web-tests
web-tests
contains:
Do not git clone
web-tests. Use the fetch
command included with depot_tools
.
mkdir src cd src fetch web-tests cd web-tests
Don't forget to run gclient sync
every time you pull new changes from origin.
web-tests uses poetry manage python dependencies.
# python3.11-dev is required for pandas sudo apt-get install python3.11 python3.11-dev python3-poetry
Check that you have poetry on your path and make sure you have the right $PATH
settings.
poetry --help || echo "Please update your \$PATH to include poetry bin location"; # Depending on your setup, add one of the following to your $PATH: echo "`python3 -m site --user-base`/bin"; python3 -c "import sysconfig; print(sysconfig.get_path('scripts'))";
Install the necessary dependencies from the lock file using poetry:
cd cuj/crossbench/runner poetry env use 3.11 poetry install
Some tests use WPR for repeatable playback of recorded web sessions. Before running these tests, the WPR archive files must first be downloaded.
cd cuj/crossbench/wpr ./wpr-setup.sh
Before running a test against an android target, make sure your device is connected through adb
:
adb devices > List of devices attached > 192.168.20.194:5555 device
Replace <DEVICE ID>
below with the actual device ID from adb devices
:
cd cuj/crossbench/runner poetry run python run.py --platform adb --device <DEVICE ID>
Before running a test against a ChromeOS target, make sure passwordless SSH is available to the device. Either an IP address or a SSH host is supported as the device id.
Replace <DEVICE>
below with the IP address or hostname of your device:
cd cuj/crossbench/runner poetry run python run.py --platform cros --device <DEVICE>
Running against a local browser on linux is minimally supported, but may require manual changes to test configuration.
cd cuj/crossbench/runner poetry run python run.py --platform local
The minimal invocation of the runner will attempt to run all benchmarks, CUJs, and corresponding variants in series.
To run a subset of tests, use the --tests
flag. --tests
supports Python regex format for matching the test names.
poetry run python run.py --platform adb --device <DEVICE ID> --tests speedometer.*
To specify only certain variants of a test, you can use the --variants
flag. --variants
also supports Python regex format for matching variants.
poetry run python run.py --platform adb --device <DEVICE ID> --tests local-conference --variants 16p
By default the runner will use ‘Chrome’ if the --browser
flag is not specified. The format accepted by the --browser
flag depends on the platform. For Android, use the package name of the installed browser. For ChromeOS or local, use the path to the browser executable.
Some CUJs require secrets to perform privileged actions (such as a test account username/password, or auth tokens for the Google Meet Bond API). Place your secrets in secrets.hjson
and pass the file to the runner:
poetry run python run.py --platform adb --device <DEVICE ID> --secrets /home/me/secrets.hjson --tests docs
Tests can be repeated for a number of iterations or for a specified amount of time using the --playback
flag. This flag is supported by crossbench and will iterate the post-setup sections of a CUJ and collect metrics for the entire invocation (instead of splitting metrics by iteration).
poetry run python run.py --platform adb --device <DEVICE ID> --tests tab-stress --variants blank-tab --playback 50x
poetry run python run.py --platform adb --device <DEVICE ID> --tests tab-stress --variants blank-tab --playback 2h
All test definitions and supporting files for crossbench based tests should be within the cuj/crossbench
directory.
Benchmarks are tests that are directly supported by and integrated into crossbench. Examples of benchmarks are speedometer, motionmark, and jetstream.
Every directory within cuj/crossbench/benchmarks
defines a crossbench benchmark that is supported by web-tests.
For example, cuj/crossbench/benchmarks/speedometer_3.0
contains the necessary configuration files for running the speedometer_3.0 benchmark as a web-test using crossbench.
Within a benchmark directory, the following files can be present:
browser-flags.hjson
probe-config.hjson
cb-args
(Optional)CUJs are tests that are implemented on top of crossbench's loading benchmark. These tests use page-config.hjson
files to define a list of actions to perform in the browser.
Every directory within cuj/crossbench/cujs
defines a CUJ that can be run using crossbench.
Within a CUJ directory, the following files determine how a CUJ is run:
page-config.hjson
<variant>.page-config.hjson
if several similar tests should be grouped together under a single CUJ directory.probe-config.hjson
or <variant>.probe-config.hjson
<variant>.page-config.hjson
, if <variant>.probe-config.hjson
exists it will be used, otherwise probe-config.hjson
will be used.browser-flags.hjson
or <variant>.browser-flags.hjson
<variant>.page-config.hjson
, if <variant>.browser-flags.hjson
exists it will be used, otherwise browser-flags.hjson
will be used.cb-args
(Optional)