| commit | f281e16a4ba34a014c89d8cb5887a0f2c8e87af3 | [log] [tgz] |
|---|---|---|
| author | Camillo Bruni <cbruni@chromium.org> | Wed Nov 23 18:19:15 2022 |
| committer | Camillo Bruni <cbruni@chromium.org> | Wed Nov 23 18:19:15 2022 |
| tree | c633d0459eb370d9d185a4e96a05ae531347e622 | |
| parent | 89f3433751dee15d7f4e92218d422229081743dd [diff] |
Various fixes - List stories and filtering options in benchnark help - Always log selected stories - Change label order to have the browser name first - Pre-check that ffmpeg and montage can run (a common issue under macOS when running unsigned binaries) - Rename a couple of variables to be make the code a bit more readable - Improve logging output (more progress, INFO level by default) - Handle KeyboardInterrupt later to print warning about potentially incomplete results Change-Id: I00ad614088f7ba585d1466c9c65d85a67c288ee2 Reviewed-on: https://chromium-review.googlesource.com/c/crossbench/+/4047202 Reviewed-by: Patrick Thier <pthier@chromium.org>
Crossbench is a cross-browser/cross-benchmark runner to extract performance numbers.
This project uses poetry deps and package scripts to setup the correct environment for testing and debugging.
pip3 install poetry
Install the necessary dependencies from lock file via poetry
poetry install
poetry run crossbench speedometer_2.0 \
--browser=/path/to/chromium \
--stories=VanillaJS.* \
--probe=profiling \
--probe=v8.log
poetry run pytest
Run detailed test coverage:
poetry run pytest --cov=crossbench --cov-report=html
Run pytype type checker:
poetry run pytype -j auto crossbench
Crossbench supports running benchmarks on one or multiple browser configurations. The main implementation uses selenium for maximum system independence.
You can specify a single browser with --browser=<name>
poetry run crossbench speedometer_2.0 \
--browser=/path/to/chromium \
-- \
--browser-flag-foo \
--browser-flag-bar \
For more complex scenarios you can use a browser.config.hjson file. It allows you to specify multiple browser and multiple flag configurations in a single file and produce performance numbers with a single invocation.
poetry run crossbench speedometer_2.0 \
--browser-config=config.hjson
The example file lists and explains all configuration details.
Probes define a way to extract arbitrary (performance) numbers from a host or running browser. This can reach from running simple JS-snippets to extract page-specific numbers to system-wide profiling.
Multiple probes can be added with repeated --probe=XXX options. You can use the describe subcommand to list all probes:
poetry run crossbench describe probes
Use the describe command to list all benchmark details:
poetry run crossbench describe benchmarks
Stories define sequences of browser interactions. This can be simply loading a URL and waiting for a given period of time, or in more complex scenarios, actively interact with a page and navigate multiple times.
Use --help or describe to list all stories for a benchmark:
poetry run crossbench speedometer_2.0 --help
Use --stories to list individual comma-separated story names, or use a regular expression as filter.
poetry run crossbench speedometer_2.0 \
--browser=/path/to/chromium \
--stories=VanillaJS.*