commit | 3b6e99a54906832392959b8363c63a4442bb8be8 | [log] [tgz] |
---|---|---|
author | Camillo Bruni <cbruni@chromium.org> | Tue Mar 05 17:08:01 2024 |
committer | V8 LUCI CQ <v8-scoped@luci-project-accounts.iam.gserviceaccount.com> | Tue Mar 05 17:08:01 2024 |
tree | 12bf047f7cab3851752f56be0b5bffd222d3c020 | |
parent | 09bd5176beaf204662e8d8f547c4c594cbe413d5 [diff] |
Support full android package name for --browser Change-Id: Iee7f4006ba556ba691084fd38e7248620515803e Reviewed-on: https://chromium-review.googlesource.com/c/crossbench/+/5344755 Reviewed-by: Tushar Agarwal <agarwaltushar@google.com> Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Crossbench is a cross-browser/cross-benchmark runner to extract performance numbers.
Mailing list: crossbench@chromium.org
Issues/Bugs: Tests > CrossBench
Supported Browsers: Chrome/Chromium, Firefox, Safari and Edge.
Supported OS: macOS, Android, linux and windows.
Use the ./cb.py
script directly to run benchmarks (requires chrome's vpython3)
pip install crossbench
,Run the latest speedometer benchmark 20 times with the system default browser (chrome-stable):
# Run chrome-stable by default: ./cb.py speedometer --repeat=20 # Compare chrome browser versions and a local chrome build on jetstream: ./cb.py jetstream --browser=chrome-stable --browser=chrome-m90 --browser=$PATH
Profile individual line items (with pprof on linux):
./cb.py speedometer --probe='profiling' --separate
Use a custom chrome build and only run a subset of the stories:
./cb.py speedometer --browser=$PATH --probe='profiling' --story='jQuery.*'
Profile a website for 17 seconds on Chrome M100 (auto-downloading on macOS and linux):
./cb.py loading --browser=chrome-m100 --probe='profiling' --url=www.cnn.com,17s
Crossbench supports running benchmarks on one or multiple browser configurations. The main implementation uses selenium for maximum system independence.
You can specify a browser with --browser=<name>
. You can repeat the --browser
argument to run multiple browser. If you need custom flags for multiple browsers use --browser-config
(or pass simple flags after --
to the browser).
./cb.py speedometer --browser=$BROWSER -- --enable-field-trial-config
For more complex scenarios you can use a browser.config.hjson file. It allows you to specify multiple browser and multiple flag configurations in a single file and produce performance numbers with a single invocation.
./cb.py speedometer --browser-config=config.hjson
The example file lists and explains all configuration details.
Probes define a way to extract arbitrary (performance) numbers from a host or running browser. This can reach from running simple JS-snippets to extract page-specific numbers to system-wide profiling.
Multiple probes can be added with repeated --probe='XXX'
options. You can use the describe probes
subcommand to list all probes:
# List all probes: ./cb.py describe probes # List help for an individual probe: ./cb.py describe probe v8.log
Some probes can be configured, either with inline json when using --probe
or in a separate --probe-config
hjson file. Use the describe
command to list all options.
# Get probe config details: ./cb.py describe probe v8.log # Use inline hjson to configure a probe: ./cb.py speedometer --probe='v8.log:{prof:true}'
For complex probe setups you can use --probe-config=<file>
. The example file lists and explains all configuration details. For the specific probe configuration properties consult the describe
command.
Use the describe
command to list all benchmark details:
# List all benchmark info: ./cb.py describe benchmarks # List an individual benchmark info: ./cb.py describe benchmark speedometer_3.0 # List a benchmark's command line options: ./cb.py speedometer_3.0 --help
Stories define sequences of browser interactions. This can be simply loading a URL and waiting for a given period of time, or in more complex scenarios, actively interact with a page and navigate multiple times.
Use --help
or describe to list all stories for a benchmark:
./cb.py speedometer --help
Use --stories
to list individual story names, or use regular expression as filter.
./cb.py speedometer --browser=$BROWSER --stories='.*Angular.*'
This project uses poetry deps and package scripts to setup the correct environment for testing and debugging.
# a) On debian: sudo apt-get install python3.10 python3-poetry # b) With python 3.8 to 3.10 installed already: pip3 install poetry
Check that you have poetry on your path and make sure you have the right $PATH
settings.
poetry --help || echo "Please update your \$PATH to include poetry bin location"; # Depending on your setup, add one of the following to your $PATH: echo "`python3 -m site --user-base`/bin"; python3 -c "import sysconfig; print(sysconfig.get_path('scripts'))";
Install the necessary dependencies from the lock file using poetry:
# Select the python version you want to use (3.8 to 3.10): poetry env use 3.10 poetry install # For python 3.11 you have to skip pytype support: poetry env use 3.11 poetry install --without=dev-pytype
For local development / non-chromium installation you should use poetry run cb ...
instead of ./cb.py ...
.
Side-note, beware that poetry eats up an empty --
:
# With cb.py: ./cb.py speedometer ... -- --custom-chrome-flag ... # With poetry: poetry run cb speedometer ... -- -- --custom-chrome-flag ...
poetry run pytest
Run detailed test coverage:
poetry run pytest --cov=crossbench --cov-report=html
Run pytype type checker:
poetry run pytype -j auto .