commit | 69c26056d652f97c01c51cdeb7c49c4e64f1a6ce | [log] [tgz] |
---|---|---|
author | Tushar Agarwal <agarwaltushar@google.com> | Wed Mar 06 12:23:11 2024 |
committer | Tushar Agarwal <agarwaltushar@google.com> | Wed Mar 06 12:23:11 2024 |
tree | 52055b55aa044ce4ad2bdf7e6b7f1beec84e8fed | |
parent | ec3f9cc99f9c33e81856b3147e8cf6322ed690ff [diff] |
Add simpleperf profiling support to crossbench's profiling probe. This supports the most common current use cases for simpleperf, and uses defaults wherever they work well. This change is partly based upon an internal change by carlscab@. Bug: b/300261556 Change-Id: I272502a5c81b6a53b09a68247be77e7e998720a5 Reviewed-on: https://chromium-review.googlesource.com/c/crossbench/+/5344760 Reviewed-by: Camillo Bruni <cbruni@chromium.org>
Crossbench is a cross-browser/cross-benchmark runner to extract performance numbers.
Mailing list: crossbench@chromium.org
Issues/Bugs: Tests > CrossBench
Supported Browsers: Chrome/Chromium, Firefox, Safari and Edge.
Supported OS: macOS, Android, linux and windows.
Use the ./cb.py
script directly to run benchmarks (requires chrome's vpython3)
pip install crossbench
,Run the latest speedometer benchmark 20 times with the system default browser (chrome-stable):
# Run chrome-stable by default: ./cb.py speedometer --repeat=20 # Compare chrome browser versions and a local chrome build on jetstream: ./cb.py jetstream --browser=chrome-stable --browser=chrome-m90 --browser=$PATH
Profile individual line items (with pprof on linux):
./cb.py speedometer --probe='profiling' --separate
Use a custom chrome build and only run a subset of the stories:
./cb.py speedometer --browser=$PATH --probe='profiling' --story='jQuery.*'
Profile a website for 17 seconds on Chrome M100 (auto-downloading on macOS and linux):
./cb.py loading --browser=chrome-m100 --probe='profiling' --url=www.cnn.com,17s
Crossbench supports running benchmarks on one or multiple browser configurations. The main implementation uses selenium for maximum system independence.
You can specify a browser with --browser=<name>
. You can repeat the --browser
argument to run multiple browser. If you need custom flags for multiple browsers use --browser-config
(or pass simple flags after --
to the browser).
./cb.py speedometer --browser=$BROWSER -- --enable-field-trial-config
For more complex scenarios you can use a browser.config.hjson file. It allows you to specify multiple browser and multiple flag configurations in a single file and produce performance numbers with a single invocation.
./cb.py speedometer --browser-config=config.hjson
The example file lists and explains all configuration details.
Probes define a way to extract arbitrary (performance) numbers from a host or running browser. This can reach from running simple JS-snippets to extract page-specific numbers to system-wide profiling.
Multiple probes can be added with repeated --probe='XXX'
options. You can use the describe probes
subcommand to list all probes:
# List all probes: ./cb.py describe probes # List help for an individual probe: ./cb.py describe probe v8.log
Some probes can be configured, either with inline json when using --probe
or in a separate --probe-config
hjson file. Use the describe
command to list all options.
# Get probe config details: ./cb.py describe probe v8.log # Use inline hjson to configure a probe: ./cb.py speedometer --probe='v8.log:{prof:true}'
For complex probe setups you can use --probe-config=<file>
. The example file lists and explains all configuration details. For the specific probe configuration properties consult the describe
command.
Use the describe
command to list all benchmark details:
# List all benchmark info: ./cb.py describe benchmarks # List an individual benchmark info: ./cb.py describe benchmark speedometer_3.0 # List a benchmark's command line options: ./cb.py speedometer_3.0 --help
Stories define sequences of browser interactions. This can be simply loading a URL and waiting for a given period of time, or in more complex scenarios, actively interact with a page and navigate multiple times.
Use --help
or describe to list all stories for a benchmark:
./cb.py speedometer --help
Use --stories
to list individual story names, or use regular expression as filter.
./cb.py speedometer --browser=$BROWSER --stories='.*Angular.*'
This project uses poetry deps and package scripts to setup the correct environment for testing and debugging.
# a) On debian: sudo apt-get install python3.10 python3-poetry # b) With python 3.8 to 3.10 installed already: pip3 install poetry
Check that you have poetry on your path and make sure you have the right $PATH
settings.
poetry --help || echo "Please update your \$PATH to include poetry bin location"; # Depending on your setup, add one of the following to your $PATH: echo "`python3 -m site --user-base`/bin"; python3 -c "import sysconfig; print(sysconfig.get_path('scripts'))";
Install the necessary dependencies from the lock file using poetry:
# Select the python version you want to use (3.8 to 3.10): poetry env use 3.10 poetry install # For python 3.11 you have to skip pytype support: poetry env use 3.11 poetry install --without=dev-pytype
For local development / non-chromium installation you should use poetry run cb ...
instead of ./cb.py ...
.
Side-note, beware that poetry eats up an empty --
:
# With cb.py: ./cb.py speedometer ... -- --custom-chrome-flag ... # With poetry: poetry run cb speedometer ... -- -- --custom-chrome-flag ...
poetry run pytest
Run detailed test coverage:
poetry run pytest --cov=crossbench --cov-report=html
Run pytype type checker:
poetry run pytype -j auto .