commit	dc4d80cc37942c7ee4dfdc2047800ea61016cc60	[log] [tgz]
author	Camillo Bruni <cbruni@chromium.org>	Mon May 13 15:00:13 2024
committer	V8 LUCI CQ <v8-scoped@luci-project-accounts.iam.gserviceaccount.com>	Mon May 13 15:00:13 2024
tree	6b94e2a6797a67d10e8ad8d2793d14e9319a66e2
parent	5589455a2be11b09f873f91b0b0c55547a37b0be [diff]

Reland "Support remote Posix paths from Windows"

This relands commit b7b58eb412e462a01d3764aafe59840a090e5759.

Use the host-platform when checking for an existing driver binary.

Original change's description:
> Support remote Posix paths from Windows
>
> When using an adb device under Windows we get broken paths for the
> remote sh commands that are generate since the paths are encoded
> for the runner platform (windows) and not the targeted remote
> platform (adb).
>
> Add explicit type aliases for paths:
> - LocalPath: Paths that are only avialble on the host running platform
> - RemotePath: Paths that can be on either a remote or a local machine
>
> Note that LocalPath (pathlib.Path) inherits from
> RemotePath (pathlib.PurePath)
>
> In order to get the existing pyfakefs tests working we need to access
> the path aliases via an indirect load, otherwise pyfakefs is not able
> to properly path the Path classes.
>
> Probes now need to be more explicit when using local vs. remote Paths
> which requires more indirections via the Platform path helper methods
> for many path operations.
>
> Bug: 335309993
> Change-Id: Ife95fb6af5d9e90172447f64ada3a29feb697def
> Reviewed-on: https://chromium-review.googlesource.com/c/crossbench/+/5463981
> Reviewed-by: Patrick Thier <pthier@chromium.org>
> Commit-Queue: Camillo Bruni <cbruni@chromium.org>

Bug: 335309993
Change-Id: I51ff9e8cf27689202d8acd5b0e5667b818a13a21
Reviewed-on: https://chromium-review.googlesource.com/c/crossbench/+/5529332
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Reviewed-by: Patrick Thier <pthier@chromium.org>

84 files changed

tree: 6b94e2a6797a67d10e8ad8d2793d14e9319a66e2

README.md

Crossbench

Crossbench is a cross-browser/cross-benchmark runner to extract performance numbers.

Mailing list: crossbench@chromium.org

Issues/Bugs: Tests > CrossBench

Supported Browsers: Chrome/Chromium, Firefox, Safari and Edge.

Supported OS: macOS, Android, linux and windows.

Basic usage:

Chromium Devs (with a full chromium checkout)

Use the ./cb.py script directly to run benchmarks (requires chrome's vpython3)

Standalone installation

Use pip install crossbench,
or use the “poetry” package manager, see the development section.

Run the latest speedometer benchmark 20 times with the system default browser (chrome-stable):

# Run chrome-stable by default:
./cb.py speedometer --repeat=20

# Compare chrome browser versions and a local chrome build on jetstream:
./cb.py jetstream --browser=chrome-stable --browser=chrome-m90 --browser=$PATH

Profile individual line items (with pprof on linux):

./cb.py speedometer --probe='profiling' --separate

Use a custom chrome build and only run a subset of the stories:

./cb.py speedometer --browser=$PATH --probe='profiling' --story='jQuery.*'

Profile a website for 17 seconds on Chrome M100 (auto-downloading on macOS and linux):

./cb.py loading --browser=chrome-m100 --probe='profiling' --url=www.cnn.com,17s

Collect perfetto data from loading separate websites on multiple attached android devices using the device ID or unique device names (see adb devices -l):

./cb.py loading --probe-config=./config/probe/perfetto.probe.config.example.hjson \
    --browser='Pixel_4:chrome-stable' --browser='AA00BB11:chrome-stable' \
    --parallel=platform \
    --url=https://theverge.com,15s,https://cnn.com,15s  --separate

Main Components

Browsers

Crossbench supports running benchmarks on one or multiple browser configurations. The main implementation uses selenium for maximum system independence.

You can specify a browser with --browser=<name>. You can repeat the --browser argument to run multiple browser. If you need custom flags for multiple browsers use --browser-config (or pass simple flags after -- to the browser).

./cb.py speedometer --browser=$BROWSER -- --enable-field-trial-config

Browser Config File

For more complex scenarios you can use a browser.config.hjson file. It allows you to specify multiple browser and multiple flag configurations in a single file and produce performance numbers with a single invocation.

./cb.py speedometer --browser-config=config.hjson

The example file lists and explains all configuration details.

Probes

Probes define a way to extract arbitrary (performance) numbers from a host or running browser. This can reach from running simple JS-snippets to extract page-specific numbers to system-wide profiling.

Multiple probes can be added with repeated --probe='XXX' options. You can use the describe probes subcommand to list all probes:

# List all probes:
./cb.py describe probes

# List help for an individual probe:
./cb.py describe probe v8.log

Inline Probe Config

Some probes can be configured, either with inline json when using --probe or in a separate --probe-config hjson file. Use the describe command to list all options.

# Get probe config details:
./cb.py describe probe v8.log

# Use inline hjson to configure a probe:
./cb.py speedometer --probe='v8.log:{prof:true}'

Probe Config File

For complex probe setups you can use --probe-config=<file>. The example file lists and explains all configuration details. For the specific probe configuration properties consult the describe command.

Benchmarks

Use the describe command to list all benchmark details:

# List all benchmark info:
./cb.py describe benchmarks

# List an individual benchmark info:
./cb.py describe benchmark speedometer_3.0

# List a benchmark's command line options:
./cb.py speedometer_3.0 --help

Stories

Stories define sequences of browser interactions. This can be simply loading a URL and waiting for a given period of time, or in more complex scenarios, actively interact with a page and navigate multiple times.

Use --help or describe to list all stories for a benchmark:

./cb.py speedometer --help

Use --stories to list individual story names, or use regular expression as filter.

./cb.py speedometer --browser=$BROWSER --stories='.*Angular.*'

Development

Setup

This project uses poetry deps and package scripts to setup the correct environment for testing and debugging.

# a) On debian:
sudo apt-get install python3.10 python3-poetry
# b) With python 3.8 to 3.10 installed already:
pip3 install poetry

Check that you have poetry on your path and make sure you have the right $PATH settings.

poetry --help || echo "Please update your \$PATH to include poetry bin location";
# Depending on your setup, add one of the following to your $PATH:
echo "`python3 -m site --user-base`/bin";
python3 -c "import sysconfig; print(sysconfig.get_path('scripts'))";

Install the necessary dependencies from the lock file using poetry:

# Select the python version you want to use (3.8 to 3.10):
poetry env use 3.10
poetry install

# For python 3.11 you have to skip pytype support:
poetry env use 3.11
poetry install --without=dev-pytype

Crossbench

For local development / non-chromium installation you should use poetry run cb ... instead of ./cb.py ....

Side-note, beware that poetry eats up an empty --:

# With cb.py:
./cb.py speedometer ... -- --custom-chrome-flag ...
# With poetry:
poetry run cb speedometer ... -- -- --custom-chrome-flag ...

Tests

poetry run pytest

Run detailed test coverage:

poetry run pytest --cov=crossbench --cov-report=html

Run pytype type checker:

poetry run pytype -j auto .