Stricter startup checks

- Add Benchmark::setup checks
- PressBenchmarks now display a warning if the default URLs are not
  reachable
- Rename args.live to args.is_live (and all users to is_live as well)
- Fix PressBenchmarkStory.from_names and friends to use correct live
  value
- Switch HostEnvironmentConfig ignore value back to None to simplify
  command line integration
- Add platform.processes for future remote implementation
- Unskip working tests again

Change-Id: I4143ddbeee25ab80679b015bcfdd1511574f9243
Reviewed-on: https://chromium-review.googlesource.com/c/crossbench/+/3999131
Reviewed-by: Marja Hölttä <marja@chromium.org>
11 files changed
tree: 5c8d6f4d7ec443861ba58a54999ccfa7f97dca24
  1. crossbench/
  2. docs/
  3. tests/
  4. .gitattributes
  5. .gitignore
  6. .pylintrc
  7. .style.yapf
  8. browser.config.example.hjson
  9. LICENSE
  10. OWNERS
  11. poetry.lock
  12. probe.config.example.hjson
  13. pyproject.toml
  14. README.chromium
  15. README.md
  16. scripts.py
README.md

Crossbench

Crossbench is a cross-browser/cross-benchmark runner to extract performance numbers.

Setup:

This project uses poetry deps and package scripts to setup the correct environment for testing and debugging.

pip3 install poetry

Install the necessary dependencies from lock file via poetry

poetry install

Basic usage:

poetry run crossbench speedometer_2.0 \
    --browser=/path/to/chromium \
    --stories=VanillaJS.* \
    --probe=profiling \
    --probe=v8.log

Run Unit tests

poetry run pytest

Run detailed test coverage:

poetry run pytest --cov=crossbench --cov-report=html

Run pytype type checker:

poetry run pytype -j auto crossbench

Main Components

Browsers

Crossbench supports running benchmarks on one or multiple browser configurations. The main implementation uses selenium for maximum system independence.

You can specify a single browser with --browser=<name>

poetry run crossbench speedometer_2.0 \
    --browser=/path/to/chromium  \
    -- \
        --browser-flag-foo \
        --browser-flag-bar \

For more complex scenarios you can use a browser.config.hjson file. It allows you to specify multiple browser and multiple flag configurations in a single file and produce performance numbers with a single invocation.

poetry run crossbench speedometer_2.0 \
    --browser-config=config.hjson

The example file lists and explains all configuration details.

Probes

Probes define a way to extract arbitrary (performance) numbers from a host or running browser. This can reach from running simple JS-snippets to extract page-specific numbers to system-wide profiling.

Multiple probes can be added with repeated --probe=XXX options. You can use the describe subcommand to list all probes:

poetry run crossbench describe probes

Benchmarks

Use the describe command to list all benchmark details:

poetry run crossbench describe benchmarks

Stories

Stories define sequences of browser interactions. This can be simply loading a URL and waiting for a given period of time, or in more complex scenarios, actively interact with a page and navigate multiple times.

Use --help or describe to list all stories for a benchmark:

poetry run crossbench speedometer_2.0 --help

Use --stories to list individual comma-separated story names, or use a regular expression as filter.

poetry run crossbench speedometer_2.0 \
    --browser=/path/to/chromium \
    --stories=VanillaJS.*