Improve documentation

- Update Readme.md
- Improve `describe` command to print tabular data by default
- Add tabulate package

Change-Id: I83ff99ec46786ce4d1a95f336cff8f0799b86b50
Reviewed-on: https://chromium-review.googlesource.com/c/crossbench/+/3909888
Reviewed-by: Patrick Thier <pthier@chromium.org>
4 files changed
tree: c1bb0623b95cf18d5ddfc2ca1e55ec9693b59f4a
  1. crossbench/
  2. docs/
  3. tests/
  4. .gitignore
  5. .pylintrc
  6. .style.yapf
  7. browser.config.example.hjson
  8. LICENSE
  9. OWNERS
  10. poetry.lock
  11. pyproject.toml
  12. README.chromium
  13. README.md
  14. scripts.py
README.md

Crossbench

Crossbench is a cross-browser/cross-benchmark runner to extract performance numbers.

Setup:

This project uses poetry deps and package scripts to setup the correct environment for testing and debugging.

pip3 install poetry

Basic usage:

poetry run crossbench speedometer_2.0 \
    --browser=/path/to/chromium \
    --stories=VanillaJS.* \
    --probe=profiling \
    --probe=v8.log

Main Components

Browsers

Crossbench supports running benchmarks on one or multiple browser configurations. The main implementation uses selenium for maximum system independence.

You can specify a single browser with --browser=<name>

poetry run crossbench speedometer_2.0 \
    --browser=/path/to/chromium  \
    -- \
    --browser-flag-foo \
    --browser-flag-bar \

For more complex scenarios you can use a browser.config.hjson file. It allows you to specify multiple browser and multiple flag configurations in a single file and produce performance numbers with a single invocation.

poetry run crossbench speedometer_2.0 \
    --browser-config=config.hjson

The example file lists and explains all configuration details.

Probes

Probes define a way to extract arbitrary (performance) numbers from a host or running browser. This can reach from running simple JS-snippets to extract page-specific numbers to system-wide profiling.

Multiple probes can be added with repeated --probe=XXX options. You can use the describe subcommand to list all probes:

poetry run crossbench describe probes

Benchmarks

Use the describe command to list all benchmark details:

poetry run crossbench describe benchmarks

Stories

Stories define sequences of browser interactions. This can be simply loading a URL and waiting for a given period of time, or in more complex scenarios, actively interact with a page and navigate multiple times.

Use --help or describe to list all stories for a benchmark:

poetry run crossbench speedometer_2.0 --help

Use --stories to list individual comma-separated story names, or use a regular expression as filter.

poetry run crossbench speedometer_2.0 \
    --browser=/path/to/chromium \
    --stories=VanillaJS.*