The collection of python scripts inside of //tools/fuchsia/comparative_tester exist to facilitate the automated building, deployment, and execution of tests on Linux and Fuchsia devices on the same LAN as the hosting PC which will run the scripts, as well as generating comparisons and valuable statistical data and displaying it in an easily viewed form.
The automated test building and launching should currently work for any test target within the base chromium/src directory. Work is also underway to support executing Catapult tests automatically, with the same stats collection capabilities.
The general usage pattern for these scripts is as follows:
target_spec.py. Make sure that the output directories are where you want them, and that you are running the chromium test targets that you're interested in collecting results from. Also check to make sure that the specifications for the Linux and Fuchsia devices are appropriate for your specific network and OS configuration.comparative_tester.py, with any flags necessary to collect the data of interest.generate_perf_report.py. This should require no extra configuration on your part.display_perf_results.html in any web browser and giving it the result JSONs in the output directory you specified.(Non-Invokable)
This file is used internally to parse test output and return objects that can be manipulated easily by the rest of the code.
(Non-Invokable)
This file contains constant definitions used by other files in this folder to represent what tests to run, where the output will live, where the test binaries live, and so on. To add more tests for automatic building and deploying, they should be added here.
Invocation: comparative_tester.py --do-config --do-build --is-debug --num-repetitions=1
This is where tests are actually executed. It has four flags of note:
--do-config: makes the test script to generate an args.gn file for the output directory, and over-writes any existing args.gn file. This option is off by default, and no files will be generated or changed.--is-debug: requires the do_config flag to be set as well. Makes the args.gn files that the script will generate have the is_debug = true line. This option is off by default, and will cause the line is_debug = false to appear in the args.gn file--do-build: makes the test script build the targets specified inside of target_spec.py. By default, this is off, and the targets will not be rebuilt for the test cases.--num-repetitions: tells the script how many times to run each test in the battery of tests specified in target_spec.py. By default, this is one, so tests will only be executed one time each.--gtest_filter: Works just like it does on the tests. Positive filters go before the ‘-’ and negative ones after. All individual filters are separated from each other via a ‘:’. Support globbing.--gtest_filter=TestFoo.Case1:TestBar.*-TestBaz.*:TestQuux.Case2 filters out all cases in TestBaz, Case2 of TestQuux, and only runs TestFoo.Case1, and all cases in TestBar.More complex configuration options are present in target_spec.py.
Invocation: generate_perf_report.py This script takes no command line flags, but works off of many of the same fields inside of target_spec.py that comparative_tester.py does, namely fields instructing it where the raw data lives, and where to place the generated statistics when it's finished generating them. It generates the mean, standard deviation, and coefficient of variance for each target, test, and individual line scraped from the test output, and writes them to appropriately named JSONs.
The HTML file is just a thin shim around display_perf_results.js that can be opened in any web browser to view the data. Due to the isolation between javascript interpreters and the host's filesystem, the web page must be manually given the final JSON files that you want to display, at which point it will draw a table full of the data, and in a more human readable format. It accepts multiple files at a time.