Add `--presubmit` for gestures library presubmit

Parsing the output of the `touchtests` command in the presubmit and
determining what to do from that would be a pain, and increase the size
of the interface between the presubmit and the touch tests. Adding a
presubmit "mode" to the touch tests allows us to format things nicely
and decide whether to pass or fail in Python.

To improve code reuse, this also involved refactoring the existing `Run`
method to separate the actual test running, deciding what to output, and
formatting that output.

BUG=none
TEST=run against a reference file, manually modified to simulate
     regressions, improvements, and failing new tests

Change-Id: I6988020e44d246a7e99095916bd037ff0b088e86
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/touchpad-tests/+/2171940
Tested-by: Harry Cutts <hcutts@chromium.org>
Commit-Queue: Harry Cutts <hcutts@chromium.org>
Reviewed-by: Sean O'Brien <seobrien@chromium.org>
1 file changed
tree: 07a915a66835f25562c84cb550915a1ae9a7052f
  1. framework/
  2. tests/
  3. .gitignore
  4. Makefile
  5. OWNERS
  6. README.md
  7. touchtests
  8. touchtests.template
README.md

Touch tests

Introduction

This repository contains automated tests for Chromium OS's Gestures library. Each test has a log of evdev events which are replayed, a properties file containing gesture properties to set while the Gestures library runs, and a Python function which verifies the output and returns a test score.

Setting up

Assuming that you've followed the developer guide, simply run the following inside your chroot:

(inside)
$ cd ~/trunk/src/platform/touchpad-tests
$ sudo make setup-in-place

Running tests

To run all tests, simply run touchtests. To run one or more specific tests, you can pass a test name or a glob:

(inside)
$ touchtests atlas-1.0/fat-thumb-fail
$ touchtests atlas-1.0/palm-while-typing*

Each test will return a status, with the following meanings:

  • success: the test succeeded, with the given score out of 1.
  • failure: the test failed.
  • error: an error occurred while running the test, so the behavior of the gestures library couldn't be evaluated.
  • incomplete: (baseline tests only) the evdev log for this platform hasn't been created.

Checking for regressions

The --out (or -o) switch creates a report file that future runs can be compared against with the --ref (or -r) switch:

(inside)
$ touchtests --out baseline.json
# (cause some regressions)
$ touchtests --ref baseline.json

The output table will contain a delta column that indicates any regressions or improvements, and an error message will be shown if regressions exist.