Tests can be executed within a ChromeOS chroot using the tast executable's run command:
tast run <target> <test-pattern> <test-pattern> ...
To run private tests (e.g. crosint test bundle), use -buildbundle=<bundle-name>.
Tests can also be run within a chrome-sdk using the cros_run_test command:
third_party/chromite/bin/cros_run_test --device=<target> --tast <test-pattern>
When running tast run from a ChromeOS chroot, it compiles the tast binary in /platform/tast and all tests in /platform/tast-tests and runs the code you have checked out with any local changes.
When running cros_run_test from a chrome-sdk, the precompiled tests which are packaged in the downloaded bundle will run which will be from the platform version shown in chrome-sdk.
The first positional argument supplied to the run subcommand specifies the “target”, i.e. the device where the test will be run, also known as the device-under-test or DUT. In the case of local tests, the test code will run directly on the DUT. For remote tests, the test code will run on the host machine but connect to the DUT. Expressions like root@10.0.0.1:22, root@localhost, and 10.0.0.2 are supported. The root user is used by default, as tests frequently require root access to the DUT.
By default, the standard testing_rsa key from chromite/ssh_keys will be used to establish an SSH connection with the device. The keyfile flag can be supplied to specify a different private key:
tast run -keyfile=$HOME/.ssh/id_rsa ...
Any additional positional arguments describe which tests should be executed:
(("dep:chrome" || "dep:android") && !informational) matches all tests with a dep:chrome or dep:android attribute but not an informational attribute. Attributes that don't consist of a letter or underscore followed by letters, digits, and underscores must be double-quoted. ‘*’ characters in quoted strings are interpreted as wildcards. See go.chromium.org/tast/core/internal/expr for details about expression syntax.ui.* matches all tests with names prefixed by ui.. Multiple patterns can be supplied: passing example.Pass and example.Fail selects those two tests."name:ui.*" expression instead.See the Test Attributes document for more information about attributes.
Tests may be skipped if they list software dependencies that aren‘t provided by the DUT. This behavior can be controlled via the tast command’s -checktestdeps flag.
When you are running Tast tests that require Chrome, you should double check that no unexpected arguments are specified in /etc/chrome_dev.conf on the DUT. It can change Chrome's behavior and some tests might fail unexpectedly.
When the -build flag is true (the default), tast run rebuilds the cros test bundle and pushes it to the DUT as /usr/local/share/tast/bundles_pushed/cros. This permits faster compilation and deployment when writing new tests than the normal emerge/cros deploy cycle can provide.
The name of the bundle to build, push, and run can be specified via the -buildbundle flag. If the bundle‘s source code is outside of the tast-tests repository, you will need to specify the repository’s path using the -buildtestdir flag.
To rebuild a test bundle, the tast command needs its dependencies' source code to be available. This code is automatically checked out to /usr/lib/gopath when building packages for the host system, as described in the [Go in Chromium OS] document. The tast command will automatically inform you when the bundle's dependencies need to be manually emerged.
To skip rebuilding a bundle and instead run all builtin bundles within the /usr/local/share/tast/bundles directory on the DUT (for local tests) and /usr/share/tast/bundles on the host system (for remote tests), pass -build=false. The default builtin cros local bundle should be present on all test system images (non-test system images are not supposed by Tast).
Some tests use servo, a physical device that connects to both the host machine and the DUT. These tests all specify servo as a runtime variable, so they must be run with that variable specifying the servo host and servo port.
If you can run Tast without port forwarding, please use following syntax.
tast run -var=servo=<servo-host>:<servo-port> <target> <test-pattern>
If you need run Tast with port forwarding, please use following syntax.
tast run -var=servo=localhost:<servo-port>:ssh:<servo_localhost_port> localhost:<DUT_localhost_port> <test-pattern>
In order for a test to interact with the servo, the servo host must be running an instance of servod (servo daemon) on the appropriate port. When Tast is run through the Tauto wrapper via test_that, Tauto takes care of initiating and closing servod. However, when Tast is run through tast run, it does not initiate servod; the user must initiate servod from the servo host:
ssh <servo-host> servod --board=<board> --model=<model> --port=<servo-port> --serialname=<servo-serial>
In automated testing in the ChromeOS lab, Tast tests can reach a working Servo device via servo runtime variable if they are scheduled with Autotest control files declaring a Servo dependency. Control files for mainline tests declare it, but other control files may not. See crrev.com/c/2790771 for an example to add a dependency declaration.
As each test runs, its output is streamed to the tast executable. Overall information about the current state of the test run is logged to stdout by default. The top-level (i.e. tast -verbose run ...) -verbose flag can be supplied to log additional information to the console, including all messages written by tests.
By default, test results are written to a subdirectory under /tmp/tast/results, but an alternate directory can be supplied via the run command's -resultsdir flag. If the default directory is used, a symlink will also be created to it at /tmp/tast/results/latest.
Various files and directories are created within the results directory:
crashes/ - Breakpad minidump files with information about crashes that occured during testing.full.txt - All output from the run, including messages logged by individual tests.results.json - Machine-parseable test results, supplied as a JSON-marshaled array of run.TestResult structs.run_error.txt - Error message describing the reason why the run was aborted (e.g. SSH connection to DUT was lost). Only written when a global error occurs.streamed_results.jsonl - Streamed machine-parseable test results, supplied as a JSONL array of run.TestResult structs. Provides partial results if the tast process is interrupted before results.json is written.system_logs/ - Diff of /var/log on the DUT before and after testing.unified/ - Unified log collected from system logs.unified.log - Human-readable system log messages.unified.export.gz - gzip-compressed logs with full metadata from croslog's export mode which is similler to journalctl -o export.tests/<test-name>/ - Per-test subdirectories, containing test logs and other output files.log.txt - Log of messages and errors reported by the test.results-chart.json - Machine-parseable performance metrics produced by the perf package.... - Other output files from the test.timing.json - Machine-parsable JSON-marshaled timing information about the test run produced by the timing package.Tast resets the device owner of the DUT before test run, and after the test run, the device owner remains to be testuser. To reset that, run the following on the DUT:
stop ui rm -rf /var/lib/devicesettings '/home/chronos/Local State' start ui
In a window outside the chroot do,
In another window inside chroot:
tast run <target> <test>
In a window outside the chroot do,
Any port is fine as long as it is not used by other applications. Leave the SSH Watcher session(s) on.
In another window inside chroot:
tast run localhost:<port> <test>
Specific example:
DUT_HOSTNAME=chromeos6-row18-rack18-host17
SERVO_HOSTNAME=chromeos6-row18-rack18-labstation4
SERVO_PORT=9972
LOCAL_SERVO_SSH_PORT=2003
LOCAL_DUT_SSH_PORT=2005
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentityFile=~/.ssh/testing_rsa -l root ${SERVO_HOSTNAME?} "start servod PORT=${SERVO_PORT?}"
go run ~/chromiumos/src/platform/dev/contrib/sshwatcher/sshwatcher.go ${SERVO_HOSTNAME?} ${LOCAL_SERVO_SSH_PORT?} ${DUT_HOSTNAME?} ${LOCAL_DUT_SSH_PORT?}
<new window>
cd ~/chromiumos ; cros_sdk --no-ns-pid
SERVO_PORT=9972
LOCAL_SERVO_SSH_PORT=2003
LOCAL_DUT_SSH_PORT=2005
tast run --var=servo=localhost:${SERVO_PORT?}:ssh:${LOCAL_SERVO_SSH_PORT?} localhost:${LOCAL_DUT_SSH_PORT?} firmware.Fixture.normal
See Tast Debugger
When a tast test fails on chromium CQ for an LKGM update, it means that some change in chromium has broken the test.
Bisecting the failure is a little bit complicated since you have to repeatedly compile chromium and deploy to a VM or DUT and then run a tast test from a cros chroot.
This process can be made easier using git bisect run with a script.
You should be able to see the first bot that fails and find a good and bad chromium commit via the BLAMELIST tab of the builder page.
This script assumes there is a local betty-pi-arc VM running and that tests are run from a ChromeOS chroot.
#!/bin/sh set -x gclient sync autoninja -C out_betty-pi-arc/Release chrome third_party/chromite/bin/deploy_chrome --board=betty-pi-arc --build-dir=out_betty-pi-arc/Release --device=ssh://localhost:9222 --verbose --strip-flags=-S --mount cd ~/chromiumos cros_sdk -- tast run -failfortests localhost:9222 terminal.Crosh
git bisect good <last-commit-in-previous-good-build> git bisect bad <last-commit-in-first-bad-build> git bisect run ./bisect.sh
There are a number of ChromeOS boards designed to run as VMs such as amd64-generic, and betty-pi-arc. Update chromium/.gclient file and add cros_boards field in custom_vars:
solutions = [ { ... "custom_vars" : { ... "cros_boards": "betty-pi-arc", # colon-separated list. }, }, ]
Download the SDK for the newly added board:
gclient sync
Enter chrome-sdk and download VM:
cros chrome-sdk --board=betty-pi-arc --download-vm --log-level=info --nogn-gen
Start the VM:
cros vm --start
You can view the screen output using VNC:
vncviewer localhost:5900 &
You may need to install a VNC viewer package:
sudo apt install xtightvncviewer
If you are going to compile and deploy chrome, you may want to update args.gn with values to match what the bots use. E.g.:
$ cat out_betty-pi-arc/Release/args.gn import("//build/args/chromeos/betty-pi-arc.gni") # Place any additional args or overrides below: dcheck_always_on = true exclude_unwind_tables = false is_chrome_branded = true use_remoteexec = true
Compile and deploy:
autoninja -C out_betty-pi-arc/Release chrome third_party/chromite/bin/deploy_chrome --board=betty-pi-arc --build-dir=out_betty-pi-arc/Release --device=ssh://localhost:9222 --verbose --strip-flags=-S --mount
Run a test:
third_party/chromite/bin/cros_run_test --device=localhost:9222 --tast terminal.Crosh
If you need to use a different VM version than the current //chromeos/CHROMEOS_LKGM used by chrome-sdk, you can download VMs directly from google cloud storage. Open https://ci.chromium.org/ui/p/chromeos/builders/postsubmit/betty-pi-arc-snapshot , select a green build, and find gs:// url for image. You can edit the URL and replace betty-pi-arc-snapshot with any other board name to find the builds for other boards.
cros flash --debug file:///tmp xbuddy://remote/betty-pi-arc-snapshot/R125-15831.0.0-96552-8752561462204607073 cros vm --start --board=betty-pi-arc --image-path /tmp/chromiumos_test_image.bin