Tests are identified by names like ui.ChromeLogin or platform.ConnectToDBus. The portion before the period, called the category, is the final component of the test's package name, while the portion after the period is the name of the exported Go function that implements the test.
Test function names should follow Go's naming conventions, and acronyms should be fully capitalized. Test names should not end with Test, both because it's redundant and because the _test.go filename suffix is reserved in Go for unit tests.
Test names are automatically derived from tests' package and function names and should not be explicitly specified when defining tests.
Public tests built into the default cros local and remote test bundles are checked into the tast-tests repository under the src/chromiumos/tast/local/bundles/cros/ and src/chromiumos/tast/remote/bundles/cros/ directories (which may also be accessed by the local_tests and remote_tests symlinks at the top of the repository). Private tests are checked into private repositories such as the tast-tests-private repository, and built into non-cros test bundles.
Tests are categorized into packages based on the functionality that they exercise; for example, the ui package contains local tests that exercise the Chrome OS UI.
Support packages used by multiple test categories located in src/chromiumos/tast/local/ and src/chromiumos/tast/remote/, alongside the bundles/ directories. For example, the chrome package can be used by local tests to interact with Chrome.
A local test named ui.DoSomething should be defined in a file named src/chromiumos/tast/local/bundles/cros/ui/do_something.go (i.e. convert the test name to lowercase and insert underscores between words) with contents similar to the following:
// Copyright 2018 The Chromium OS Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. package ui import ( "context" "chromiumos/tast/testing" ) func init() { testing.AddTest(&testing.Test{ Func: DoSomething, Desc: "Does X to verify Y", Contacts: []string{"me@chromium.org"}, Attr: []string{"informational"}, SoftwareDeps: []string{"chrome_login"}, }) } func DoSomething(ctx context.Context, s *testing.State) { // The actual test goes here. }
Tests have to specify email addresses of persons and groups who are familiar with those tests in Contacts. At least one personal email address of an active committer should be specified so that we can file bugs or ask for code reviews.
Tests may specify attributes and software dependencies when they are declared. Setting the informational attribute on new tests is recommended, as tests without this attribute will block the Commit Queue on failure otherwise.
If there‘s a support package that’s specific to a single category, it‘s often best to place it underneath the category’s directory. See the “Scoping and shared code” section.
When adding a new test category, you must update the test bundle's main.go file (either local/bundles/cros/main.go or remote/bundles/cros/main.go) to underscore-import the new package so its init functions will be executed to register tests.
Test code should be formatted by gofmt and checked by go vet. It should follow Go's established best practices as described by these documents:
The Go FAQ may also be helpful. Additional resources are linked from the Go Documentation page.
Packages and exported identifiers (e.g. types, functions, constants, variables) should be documented by Godoc-style comments. Godoc comments are optional for test functions, since the Test.Desc field already contains a brief description of the test.
Support packages should be exercised by unit tests when possible. Unit tests can cover edge cases that may not be typically seen when using the package, and they greatly aid in future refactorings (since it can be hard to determine the full set of Tast-based tests that must be run to exercise the package). See How to Write Go Code: Testing and Go's testing package for more information about writing unit tests for Go code. The Best practices for writing Chrome OS unit tests document contains additional suggestions that may be helpful (despite being C++-centric).
Setting FEATURES=test when emerging a test bundle package (tast-local-tests-cros or tast-remote-tests-cros) will run all unit tests for the corresponding packages in the tast-tests repository (i.e. chromiumos/tast/local/... or chromiumos/tast/remote/..., respectively).
During development, the fast_build.sh script can be used to quickly build and run tests for a single package or all packages.
Entries in import declaration must be grouped by empty line, and sorted in following order.
In each group, entries must be sorted in the lexicographical order. For example:
import (
"context"
"fmt"
"github.com/godbus/dbus"
"golang.org/x/sys/unix"
"chromiumos/tast/errors"
"chromiumos/tast/local/chrome"
)
Note that, although github.com and golang.org are different domains, they should be in a group.
This is how goimports --local=chromiumos/ sorts. It may be valuable to run the command. Note that, 1) the command preserves existing group. So, it may be necessary to remove empty lines in import() in advance, and 2) use the command to add/remove import entries based on the following code. The path resolution may require setting GOPATH properly.
As seen in the test declaration above, each test is comprised of a single exported function that receives a testing.State struct. This is defined in the Tast testing package (not to be confused with [Go's testing package] for unit testing) and is used to log progress and report failures.
If a test requires the system to be in a particular state before it runs, it should include code that tries to get the system into that state if it isn‘t there already. Previous tests may have aborted mid-run; it’s not safe to make assumptions that they undid all temporary changes that they made.
Tests should also avoid performing unnecessary de-initialization steps on completion: UI tests should leave Chrome logged in at completion instead of restarting it, for example. Since later tests can‘t safely make assumptions about the initial state of the system, they’ll need to e.g. restart Chrome again regardless, which takes even more time. In addition to resulting in a faster overall running time for the suite, leaving the system in a logged-in state makes it easier for developers to manually inspect it after running the test when diagnosing a failure.
Note that tests should still undo atypical configuration that leaves the system in a non-fully-functional state, though. For example, if a test needs to temporarily stop a service, it should restart it before exiting.
Use defer statements to perform cleanup when your test exits. defer is explained in more detail in the Defer, Panic, and Recover blog post.
Put more succintly:
Assume you‘re getting a reasonable environment when your test starts, but don’t make assumptions about Chrome‘s initial state. Similarly, try to leave the system in a reasonable state when you go, but don’t worry about what Chrome is doing.
Tast uses context.Context to implement timeouts. A test function takes as its first argument a context.Context with an associated deadline that expires when the test‘s timeout is reached. The context’s Done function returns a channel that can be used within a select statement to wait for expiration, after which the context's Err function returns a non-nil error.
The testing.Poll function makes it easier to honor timeouts while polling for a condition:
err := testing.Poll(ctx, func (ctx context.Context) error {
var url string
if err := conn.Eval(ctx, "location.href", &url); err != nil {
return errors.Wrap(err, "failed to evaluate location.href")
}
if url != targetURL {
return errors.Errorf("current URL is %s", url)
}
return nil
}, &testing.PollOptions{Timeout: 10 * time.Second})
if err != nil {
return errors.Wrap(err, "failed to navigate")
}
Sleeping without polling for a condition is discouraged, since it makes tests flakier (when the sleep duration isn't long enough) or slower (when the duration is too long). If you really need to do so, use testing.Sleep to honor the context timeout.
Any function that performs a blocking operation should take a context.Context as its first argument and return an error if the context expires before the operation finishes.
Several blog posts discuss these patterns in more detail:
Concurrency is rare in integration tests, but it enables doing things like watching for a D-Bus signal that a process emits soon after being restarted. It can also sometimes be used to make tests faster, e.g. by restarting multiple independent Upstart jobs simultaneously.
The preferred way to synchronize concurrent work in Go programs is by passing data between goroutines using a channel. This large topic is introduced in the Share Memory by Communicating blog post, and the Go Concurrency Patterns talk is also a good summary. The Go Memory Model provides guarantees about the effects of memory reads and writes across goroutines.
Global variables in Go are scoped at the package level rather than the file level:
The scope of an identifier denoting a constant, type, variable, or function ... declared at top level (outside any function) is the package block.
As such, all tests within a package like platform or ui share the same namespace. Please be careful of conflicts. Also, please avoid referencing identifiers declared in other files; otherwise repo upload will fail with lint errors.
If you need to share functionality between tests in the same package, please introduce a new descriptively-named subpackage; see e.g. the chromecrash package within the ui package, used by the ui.ChromeCrashLoggedIn and ui.ChromeCrashNotLoggedIn tests. Subpackages are described in more detail later in this document.
Much praise has been written for verifying just one thing per test. A quick sampling of internal links:
While this is sound advice for fast-running, deterministic unit tests, it isn't necessarily always the best approach for integration tests:
If you need to verify multiple related aspects of a single feature that requires a time-consuming setup process like logging in to Chrome, starting Android, or launching a container, it's often preferable to write a single test that just does the setup once and then verifies all aspects of the feature. As described in the next section, multiple errors can be reported by a single test, so coverage need not be reduced when tests are consolidated and an early expectation fails.
For lightweight testing that doesn‘t need to interact with Chrome or restart services, it’s fine to use fine-grained tests — there's almost no per-test overhead in Tast; the overhead comes from repeating the same slow operations within multiple tests.
A Tast test either passes (by reporting zero errors) or fails (by reporting one or more errors, timing out, or panicking). If a test requires functionality that isn't provided by the DUT, the test is skipped entirely.
Avoid writing tests that probe the DUT's capabilities at runtime, e.g.
// WRONG: Avoid testing for software or hardware features at runtime. func CheckCamera(ctx context.Context, s *testing.State) { if !supports720PCamera() { s.Log("Skipping test; device unsupported") return } // ... }
This approach results in the test incorrectly passing even though it actually didn‘t verify anything. (Tast doesn’t let tests report an “N/A” state at runtime since it would be slower than skipping the test altogether and since it will prevent making intelligent scheduling decisions in the future about where tests should be executed.)
Instead, specify software dependencies when declaring tests:
// OK: Specify dependencies when declaring the test. func init() { testing.AddTest(&testing.Test{ Func: CheckCamera, SoftwareDeps: []string{"camera_720p", "chrome_login"}, // ... }) }
The above document describes how to define new dependencies.
If a test depends on the DUT being in a specific configurable state (e.g. tablet mode), it should put it into that state. For example, chrome.ExtraArgs can be passed to chrome.New to pass additional command-line flags (e.g. --force-tablet-mode=tablet) when starting Chrome.
The tast-users mailing list is a good place to ask questions about test dependencies.
Sometimes a lengthy setup process (e.g. restarting Chrome and logging in, which takes at least 6-7 seconds) is needed by multiple tests. Rather than calling chrome.New at the beginning of each test, tests can declare that they require a logged-in Chrome instance by setting testing.Test.Pre to chrome.LoggedIn. This enables Tast to just perform login once and then share the same Chrome instance with all tests that specify the precondition. See the chrome.LoggedIn documentation for more details.
If you want a new Chrome precondition with custom options, call chrome.NewPrecondition from a single place in your test package and save the precondition as a global variable so that all of your tests can use it when registering themselves. It's best to initialize and store the precondition in a subpackage so it can be shared by multiple test files. For example, see video tests' pre subpackage.
The testing.State struct provides functions that tests may use to report their status:
Log and Logf record informational messages about the test's progress.Error and Errorf record errors and mark the test as failed but allow it to continue, similar to Google Test's EXPECT_ set of macros. Multiple errors may be reported by a single test.Fatal and Fatalf record errors and stop the test immediately, similar to the ASSERT_ set of macros.Note that higher-level functions for stating expectations and assertions are not provided; this was a conscious decision. See “Where is my favorite helper function for testing?” from the Go FAQ. That answer refers to Go's testing package rather than Tast's, but the same reasoning and suggestions are applicable to Tast tests.
When you‘re about to do something that could take a while or even hang, log a message using Log or Logf first. This both lets developers know what’s happening when they run your test interactively and helps when looking at logs to investigate timeout failures.
On the other hand, avoid logging unnecessary information that would clutter the logs. If you want to log a verbose piece of information to help determine the cause of an error, only do it after the error has occurred.
See the fmt package's documentation for available “verbs”.
Log, Error, and Fatal should be used in conjunction with a single string literal or when passing a string literal followed by a single value:
s.Log("Doing something slow")
s.Log("Loading ", url)
s.Error("Encountered an error: ", err)
s.Fatal("Everything is broken: ", err)
Logf, Errorf, and Fatalf should only be used in conjunction with printf-style format strings:
s.Logf("Read %q from %v", data, path)
s.Errorf("Failed to load %v: %v", url, err)
s.Fatalf("Got invalid JSON object %+v", obj)
When concatenating a string and a value using default formatting, use s.Log("Some value: ", val) rather than the more-verbose s.Logf("Some value: %v", val).
The same considerations apply to testing.ContextLog vs. testing.ContextLogf.
To construct new errors or wrap other errors, use the chromiumos/tast/errors package rather than standard libraries (errors.New, fmt.Errorf) or any other third-party libraries. It records stack traces and chained errors, and leaves nicely formatted logs when tests fail.
To construct a new error, use errors.New or errors.Errorf.
errors.New("process not found")
errors.Errorf("process %d not found", pid)
To construct an error by adding context to an existing error, use errors.Wrap or errors.Wrapf.
errors.Wrap(err, "failed to connect to Chrome browser process")
errors.Wrapf(err, "failed to connect to Chrome renderer process %d", pid)
Sometimes you may want to define custom error types, for example, to inspect and react to errors. In that case, embed *errors.E to your custom error struct.
type CustomError struct {
*errors.E
}
if err := doSomething(); err != nil {
return &CustomError{E: errors.Wrap(err, "something failed")}
}
Please follow Go's error string conventions when producing error values.
Error strings should not be capitalized (unless beginning with proper nouns or acronyms) or end with punctuation, since they are usually printed following other context.
For example:
if err := doSomething(id); err != nil {
return errors.Wrapf(err, "doing something to %q failed", id)
}
Log and error messages printed by tests via testing.State's Log, Logf, Error, Errorf, Fatal, or Fatalf methods, or via testing.ContextLog or testing.ContextLogf, should be capitalized phrases without any trailing punctuation that clearly describe what is about to be done or what happened:
s.Log("Asking Chrome to log in")
...
if err != nil {
s.Fatal("Failed to log in: ", err)
}
s.Logf("Logged in as user %v with ID %v", user, id)
In all cases, please avoid multiline strings since they make logs difficult to read. To preserve multiline output from an external program, please write it to an output file instead of logging it.
When including a path, URL, or other easily-printable value in a log message or an error, omit leading colons or surrounding quotes:
s.Logf("Trying to log in up to %d time(s)", numLogins)
errors.Errorf("%v not found", path)
Use quotes when including arbitrary data that may contain hard-to-print characters like spaces:
s.Logf("Successfully read %q from %v", data, path)
Use a colon followed by a space when appending a separate clause that contains additional detail (typically an error):
s.Error("Failed to log in: ", err)
Semicolons are appropriate for joining independent clauses:
s.Log("Attempt failed; trying again")
Support packages should not record test failures directly. Instead, return error values (using the errors package) and allow tests to decide how to handle them. Support packages' exported functions should typically take context.Context arguments and use them to return an error early when the test's deadline is reached and to log informative messages using testing.ContextLog and testing.ContextLogf.
Similarly, support packages should avoid calling panic when errors are encountered. When a test is running, panic has the same effect as State's Fatal and Fatalf methods: the test is aborted immediately. Returning an error gives tests the ability to choose how to respond.
The Error handling and Go and Errors are values blog posts offer guidance on using the error type.
The above guidelines do not necessarily apply to test subpackages that are located in subdirectories below test files. If a subpackage actually contains the test implementation (typically because it‘s shared across several tests), it’s okay to pass testing.State to it so it can report test errors itself.
Subpackages are typically aware of how they will be used, so an argument can be made for letting them abort testing using Fatal or even panic in cases where it improves code readability (e.g. for truly exceptional cases like I/O failures). Use your best judgement.
Note that it‘s still best to practice information hiding and pass only as much data is needed. Avoid passing testing.State when it’s not actually necessary:
context.Context so it can call testing.ContextLog.Tests can write output files that are automatically copied to the host system that was used to initiate testing:
func WriteOutput(s *testing.State) {
if err := ioutil.WriteFile(filepath.Join(s.OutDir(), "my_output.txt"),
[]byte("Here's my output!"), 0644); err != nil {
s.Error(err)
}
}
As described in the Running tests document, a test's output files are copied to a tests/<test-name>/ subdirectory within the results directory.
The perf package is provided to record the results of performance tests. See the perf documentation for more details.
Tests can register ancillary data files that will be copied to the DUT and made available while the test is running; consider a JavaScript file that Chrome loads or a short binary audio file that is played in a loop, for example.
Small non-binary data files should be directly checked into a data/ subdirectory under the test package as internal data files. Prefix their names by the test file's name (e.g. data/user_login_some_data.txt for a test file named user_login.go) to make ownership obvious.
Per the Chromium guidelines for third-party code, place (appropriately-licensed) data that wasn't created by Chromium developers within a third_party subdirectory under the data directory.
Larger data files like audio, video, or graphics files should be stored in Google Cloud Storage and registered as external data files to avoid permanently bloating the test repository. External data files are not installed to test images but are downloaded at run time by local_test_runner on DUT.
To add external data files, put external link files named <original-name>.external in data/ subdirectory whose content is JSON in the external link format.
For example, a data file belonging to a test named ui.UserLogin in the default cros bundle might be declared in user_login_some_image.jpg.external with the following content:
{
"url": "gs://chromiumos-test-assets-public/tast/cros/ui/user_login_some_image_20181210.jpg",
"size": 12345,
"sha256sum": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
}
Old versions of external data files should be retained indefinitely in Google Cloud Storage so as to not break tests on older system images. Include the date as a suffix in the filename to make it easy to add a new version when needed, e.g.
user_login_data_20180812.bin.
If data files are produced as build artifacts of Chrome OS, they can be also used as external data files. However, build artifacts are available only for Chrome OS images built by official builders; for developer builds, tests requiring build artifacts will fail.
An example external link file to reference a build artifact is below:
{
"type": "artifact",
"name": "license_credits.html"
}
As internal data files are much easier to view and modify than external data files, it's usually better to check in textual data. Only store binaries as external data.
If your test depends on outside executables, use Portage to build and package those executables separately and include them in test Chrome OS system images. Tast intentionally does not support compiling or deploying other packages that tests depend on.
If a data file is needed by a support package that‘s used by tests in multiple packages, it should be stored in a data subdirectory within the support package and symlinked into each test package’s data subdirectory. See the media_session_test.html file used by the mediasession package and shared by the arc.PlayPauseChrome and ui.MediaSessionGain tests, for example.
To register data files (regardless of whether they‘re checked into the test repository or stored externally), in your test’s testing.AddTest call, set the testing.Test struct's Data field to contain a slice of data file names (omitting the data/ subdirectory, and the .external suffix for external data files):
testing.AddTest(&testing.Test{
...
Data: []string{"user_login_data.bin"},
...
})
Later, within the test function, pass the same filename to testing.State's DataPath function to receive the path to the data file on the DUT:
b, err := ioutil.ReadFile(s.DataPath("user_login_data.bin"))
See the example.DataFiles test for a complete example of using both local and external data files.