This document assumes that you've already completed A Tour of Go, Codelab #1 and Codelab #2.
This codelab follows the creation of a remote firmware test in Tast. In doing so, we'll learn how to do the following:
firmware.Reporterfirmware.Configfirmware.Helperfirmware.ModeSwitcherfirmware.fixtureIn order to demonstrate what's happening “under the hood,” some sections of this codelab will overwrite code from earlier sections. Thus, working through the codelab will teach you more than just studying the final code.
Most firmware tests are remote tests, because they tend to disrupt the DUT, such as by rebooting it or corrupting its firmware.
Create a new remote test in the firmware bundle by creating a new file, ~/trunk/src/platform/tast-tests/src/chromiumos/tast/remote/bundles/cros/firmware/codelab.go, with the following contents:
// Copyright 2021 The Chromium OS Authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. package firmware import ( "context" "chromiumos/tast/testing" ) func init() { testing.AddTest(&testing.Test{ Func: Codelab, Desc: "Demonstrates common functionality for remote firmware tests", Contacts: []string{ "me@chromium.org", // Test author "my-team@chromium.org", // Backup mailing list }, // TODO: Move to firmware_unstable, then firmware_ec Attr: []string{"group:firmware", "firmware_experimental"}, }) } func Codelab(ctx context.Context, s *testing.State) { s.Log("FAFT stands for Fully Automated Firmware Test") }
Try running the test with the following command (inside the chroot). You‘ll need to replace ${HOST} with your DUT’s IP.
> tast run ${HOST} firmware.Codelab
You can find a copy of this code at codelab_basic.txt.
This directory contains many sample test files. Those test files have the
.txtextension instead of the normal.goin order to avoid running in automated suites, and to avoid preupload errors for using AddTest in a support package. Those tests won't be able to run unless you move them intotast-tests/src/chromiumos/tast/remote/bundles/cros/firmware, and rename them with.goextensions.
Notice the Attr line in the above snippet. In previous Tast codelabs, we used the attributes "group:mainline" and "informational". Those attributes cause tests to run in the CQ. However, most firmware tests are very expensive to run, and might not be appropriate to run in the CQ. You can read more about effective CQ usage on-corp at go/effective-cq. Additionally, most firmware tests should be run on the faft-test device pool, unlike the mainline tests.
For those reasons, firmware tests have a separate group of attributes. The group is called "group:firmware", and has a handful of sub-attributes. You can find all of those sub-attributes in attr.go, and you can learn more about how we use them to run FAFT tests at go/faft-tast-via-tauto.
The firmware_experimental attribute is for tests that are particularly unstable. This mitigates the risk of accidentally putting a DUT into into a state that would cause other tests to fail. If we find that our test is stable enough, then we can promote it to another attribute, like firmware_unstable and eventually firmware_ec (or smoke, cr50, slow, ccd as appropriate). But for now, let's use firmware_experimental.
Many FAFT tests rely on certain hardware or software features. Per go/tast-deps, the correct way to handle that in Tast is via HardwareDeps and SoftwareDeps.
Let's write a test that needs a Chrome EC. For context, some platforms have a Chrome EC (such as octopus), some platforms have a Wilco EC (such as sarien), and some platforms have no EC (such as rikku).
There is already a HardwareDep for ChromeEC, so let's use it.
We'll need to import "chromiumos/tast/testing/hwdep", so add that to the imports:
import (
"context"
"chromiumos/tast/testing"
"chromiumos/tast/testing/hwdep"
)
Then, we'll need to add a HardwareDep to the test definition:
testing.AddTest(&testing.Test{
...
HardwareDeps: hwdep.D(hwdep.ChromeEC()),
})
Now, if you run your test on a DUT without a Chrome EC (such as rikku or sarien), it should skip without running.
For more information about HardwareDeps and SoftwareDeps, see go/tast-deps. If your test requires a dependency that isn't supported by Tast yet, make it! Others will thank you.
At this point (after running gofmt), your test file should resemble codelab_dependency.txt.
The remote firmware library has a utility structure called Reporter, which has several methods for collecting useful firmware information from the DUT.
Let's use the Reporter to collect some basic information about the DUT, such as its board and model.
First, add the remote firmware/reporters library to your imports.
import (
"context"
"chromiumos/tast/remote/firmware/reporters"
"chromiumos/tast/testing"
"chromiumos/tast/testing/hwdep"
)
Then, in the main body of your test, use reporters.New to initialize a Reporter object. (You can also remove that s.Log line about FAFT.)
func Codelab(ctx context.Context, s *testing.State) {
r := reporters.New(s.DUT())
Finally, use some reporter methods to collect information about the DUT.
board, err := r.Board(ctx)
if err != nil {
s.Fatal("Failed to report board: ", err)
}
model, err := r.Model(ctx)
if err != nil {
s.Fatal("Failed to report model: ", err)
}
s.Logf("Reported board=%s, model=%s", board, model)
}
Try running this test on your DUT. Did you get the results you expected?
At this point (after running gofmt), your test file should resemble codelab_reporter.txt.
fw-testing-configs are a set of JSON files defining platform-specific attributes for use in FAFT testing. You can read all about it at go/cros-fw-testing-configs-guide. Config data for all platforms gets consolidated into a single data file called CONSOLIDATED.json.
In Tast, we access that consolidated JSON as a data file. The relative path to that data file is exported in the remote firmware library as firmware.ConfigFile.
This section is for background, you should actually use the Helper class to read the configs.
To use that data file in our test, we first have to import the remote firmware library, and declare that our test uses the data file:
import (
"context"
"chromiumos/tast/remote/firmware"
"chromiumos/tast/remote/firmware/reporters"
"chromiumos/tast/testing"
"chromiumos/tast/testing/hwdep"
)
func init() {
testing.AddTest(&testing.Test{
...
Data: []string{firmware.ConfigFile},
})
}
The firmware.NewConfig constructor requires three parameters: the full path to the data file, and the DUT's board and model. The full path to the data file can be acquired via s.DataPath. Thanks to the previous section, we already have the board and model.
func Codelab(ctx context.Context, s *testing.State) {
...
cfg, err := firmware.NewConfig(s.DataPath(firmware.ConfigFile), board, model)
if err != nil {
s.Fatal("Failed to create config: ", err)
}
Finally, we can access the config data via the Config struct‘s fields. If the field you want to reference isn’t yet included in the Config struct, go ahead and add it.
s.Log("This DUT's mode-switcher type is: ", cfg.ModeSwitcherType)
}
At this point (after running gofmt), your test file should resemble codelab_config.txt.
Many firmware tests rely on Servo for controlling the DUT. Let's use Servo in our test.
Package “servo” can be used in both remote and local tests depending upon the requirements.
In order to send commands via Servo, the test needs to know the address of the machine running servod (the “servo_host”), and the port on which that machine is running servod (the “servo_port”). These values are supplied at runtime as a runtime variable, of the form ${SERVO_HOST}:${SERVO_PORT}.
This section is for background, you should actually use the Helper class.
To start, we will need to declare "servo" as a variable in the test:
func init() {
testing.AddTest(&testing.Test{
...
Vars: []string{"servo"},
})
}
Next, in the test body, we will need to create a servo.Proxy object, which forwards commands to servod. The NewProxy constructor requires the servo host:port, and a keyFile and keyDir that can be obtained via the test's DUT object. Additionally, we should close the Proxy at the end of the test (via defer).
First, import the common servo library:
import (
...
"chromiumos/tast/common/servo"
)
Append the following to the test body:
func Codelab(ctx context.Context, s *testing.State) { ... // Set up Servo in remote tests dut := s.DUT() servoSpec, _ := s.Var("servo") pxy, err := servo.NewProxy(ctx, servoSpec, dut.KeyFile(), dut.KeyDir()) if err != nil { s.Fatal("Failed to connect to servo: ", err) } defer pxy.Close(ctx) }
Let‘s use Servo to find out the DUT’s ec_board. ec_board is a simple GPIO control returning a string. We can get the value of a string control via the Servo method GetString, defined in methods.go. That method takes a parameter of the type StringControl. methods.go defines a bunch of different StringControls, including one called ECBoard (with value ec_board).
We'll have to extract the Servo object from our Proxy, and then call its GetString method with servo.ECBoard as the control parameter. Add the following to the test body:
func Codelab(ctx context.Context, s *testing.State) { ... // Get the DUT's ec_board via Servo ecBoard, err := pxy.Servo().GetString(ctx, servo.ECBoard) if err != nil { s.Fatal("Getting ec_board control from servo: ", err) } s.Log("EC Board: ", ecBoard) }
methods.go defines a lot of Servo commands, but not nearly all of them. If you want to use a command that isn't represented in methods.go, go ahead and add it!
Note that methods.go takes advantage of Go's type system to define which values can be sent to certain controls. For example, Servo supports several controls representing keypresses, such as ctrl_enter and power_key, which each accept a duration-type string value: "press", "long_press", or "tab". In methods.go, these controls are given the type KeypressControl, and their acceptable values are given the type KeypressDuration. This allows tests to call the Servo method KeypressWithDuration, such as pxy.Servo().KeypressWithDuration(ctx, servo.PowerKey, servo.LongPress). This reduces the chance of inadvertently sending an invalid string, and makes it easy for future developers to understand what acceptable values are for each control.
Try running your test using the same syntax as in previous sections:
(inside) > tast run ${HOST} firmware.Codelab
What happened? Your test failed, because you didn't supply the command-line variable servo, which our code referred to as a RequiredVar. So, treating ${SERVO_HOST} and ${SERVO_PORT} as your servo host and servo port respectively, try the following command:
(inside) > tast run -var=servo=${SERVO_HOST}:${SERVO_PORT} $HOST firmware.Codelab
What happened? If the servo host machine was running servod on the servo port, then your test probably ran successfully. Otherwise, you probably saw the following error message:
Error at codelab.txt:54: Failed to create servo: Post "http://127.0.0.1:42529": read tcp 127.0.0.1:60326->127.0.0.1:42529: read: connection reset by peer
You‘ll need to SSH into the servo host machine (if it’s different from your workstation) and run servod (such as via start servod PORT=${SERVO_PORT}). Then try your tast run command again. Did it work? It should have.
At this point (after running gofmt), your test file should resemble codelab_servo.txt.
Servo functionality and usage remains exactly the same as both in the remote and local tests however they just differ in the way the connection is being established to communicate with servo devices. This subsection discusses the details and possible problems that you might encounter while using servo in local tests.
In local tests, we don‘t have to establish a proxy (it’s doable but not recommended) to communicate with servod instance, a simple direct connection will be enough. The NewDirect constructor takes the servo host:port and returns a servo.Servo object. Additionally, we should close the Servo connection at the end of the test (via defer). Make sure, the host:port address is reachable from DUT.
Append the following to the test body:
func CodelabLocal(ctx context.Context, s *testing.State) { ... // Set up Servo in local tests servoSpec, _ := s.Var("servo") srvo, err := servo.NewDirect(ctx, servoSpec) if err != nil { s.Fatal("Failed to connect to servo: ", err) } defer srvo.Close(ctx) }
Now, let‘s use servo to find out DUT’s ec_board exactly the same as we did in remote but this time as a local test. We will use the same GetString method. Add the following to the test body:
func CodelabLocal(ctx context.Context, s *testing.State) { ... // srvo is the Servo object that we created earlier through NewDirect constructor. ecBoard, err := srvo.GetString(ctx, servo.ECBoard) if err != nil { s.Fatal("Getting ec_board control from servo: ", err) } s.Log("EC Board: ", ecBoard) }
Try running your test using the same syntax as in previous sections:
(inside) > tast run -var=servo=${SERVO_HOST}:${SERVO_PORT} $HOST firmware.CodelabLocal
Note: While using servo in local tests, be sure your servod instance is not attached to the loopback interface (if DUT and servo host are different devices). You can bind it to a network interface by running servod --host ${SERVO_HOST_Interface_IP} --port ${SERVO_PORT} or make it listen to all available interfaces including loopback by using 0.0.0.0 for the flag --host. Now rerun your tast command again. Did it work?
If you are still getting error similar to this:
Failed to establish proxy with servo: timeout = 10s: Post "http://192.168.2.139:9999": dial tcp 192.168.2.139:9999: i/o timeout (Client.Timeout exceeded while awaiting headers)
Please make sure, the ${SERVO_PORT} that you are using to bind servod is open on the servo host device firewall. You can check that by running on servo host:
(inside servo host) > iptables -L
# if there is no entry for $SERVO_PORT, allow the port by running:
(inside servo host) > iptables -A INPUT -p tcp --dport ${SERVO_PORT} -j ACCEPT
At this point (after running gofmt), your test file should resemble codelab_servo_local.txt.
For reference on running tests with Servo, you can review the relevant section of go/tast-running.
Many firmware tests need to perform complicated subroutines on the DUT. Rather than calling many individual SSH commands, it is faster and stabler to send a single command via RPC. Fortunately, Tast has built-in gRPC support.
Let‘s use the BIOS service to get the DUT’s current GBB flags.
We‘ll need to add three imports: Tast’s rpc library, the Tast firmware service library, and a library called empty (which we use for sending RPC requests containing no data). Add these to the file's imports:
import (
...
"github.com/golang/protobuf/ptypes/empty"
...
fwService "chromiumos/tast/services/cros/firmware"
"chromiumos/tast/rpc"
)
Note that we have imported the firmware service library under the alias fwService. This is to avoid a namespace collision with the remote firmware library ("chromiumos/tast/remote/firmware")—otherwise, both would be called firmware.
Next, declare the BIOS service as a ServiceDep in the test's initialization:
func init() {
testing.AddTest(&testing.Test{
...
ServiceDeps: []string{"tast.cros.firmware.BiosService"},
}
In the test body, initialize an RPC connection:
func Codelab(ctx context.Context, s *testing.State) { ... // Connect to RPC cl, err := rpc.Dial(ctx, dut, s.RPCHint()) if err != nil { s.Fatal("Failed to connect to RPC service on the DUT: ", err) } defer cl.Close(ctx)
Create a BIOS service client, which we will use to call BIOS-related RPCs:
bios := fwService.NewBiosServiceClient(cl.Conn)
Finally, call the GetGBBFlags RPC, and report results:
// Get current GBB flags via RPC flags, err := bios.GetGBBFlags(ctx, &empty.Empty{}) if err != nil { s.Fatal("Failed to get GBB flags: ", err) } s.Log("Clear GBB flags: ", flags.Clear) s.Log("Set GBB flags: ", flags.Set) }
At this point (after running gofmt), your test file should resemble codelab_rpc.txt.
In the above sections, we wrote 25 lines of code just to initialize a Servo, Config, and RPC client—not to mention additional code to actually use those structures. If we had to include all that boilerplate in every firmware test, it would violate the DRY principle.
For that reason, we have a structure called firmware.Helper, whose job is to manage other remote firmware structures. Let's simplify our test using a Helper.
At the start of your test body, initialize a firmware.Helper. The NewHelper constructor requires several parameters, which it will use later to initialize other structures: dut (to construct the Reporter and Servo), rpcHint (for the RPC connection), cfgFilepath (for the Config), and servoHostPort (for Servo).
This is simpler, but keep reading. firmware.fixture is simpler yet!
func Codelab(ctx context.Context, s *testing.State) {
servoSpec, _ := s.Var("servo")
h := firmware.NewHelper(s.DUT(), s.RPCHint(), s.DataPath(firmware.ConfigFile), servoSpec, "", "", "", "")
defer h.Close(ctx)
...
}
The Helper now has all the information it needs to create a Reporter, Servo, Config, and RPC connection. Additionally, h.Close will close any firmware structures that it initialized.
The Helper constructs a Reporter during NewHelper, using the DUT that we passed in, so we can use that right away. Replace the reporters.New constructor in your test:
// OLD r := reporters.New(s.DUT()) // NEW r := h.Reporter
If you prefer, you can use h.Reporter directly, without binding to a new variable:
board, err := h.Reporter.Board(ctx)
But for today, we'll leave it as r.
The other constructors are a little more complicated. Helper is lazy about initializing most structures, so that it can avoid unnecessary operations. For example, if a test doesn‘t require Config, then there is no need to spend time fetching the DUT’s board and model. But we do want to use a Config, so let's create one.
The convention for such constructing a Foo via Helper is h.RequireFoo(). If the Helper is already managing a Foo, then it won't create a new one; thus, rather than specifying that we need a new Foo, we require that one exist.
So, let's replace the Config constructor in our test with h.RequireConfig.
// OLD cfg, err := firmware.NewConfig(s.DataPath(firmware.ConfigFile), board, model) if err != nil { s.Fatal("Failed to create config: ", err) } s.Log("This DUT's mode-switcher type is: ", cfg.ModeSwitcherType) // NEW if err := h.RequireConfig(ctx); err != nil { s.Fatal("Failed to create config: ", err) } s.Log("This DUT's mode-switcher type is: ", h.Config.ModeSwitcherType)
Note that we didn‘t need to pass the board and model to RequireConfig; it fetched them via its Reporter. And, note that RequireConfig didn’t return a Config object; it was stored as h.Config.
RequireConfigwill fail if yourtesting.Testblock doesn't containData: []string{firmware.ConfigFile},
Next, let's use our Helper to create a Servo.
// OLD // Set up Servo dut := s.DUT() servoSpec, _ := s.Var("servo") pxy, err := servo.NewProxy(ctx, servoSpec, dut.KeyFile(), dut.KeyDir()) if err != nil { s.Fatal("Failed to connect to servo: ", err) } defer pxy.Close(ctx) // Get the DUT's ec_board via Servo ecBoard, err := pxy.Servo().GetString(ctx, servo.ECBoard) if err != nil { s.Fatal("Getting ec_board control from servo: ", err) } s.Log("EC Board: ", ecBoard) // NEW // Get the DUT's ec_board via Servo if err := h.RequireServo(ctx); err != nil { s.Fatal("Failed to connect to servo: ", err) } ecBoard, err := h.Servo.GetString(ctx, servo.ECBoard) if err != nil { s.Fatal("Getting ec_board control from servo: ", err) } s.Log("EC Board: ", ecBoard)
As described above, note that we don't need to defer h.Servo.Close. That will be called by h.Close, which we have already deferred.
Finally, let's use our Helper to initialize the RPC connection and BIOS service client.
// OLD // Connect to RPC cl, err := rpc.Dial(ctx, dut, s.RPCHint()) if err != nil { s.Fatal("Failed to connect to RPC service on the DUT: ", err) } defer cl.Close(ctx) // Get current GBB flags via RPC bios := fwService.NewBiosServiceClient(cl.Conn) flags, err := bios.GetGBBFlags(ctx, &empty.Empty{}) if err != nil { s.Fatal("Failed to get GBB flags: ", err) } // NEW // Get current GBB flags via RPC if err := h.RequireBiosServiceClient(ctx); err != nil { s.Fatal("Failed to connect to RPC service on the DUT: ", err) } flags, err := h.BiosServiceClient.GetGBBFlags(ctx, &empty.Empty{}) if err != nil { s.Fatal("Failed to get GBB flags: ", err) }
Notice that we didn‘t have to dial an RPC connection before creating the BIOS service client. That’s because h.RequireBiosServiceClient calls another Require method in its implementation, h.RequireRPCClient. As we will see in the next section, some Helper constructors make heavy use of nested requirements like this.
If you try to run your code, the compiler will throw unused-import errors, due to our removed code. You can go ahead and delete any unused imports (fwService, reporters, and rpc).
At this point (after running gofmt), your test file should resemble codelab_helper.txt.
Let's reboot the DUT into recovery mode.
There is a structure called a ModeSwitcher, which can boot the DUT into normal mode, recovery mode, and developer mode. It can also perform a mode-aware reset, which resets the DUT while retaining the boot-mode. The NewModeSwitcher constructor requires a Helper, because switching boot-modes requires a Config, a Servo, and an RPC connection.
Append the following to the test body to create a ModeSwitcher:
func Codelab(ctx context.Context, s *testing.State) { ... // Switch to recovery mode ms, err := firmware.NewModeSwitcher(ctx, h) if err != nil { s.Fatal("Failed to create mode-switcher: ", err) }
Then use the ModeSwitcher to switch to recovery mode. The constants for different boot-modes are defined in Tast‘s common/firmware library, which allows us to access them from both local and remote tests. Let’s add that to the imports:
import (
...
fwCommon "chromiumos/tast/common/firmware"
)
Finally, in the main test body, use the ModeSwitcher to reboot to recovery mode.
if err := ms.RebootToMode(ctx, fwCommon.BootModeRecovery); err != nil {
s.Fatal("Failed to boot to recovery mode: ", err)
}
}
We don‘t need to verify the DUT’s boot mode after rebooting; ms.RebootToMode does that, and returns an error if the DUT ends up in an unexpected boot-mode.
If you try running this test, it will fail due to an undeclared service dependency. RebootToMode uses an RPC service, tast.cros.firmware.UtilsService, which we didn't declare in ServiceDeps. So, update the ServiceDeps line in the test initialization:
func init() {
testing.AddTest(&testing.Test{
...
ServiceDeps: []string{"tast.cros.firmware.BiosService", "tast.cros.firmware.UtilsService"},
...
})
}
As it stands, this test does something rude: it leaves the DUT in recovery mode. When the next test starts, the DUT will still be in recovery mode, which could cause unexpected behavior. This is bad! You could clean up manually by rebooting back to normal mode. But in the next section, we'll explore a more defensive alternative.
At this point (after running gofmt), your test file should resemble codelab_boot_mode.txt.
Tast has a wonderful feature called Fixtures. This allows us to perform certain actions before and after each test. If several tests have the same Fixture, they will all be run in a row.
This is really useful for firmware testing. In FAFT, we like to ensure that the GBB flags start and end in an expected state. We also have many tests that have to run in normal mode, many others that run in recovery mode, and others that run in developer mode. Clumping those tests together means that we can boot into recovery mode once, and then run all of the recovery mode tests. It also makes cleanup easier, because if a DUT ends the test in an unexpected state (such as a strange boot mode or strange GBB flags), the Fixture will return it to the expected state.
Let's add a Fixture to our test to ensure that it starts and ends in normal mode. First, import the remote/firmware/fixture library:
import (
...
"chromiumos/tast/remote/firmware/fixture"
)
In the test initialization, declare a Fixture of fixture.NormalMode:
func init() {
testing.AddTest(&testing.Test{
...
Fixture: fixture.NormalMode,
})
}
Now, before the test runs, the test harness will invoke the fixtures‘s SetUp and PreTest methods, which will put it into normal-mode. After all tests using this same fixture have finished, the test harness will invoke its TearDown method, which restores the DUT’s GBB flags and boot-mode from before the tests began.
The Fixture has a built-in Helper, so we don‘t need to create our own. Let’s replace the NewHelper line so that we can reuse the Fixture's Helper.
// OLD servoSpec, _ := s.Var("servo") h := firmware.NewHelper(s.DUT(), s.RPCHint(), s.DataPath(firmware.ConfigFile), servoSpec, "", "", "", "") defer h.Close(ctx) // NEW h := s.FixtValue().(*fixture.Value).Helper
Note that we don't need to close the Helper, because the Fixture will use it again at the end of all tests, and will close it afterward.
Go ahead and run your code—this is the last time we'll modify it.
At this point (after running gofmt), your test file should resemble codelab_fixt.txt.
When you write firmware-related CLs in Tast, please follow the process prescribed at go/tast-reviews. Your CL should be reviewed by a test-owner and by a Tast-owner: that is, someone with subject-matter expertise, and someone with harness expertise.
There is a gwsq alias for Tast firmware-library reviews: tast-fw-library-reviewers@google.com. If you set that alias as a reviewer in Gerrit, it will be re-assigned to somebody with domain expertise. If you‘re not sure who should review your code, that’s a great place to start. If you'd like to join that group of reviewers (which is a great way to learn more about FAFT), please email cros-fw-engrod@google.com.
This concludes the FAFT-in-Tast codelab. Congratulations! We look forward to reviewing your CLs.