FAFT (Fully Automated Firmware Test): Design Doc

Last updated: 2011/11/08

FAFT (Fully Automated Firmware Test)

FAFT Related Documents

FAFT Setup Instructions (Google specific instructions)
FAFT Test Coverage (Google internal only): http://goto/faft-coverage
FAFT Comparing Existing Manual Test Cases (Google internal only): http://goto/faft-comparison

Introduction

Security is one of the selling points of ChromeOS netbooks. Verified boot provides a solution to ChromeOS security model. In the verified boot design, firmware is the foundation of this secured castle. In addition, part of the firmware is marked as read-only, that means we are unable to patch security holes via autoupdate. So firmware testing is very important; however, our current firmware is lack of automated tests. It only relies on the manual tests by developers, our test team, and dogfooders. It seems to be a risk.

This document proposes a solution to the test problem. The advantages of FAFT:

Fully automatic, no human involved;
Real test of physical hardware, like USB plug-in, Ctrl-D key press;
High test coverage of complicated verified boot flows;
Easy to integrate to our existing test infrastructure, i.e. deploy to test lab, run remotely, continue test, etc.

Backgrounds

This document uses U-Boot as an example to discuss the FAFT test approach. It can be also applied to coreboot and any proprietary BIOS in a similar way.

The U-Boot software stack looks like the following graph:

faft-u-boot-stack

Most of the verified boot logic and crypto algorithms are inside the vboot_reference library. The main control logic in U-Boot is simple, just like reading some headers from SPI and filling them in a data structure which is then passed to vboot_reference.

The underlying U-Boot drivers are abstracted into VBoot Export APIs and some (not too many) APIs which the main control logic is used. VBoot export APIs are the most, e.g., TPM, display, keyboard, disk, system related, etc. The main control logic used APIs are not many, e.g., SPI, GPIO, etc. The latter APIs are not included in VBoot Export APIs because some x86 legacy reasons. They may be merged in the future.

The firmware boot path is like the following graph:

faft-firmware-boot-path

Our tests are mostly focused on VBoot Bootstub and VBoot Main Firmware since they are complicated and many decision branches. U-Boot initialization and kernel booting are simple and single path decisions. So full system regression tests are enough.

State of the Art

VBoot_Reference Library Tests

Source: src/platform/vboot_reference/tests

vboot_reference provides a lot of tests varying from crypto algorithms to vboot main logic and control flows. These tests stub the underlying APIs as all success functions. So it is unable to validate the correctness of underlying BIOS platform and the hardware behaviors.

faft-vboot-reference-tests

U-Boot vbexport/vboot Tests

Source: src/third_party/u-boot/files/common/cmd_vb*_test.c

Almost all the U-Boot drivers are abstracted into couple of major APIs: VBExport APIs and main logic used driver APIs. U-Boot provides a CLI for manual diagnostic. We wrote some commands to test these APIs. For example, for testing the disk read/write functions, a test command is to write some patterns to the disk and read them back to verify. However, some of these tests needs human involved, like inserting an USB stick to check if it is detectable.

faft-u-boot-vbexport-vboot-test

Factory Regression Testing Using Servo

Design Doc: https://docs.google.com/a/google.com/document/d/1XZgX4_v-Ps7YBRnjZmCFYjrjyBZQz5YmRrg7YWBFfV0/edit?hl=en_US

This work is an on-going project to ensure the whole factory install flow work fine. It downloads a factory bundle from buildbot, setup TFTP and Omaha servers. It uses a mini-Servo board to control a ChromeOS device, like switching dev switch, resetting device, sending Ctrl-D key, etc. This work is done by Rong Chang and the factory team to ensure partners get a good factory bundle.

Since using a Servo board can automate almost all the human behaviors. The test can be done fully automated. The proposed FAFT in this document is an extension of this work.

U-Boot Test System

Design Doc: https://docs.google.com/a/google.com/document/d/1JorndqNOrEOXgdMRuNBPk1o73XKl1JwCwLkDSOQy7wU/edit?hl=en_US

Simon proposed a U-Boot native test system to split the U-Boot components into platform code and generic U-Boot code + Verified Bood code. By mocking the platform code. we can test the functions of VBoot library and it integration in a platform-independent way. On the other side, the architecture-specific platform code can be tested in a separated way.

FAFT Implementation Details

Test Environment

We need a Servo board to connect a DUT such that we can:

flash a new firmware image;
run regression tests;
simulate lid close, power cycles, gpios, and key presses;
get console output via UART;
simulate USB plug in/out, by either muxing a physical USB stick or emulating a USB storage gadget in the host machine.

faft-test-environment

Test Harness

We use Autotest as our test harness and run FAFT as server tests. The host uses Ethernet to connect a DUT to push the test commands to it. Since Autotest is widely used in the ChromeOS projects. We can benefit in:

easy to integrate to buildbot;
easy to run remotely;
easy to deploy to the test lab.

Some existing works using Servo board test on top on Autotest can be found in:

src/third_party/autotest/files/server/site_tests/platform_CloseOpenLid
src/third_party/autotest/files/server/site_tests/platform_LongPressPower

FAFT is also based on the existing ServoTest approach and does some enhancements.

Software Architecture

Here is our proposed FAFT framework. The red pieces are missing and need to be implemented in this project.

faft-software-architecture

All the FAFT test cases are server side tests, meaning that all the actions are controlled by Host. It is flexible to keep states after sequential reboot. However, we still want to get the DUT state and run some command on it, like getting current firmware version, setting the fw_try_b flag, etc. These actions should be completed in the DUT. So we setup a FAFT Client for this purpose and use RPC to communicate with it.

Test Case Example

Here is an example to demonstrate how easy it is to write a FAFT test case. It test the firmware like:

FW A normal boot -> enable devsw + set fw_try_next=B -> reboot -> press ctrl-d
FW B developer boot -> do nothing -> reboot -> press ctrl-d
FW A developer boot -> disable devsw -> reboot -> do nothing
FW A normal boot -> finished

class firmware_DevModeTryB(FAFTSequence):
    def run_once(self, host=None):
        self.register_faft_sequence((
            # ------------------------- step 1 -------------------------
            {‘state_checker’: (self.faft_util.crossystem_checker,
                 {‘devsw_boot’: ‘0’, ‘mainfw_type’: ‘normal’, ‘mainfw_act’: ‘A’}),
             ‘userspace_action’: self.enable_devmode_and_set_try_fwb,
             ‘firmware_action’: self.wait_and_press_ctrl_d},
            # ------------------------- step 2 -------------------------
            {‘state_checker’, (self.faft_util.crossystem_checker,
                 {‘devsw_boot’: ‘1’, ‘mainfw_type’: ‘developer’, ‘mainfw_act’: ‘B’}),
             ‘userspace_action’: None,
             ‘firmware_action’: self.wait_and_press_ctrl_d},
            # ------------------------- step 3 -------------------------
            {‘state_checker’, (self.faft_util.crossystem_checker,
                 {‘devsw_boot’: ‘1’, ‘mainfw_type’: ‘developer’, ‘mainfw_act’: ‘A’}),
             ‘userspace_action’: self.servo.disable_development_mode,
             ‘firmware_action’: None},
            # ------------------------- step 4 -------------------------
            {‘state_checker’, (self.faft_util.crossystem_checker,
                 {‘devsw_boot’: ‘0’, ‘mainfw_type’: ‘normal’, ‘mainfw_act’: ‘A’}),
            }
        ))
        self.run_faft_sequence()

        def enable_devmode_and_set_try_fwb(self):
            self.faft_client.crossystem.set_try_fwb()
            self.servo.enable_development_mode()

        def wait_and_press_ctrl_d(self):
            time.sleep(10)
            self.servo.ctrl_d()

FAFT Scopes

There are several scopes about FAFT.

Scope 1: Test the Layer of VBExport APIs

In order to increase the test coverage and verify the underlying BIOS functions and drivers, a good entry is to re-run all existing vbexport / vboot tests in an automated way using our test framework!. See more detail in the State of the Art Section. We can catch bugs easily if any drivers changed or the BIOS source rebased.

faft-u-boot-vbexport-vboot-test

Scope 2: Verify Recovery, Factory Install, Firmware AU, RMA, and Many Complicated Flows

The normal boot and developer boot can be verified by SAFT. However, we still missed the test of the whole Recovery process. The whole recovery sequence is complicated:

DD the recovery shim image to a USB stick;
Hold Recovery button and power on DUT;
Press SPACE;
Insert a recovery shim;
Wait for the complete of recovery process;
Reboot;
Boot to normal ChromeOS.

This sequence can be fully automated by FAFT. For example, in step 1, we require an USB stick plugged to the Servo board by default. We first mux it to connect to the host machine such that we can write the recovery shim image to it. In step 4, we then mux the USB stick to the DUT. Another approach is to emulate an USB storage gadget in the host machine, it would be more flexible to control its functionalities by software.

Moreover, there are many recovery mode variants, like normal recovery/devloper recovery, physical recovery button/keyboard simulated button, etc. It’d be easy to verify all variants by writing different test cases.

Besides, factory install shim, firmware AU, reset shim, RMA, etc, are more complicated and have many variants. Factory install process currently will be tested by an automated script written by the factory team. Firmware AU still relies on manual tests. Reset shim and RMA has no test yet. Actually, all these complicated flows can be tested in FAFT.

Scope 3: Monkey Test during Firmware/EC update

Firmware/EC update is a risky thing and may brick the machine. For example, a bug reported, during the EC update, a user presses Power Button at the same time that bricks the machine. It is because the powerd triggers the shutdown process immediately and the system goes to off, but it cannot be turned on again by power button (until a cold boot). Note that the EC is still in update mode now.

To catch bugs like that, we need a monkey test during firmware/EC update, like randomly switching GPIO, pressing Power Button, closing LID, pressing keyboard, plugging USB/SD disk, etc.

Other Ideas

Run FAFT on every build, integrated with buildbot
Stress test to ensure the stability of hardware and drivers
Cold reboot test
Use HDMI capture device or camera + image recognition to verify the display output