Authors: dawidn@google.com
Reviewers: tomhughes@chromium.org
Last Updated: 2024-05-09
Run all tests that are listed in the run_device_tests.py with Zephyr FPMCU firmware.
There is a dedicated run_device_tests.py script that runs tests on FPMCU. The tests are run on a dev board like dragonclaw, with the servod connected. The script builds full firmware with a test, flashes it, runs with a console command and checks results. For some tests, the script also runs an additional console command e.g. reboot ro, or runs dut-control, e.g. _dut-control fpmcu_slp:1 _if it is needed.
As a part of migration FPMCU to Zephyr, the tests have to be migrated as well.
There are a few rather independent topics to cover.
For tests written in C, Ztest is an obvious choice. There are plenty of such tests in EC and Zephyr.
For Cpp tests - try to use GTest, but this has to be explored and confirmed. Otherwise use Ztest as well.
From migrating perspective there are 3 types of test:
Some tests may not be viable with Zephyr for a few reasons, e.g. functionality is not supported with Zephyr or a feature is already tested by Zephyr tests. To skip a test, an additional flag in the run_device_tests script should be used with a reason.
Some tests should use the same logic but need some adjustments. The abort test is a good example. There is a check of the panic data after the abort call. CrosEC uses its version of the abort function, which sets a panic reason. Zephyr uses its version. That means the check of the panic data is different - Zephyr uses saved registers and CrosEC checks the panic reason. Additionally, Zephyr version uses its mechanism (syswork_q) to run a test after reboot caused by a test (test staging).
There is a way to “translate” CrosEC tests into Zephyr Ztest with a mapping done in test_util.h. However the mentioned adjustments would require many #ifdef statements in the test code and additional shim layer for test staging, which is what we want to avoid. It would probably cause more unexpected problems.
There are also tests that will be copy-paste with test asserts replacement. As it is not a perfect solution, it is hard to find a better one. To avoid desynchronisation of the CrosEC and Zephyr tests, there is a precheck that displays a warning when a CrosEC version is edited.
Additionally, there are tests that can be run not only on hardware. These ones should be placed alongside other tests and used with a proper config, e.g. crc test.
CrosEC builds a full firmware image + test, which is run with a console command. Zephyr follows that approach. There are already a few configs that allow adding test code to a firmware image built with zmake.
The run_device_tests.py script has been already adjusted to build Zephyr EC with zmake and flash it with flash_ec. It is able to parse a twister style yaml file, to set proper configs per test. The yaml file is at hardcoded location at the moment.
An obvious alternative is using twister, but it implicates a few changes.
Twister doesn’t use zmake - no RO/RW parts and rollback regions. Some tests run by run_device_tests.py require that.
Using twister also means setting configs per test. It is a correct approach for unit tests e.g. CRC test doesn’t need fingerprint code. However there are more complex examples which would require additional work to prepare a sufficient set configs per test.
Additionally, there are extra actions taken by the run_device_tests.py script that are not supported by twister by default. It includes changing WP, setting fp_sensor_sel and fpmcu_slp signals and measuring power consumption. Flashing would also need to be investigated.
The twister alternative is definitely worth exploring. It looks like the main task would be to group tests into types and clearly mark those ones that are not pure unit tests and require additional, external actions. It would allow us to switch to fully native Zephyr testing solutions.
