The new test lists will be written in JSON format, and will be loaded by cros.factory.test.test_lists.manager
.
A test list file is a JSON file containing one object, the object can have following fields:
inherit
: a list of strings, e.g. ["a.test_list", "b.test_list"]
. Specifies base config files for this config file, e.g. a.test_list.json
and b.test_list.json
. Fields constants
, options
, definitions
will be loaded and merged with current config files. Inheritance order is resolved by C3 linearization.constants
: key value pairs to define some constants that could be used later. Please refer to Evaluation section for usage of constants.options
: test list options, please refer to cros.factory.test.test_lists.test_list.Options
.definitions
: define some reusable test objects.tests
: a list of test objects, which are the top level tests in this test list.__comment
: just a comment. The test list manager will ignore this field.override_args
: a dictionary where keys are test paths you want to override test arguments, values are the test arguments that will be merged with the old one.You can find examples under ./manager_unittest/
folder, e.g. a.test_list.json.
Each test object represents a cros.factory.test.factory.FactoryTest
object (with some additional information). Here are some attributes you can set to a test object.
Each test item must have an ID. ID will be used to define the path of a test. path is defined as:
test.path = test.parent.path + '.' + test.ID
For example, the test group SMT
will have test path SMT
, and the test item ProbeCamera
will have test path SMT.ProbeComponents.ProbeCamera
.
ID
can be auto generated, see Syntactic Sugar for more details.
label
is a string that will be shown on UI. label
will be treated as an i18n string by default.
Leaf nodes (the test objects have no subtests) of a test list should be a pytest. A pytest is a test written in python and placed under py/test/pytests/
in a public or private factory source tree.
Each pytest can define their arguments, the ARGS
variable in the class. And the args
is used to assign values to these arguments. args
is a dictionary of key value pairs where keys are mapped to the name of each argument in ARGS
.
To create a group of tests, you just need to
{ "label": "Test Group Label", "subtests": [ "a", "b", "c" ] }
By default, the test group will be tested in a sequence, that is, if any subtest will be retested, all of the subtests will be retested in order. To create a group that subtests can be run individually, inherit from TestGroup
:
{ "label": "Test Group Label", "inherit": "TestGroup", "subtests": [ "a", "b", "c" ] }
Each test object can have a locals
attribute, which is a dictionary of key value pairs. locals
will be available when Goofy is resolving args
that will be passed to pytest.
Set allow_reboot
to true
if you want to allow unexpected shutdown or reboot when running the test.
More specifically, when goofy starts up, it checks if the last running test is marked as allow_reboot
or not. If it‘s false
, it means an unexpected shutdown just happened, so it stops all pending tests and waits for further manual inspection. If it’s true
, it means the shutdown is allowed or expected, and the test will be re-run.
Please use this option with care. The rule of thumb is to properly shutdown the DUT whenever possible, by using the shutdown test. One usage of this option is at SMT line, when display is not attached to the mlb, and the operator might move the mlb to different test stations to perform the test multiple times.
To make two or more tests run at the same time, you need to group them under a test group, and add attribute "parallel": true
to this test group.
The action_on_failure
attribute allows you to decide what the next test should be when this test fails. There are three possible values:
NEXT
: this is the default value, the test list will continue on the next test item.PARENT
: stop running other tests under the current parent. Let parent decide what the next test should be.STOP
: stop running other tests.Sometimes, you want a test be run after some tests are finished, no matter those tests success or not. For example, a test item that uploads log files to the shopfloor server should always be run despite the status of previous tests.
{ "label": "Test Group Label", "subtests": [ { "normal tests...", { "label": "Tear Down Group", "teardown": true, "subtests": [ ... ] } } ] }
Tests in teardowns can have their subtests as well. Those tests will become teardown tests as well. We assume that teardowns will never fail. If a teardown test fails, Goofy will ignore the failure and continue on the next teardown test. Therefore, for teardown tests, action_on_failure
will always be set to NEXT
.
Test list manager will process these fields, they are not directly used by FactoryTest
.
inherit
: a string, the name of the base test object, default to FactoryTest
. The base test object should be defined in definitions
section. For example, you can define a LEDTest
test object in definitions
section:
{ "definitions": { "LEDTest": { "pytest_name": "led", "args" : { "colors": ["RED", "GREEN"] } } } }
Now you can use “LEDTest” as base test object:
{ "tests": [ { "id": "MyFirstLEDTest", "inherit": "LEDTest" } ] }
If a test object defined in definitions
section inherits itself, it means that it's a class defined in cros.factory.test.test_lists.test_object
module. For example, the definition of FactoryTest
is:
{ "definitions": { "FactoryTest": { "inherit": "FactoryTest" } } }
Please refer to base test list for more examples.
locals
: in JSON style pytest, attribute locals
of a test object will be passed to its subtests. And locals
will be evaluated before being set to the FactoryTest
object. See locals.test_list.json
as an example.
child_action_on_failure
: default value of action_on_failure
of subtests.
__comment
: this field will be ignored and discarded by the test list manager, it's just a comment.
override_args
: another way to override arguments of a test specified by its path. See override_args.test_list.json
as an example.
For args
of a test object, if a value is a string and starts with "eval! "
, the rest of the string will be interpreted as a python expression. The expression will be evaluated by the python eval
statement. However, for simplicity, the expression has the following restrictions:
Single expression (not necessary single line, but the parsed result is a single expression)
Not all operators are allowed, currently, the following expressions are not allowed:
(x * x for x in range(10))
)lambda x: x * x
)yield
, return
When executing, the namespace will only contains:
constants
and options
dut
, station
locals
state_proxy
(created by state.GetInstance()
)device
(a shortcut for state_proxy.data_shelf.device
)The result of the evaluated expression will not be evaluated again even if it starts with "eval! "
as well.
You can use "i18n! English string"
to specify an i18n string. Or, an alternative way to create an i18n string is: {"en-US": "English string", "zh-CN": "Chinese string"}
. Therefore, both definitions.Start.label
and definitions.Start.args.prompt
are valid i18n strings.
{ "definitions": { "Start": { "pytest_name": "start", "label": "i18n! Start", "args": { "prompt": {"en-US": "English prompt", "zh-CN": "Chinese prompt"} } } } }
For simplicity, we provide the following syntactic sugar:
We support auto generated labels from pytest_name
. For example, if the pytest name is some_test
, then the label will be i18n! Some Test
by default.
Since the label should always be an i18n string, so "label": "Start"
is equivalent to "label": "i18n! Start"
.
If a test object defined in definitions
(before merged with base test object) does not contain id
field, the key string will be the default ID, for example:
{ "definitions": { "LEDTest": { "pytest_name": "led", "args" : { "colors": ["RED", "GREEN"] } } } }
The test object LEDTest
will have default "id": "LEDTest"
.
If a test object defined in tests
does not specify id
field, it will inherit from its parent, therefore, in the following snippet,
{ "tests": [ { "inherit": "LEDTest" }, { "inherit": "LEDTest" } ] }
The first led test will be: LEDTest
, the second one will be: LEDTest_2
(_2
is automatically appended to resolve path duplication).
If the test object contains only inherit
field, you can just write a string, so the previous example can be simplified to:
{ "tests": [ "LEDTest", "LEDTest" ] }
generic_main.test_list.json is an example. And you should reuse generic_*.test_list.json
to create a test list for your board.
generic_common.test_list.json
This defines all kinds of tests that might be used in the Chromebook factory. By inheriting this test list, you can use most pytests directly.
generic_<station>.test_list.json
We have defined GRT
, FAT
, RUN_IN
, FFT
, SMT
stations. All of them are based on generic_common.test_list.json
, with some additional station specific tests, or test arguments overriding.
Try to reuse generic test lists if possible. In general, you need to define the following files in private overlay:
common.test_list.json
: this inherits generic_common.test_list.json
, overrides some test arguments, and adds some board specific pytests.<station>.test_list.json
: each of them inherits corresponding generic_<station>.test_list.json
and common.test_list.json
.main.test_list.json
: this inherits <station>.test_list.json
and common.test_list.json
(and generic_main.test_list.json
if you want to inherit options
and constants
). Defines override_args
, constants
, options
in this file, and also the tests
.In most of the cases, if you are not adding / removing / reordering any tests, you only need to override constants
, options
and override_args
.
On a DUT, you can use factory dump-test-list <test list id> --format json
to dump a test list in JSON format.
This is a special test list which only runs ModelSKU, updates fingerprint firmware and finalization. The test list could be useful if you would like to test different versions of recovery image on a finalized DUT. A typical use case could be:
IMPORTANT: Please complete the finalization step. If the pre-condition is not met, then you can not ship the device because it violates factory requirements.
By default, the main_wipe test list inherits from generic_fat and generic_grt test lists. You should modify the inherited test list accordingly. For example, inherit from common test list. Moreover, you should also modify the name of this test list to main_.test_list.json. So that the DUT will execute this test list automatically after booting into test image.
ModelSKU
and Fingerprint Test Group
?Finalization will reinitialize FPMCU entropy and this will cause some problems. (see b/194449380 for more info). Therefore, we need to first run ModelSKU to get the device data, then reflash the FPFW, and finally, run finalization. If the DUT you're using does not have a fingerprint sensor, then you can remove ModelSKU and fingerprint test group from the test list.