fwtc: Document utility scripts usage

TEST=view in gitiles

Change-Id: I6c5262ed65ab43c63c0a42fd29c1750904bc47c2
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/fw-testing-configs/+/2576586
Tested-by: Greg Edelston <gredelston@google.com>
Reviewed-by: Kevin Shelton <kmshelton@chromium.org>
Commit-Queue: Greg Edelston <gredelston@google.com>
2 files changed
tree: 7351a21ed3d8b41cceeb3acfa305a1c42aaf07e4
  1. .gitignore
  2. .style.yapf
  4. DEFAULTS.json
  6. PRESUBMIT.cfg
  8. README.md
  9. arkham.json
  10. asuka.json
  11. asurada.json
  12. atlas.json
  13. auron.json
  14. banjo.json
  15. banon.json
  16. betty.json
  17. bob.json
  18. brain.json
  19. buddy.json
  20. candy.json
  21. caroline.json
  22. cave.json
  23. celes.json
  24. chell.json
  25. cheza.json
  26. consolidate.py
  27. consolidate_unittest.py
  28. coral.json
  29. cyan.json
  30. dedede.json
  31. dragonegg.json
  32. drallion.json
  33. edgar.json
  34. elm.json
  35. endeavour.json
  36. enguarde.json
  37. eve.json
  38. expresso.json
  39. fievel.json
  40. fizz.json
  41. gale.json
  42. gandof.json
  43. glados.json
  44. gnawty.json
  45. gru.json
  46. grunt.json
  47. guado.json
  48. hana.json
  49. hatch.json
  50. heli.json
  51. jacuzzi.json
  52. jecht.json
  53. jetstream.json
  54. kalista.json
  55. kefka.json
  56. kevin.json
  57. kevin64.json
  58. kevin_tpm2.json
  59. kip.json
  60. kukui.json
  61. kunimitsu.json
  62. lars.json
  63. lulu.json
  64. mickey.json
  65. mistral.json
  66. nami.json
  67. nasher.json
  68. nautilus.json
  69. ninja.json
  70. nocturne.json
  71. oak.json
  72. octopus.json
  73. orco.json
  74. paine.json
  75. pinky.json
  76. platform_json.py
  77. platform_json_unittest.py
  78. poppy.json
  79. puff.json
  80. pyro.json
  81. query_by_field.py
  82. query_by_field_unittest.py
  83. rambi.json
  84. rammus.json
  85. reef.json
  86. reef_uni.json
  87. reks.json
  88. relm.json
  89. rikku.json
  90. samus.json
  91. sand.json
  92. sarien.json
  93. scarlet.json
  94. sentry.json
  95. setzer.json
  96. slippy.json
  97. snappy.json
  98. soraka.json
  99. storm.json
  100. strago.json
  101. strongbad.json
  102. sumo.json
  103. swanky.json
  104. terra.json
  105. tidus.json
  106. tiger.json
  107. trogdor.json
  108. ultima.json
  109. veyron.json
  110. volteer.json
  111. whirlwind.json
  112. winky.json
  113. wizpig.json
  114. yuna.json
  115. zork.json

CrOS fw-testing-configs: User’s Guide



End-to-end firmware testing in ChromeOS relies on board-specific configuration files. Previously, those files were stored in Autotest. Now, they have been moved into a separate repository, fw-testing-configs, so that other testing frameworks (e.g. Tast, SerialTest) can access them too.

The design document for the new repository is at go/cros-fw-testing-configs.

Working With Configs

File Locations & Names

All the config files are located at the top directory of the fw-testing-configs repository.

The fw-testing-configs repository currently has two checkouts in the ChromeOS manifest: one inside of Autotest, and one inside of tast-tests.

There is one config file for each platform, named as ${PLATFORM}.json: for example, octopus.json. There are also one special config file, DEFAULTS.json and CONSOLIDATED.json. DEFAULTS.json contains default values and documentation for each attribute. CONSOLIDATED.json is a generated file, containing the config data from all the other JSON files.

File Contents

Each config file should contain a single object whose fields override the values specified in DEFAULTS.json. Any fields which do not have a corresponding value in DEFAULTS.json will be ignored.

There are a few special fields to be aware of:

  • platform (required): A string which should exactly match the config file’s basename (minus the .json extension).
  • parent (optional): A string which can contain the name of a parent platform, whose fields will be inherited with lower precedence. See “Inheritance” below. Example: asuka.json
  • models (optional): An object which can contain the names of any models, whose fields will be inherited with higher precedence. See “Inheritance” below. Example: octopus.json


There is an inheritance model in these config files. In all cases, the most specific configuration is used:

[Model > ] Platform [ > Parent [...] ] > DEFAULTS

For each attribute that exists in DEFAULTS.json (besides the documentation attributes):

  • If that attribute is given a value in the platform’s models[${MODEL}], then that is the value that is used.
  • Otherwise, if that attribute is given a value in the ${PLATFORM}.json, then that is the value that is used.
  • Otherwise, if ${PLATFORM}.json specifies a parent configuration via the parent attribute, then that is the value that is used.
  • Parent configurations can specify parents as well, and the inheritance recurses until a config file does not have a parent attribute.
  • Finally, for any attribute which was not otherwise given a value, the value from DEFAULTS.json is used.


Consolidate before uploading

CONSOLIDATED.json is a generated script containing the data from all other JSON files. This allows Tast tests to access the config data. Thus, it is important to keep CONSOLIDATED.json up-to-date.

When you make a change to fw-testing-configs JSON, before you upload, be sure to run ./consolidate.py (with no command-line args). This will update CONSOLIDATED.json. Amend your change to include the update CONSOLIDATED.json, and then re-upload.

A preupload hook will verify that your CONSOLIDATED.json is up-to-date.

Modifying Tests and Configs

fw-testing-configs is a separate repository from both Autotest and tast-tests. Thus, edits to fw-testing-configs require a separate CL from edits to Autotest and tast-tests. If you are accustomed to working with config files directly in Autotest, this is a slightly different workflow.

When you are editing config files, please be sure to run git commands from within the fw-testing-configs checkout. If you run git add . or repo upload --cbr . from within autotest/server/cros/faft/, then your changes to the config files will not be captured.

If you are modifying both test files and config files, then you will need at least two CL’s: one for the test change, and one for the fw-testing-configs change. Consider also whether your change will impact the other testing repositories (Autotest, Tast, SerialTest): you might need to run tests or make changes there, too.

When submitting multiple CL’s like that, please use the Cq-Depend syntax as appropriate. If you don‘t use Cq-Depend, you risk the name of a config attribute changing while tests are still looking for the old name, and then tests will break. The reverse is a risk, too. You might need both CL’s to depend on each other. Note that this while this precaution mitigates risk, it does not fully eliminate risk: only one of the CL’s might get uploaded to the lab machines running Autotest/Tast, or one CL may be reverted without the other.

Accessing Configs During Tests

In Autotest, configs are loaded during FirmwareTest.initialize. Config values can be accessed via self.faft_config.${ATTRIBUTE}, such as self.faft_config.chrome_ec.

In Tast, configs can be created via firmware.NewConfig. In order to conform with Go’s style, attribute names are modified to MixedCaps. Config values can be accessed via cfg[${ATTRIBUTE}], such as cfg[ChromeEC].

Utility Scripts

This repo offers a few handy-dandy utility scripts.

consolidate.py, as described above, combines all the individual ${PLATFORM}.json files into one file, CONSOLIDATED.json:

> consolidate.py

platform_json.py outputs the final configuration for a single platform. --model or -m will include a model-specific override. --field or -f will only print a single field value. --condense-output will trim unnecessary whitespace, making it easier to consume via other scripts.

> platform_json.py octopus
> platform_json.py octopus -m robo360
> platform_json.py octopus -f has_lid
> platform_json.py octopus -m robo360 -f has_lid --condense-output

query_by_field.py reports a list of all platforms and models matching certain field conditions. Multiple conditions can be specified. See usage info (query_by_field.py -h) for details about supported operators. Note that you might need to escape some of the operators (like \!) for Bash to parse them.

> query_by_field.py has_lid=false
> query_by_field.py "firmware_screen<=8"
> query_by_field.py chrome_ec=true ec_capability\!:x86

Each of these scripts also supports --help (or -h) for usage info.