| [TOC] | |
| # Chromium Integration with ResultDB | |
| ResultDB is a LUCI service for storing and retrieving test results. See also | |
| its [public source] and [Google internal documentation]. As of Q4 2021, all | |
| tests on Chrome/Chromium builders have their test results uploaded to ResultDB, | |
| and nearly all pass-fail decisions on the builders are based on these results. | |
| Consequently, any JSON output from a test no longer has little to no impact on | |
| the results of the bots. (For example, if a test's JSON say that there was a | |
| failure but this isn't similarly reflected in ResultDB, then the build _will not | |
| fail._ | |
| However, the JSON output can still have an indirect effect on builds. See | |
| [below](#uses-of-the-legacy-json) for more details. | |
| ## ResultDB API & ResultSink | |
| All test harnesses are responsible for uploading their results to ResultDB. This | |
| is most easily done via ResultSink (see [internal documentation]). ResultSink is | |
| a local HTTP server that proxies requests to ResultDB's backend. It can be | |
| launched via the `rdb stream ...` command and listens on a port specified in a | |
| file pointed to by the `LUCI_CONTEXT` environment variable. This server | |
| currently runs in the background on all test bots. | |
| On a machine with the server running, a test can report its results to ResultDB | |
| by making JSON-formatted RPCs to the local HTTP ResultSink server. See | |
| [ResultSink's API] for more details. | |
| ## ResultSink integration/wrappers within Chromium | |
| There are several libraries used within Chromium that have ResultSink | |
| integration: | |
| - **Web Tests**: Blink's web tests upload their results through ResultSink via | |
| [test_result_sink.py]. | |
| - **typ**: Typ is a testing library used by performance benchmarks and GPU | |
| tests. It integrates with ResultSink in [result_sink.py]. | |
| - **//build/util/lib/results/**: [//build/util/lib/results/] contains a generic | |
| wrapper around ResultSink. This is used by both the Android and ChromeOS test | |
| runners to upload their results to ResultDB. | |
| - **iOS test runner**: The iOS test runner has its own ResultSink integration in | |
| [result_sink_util.py]. | |
| - **result_adapter**: Most remaining tests use the `result_adapter` tool. See | |
| below for more details. | |
| ## result_adapter | |
| [result_adapter] is a command-line tool that wraps a test invocation and will | |
| automatically convert the test's output JSON to the ResultSink format and | |
| uploads those results to ResultDB via ResultSink. Known JSON formats include: | |
| - `gtest` for the JSON generated by running GTests using the support code in | |
| //base/test/launcher/, specifically the `SaveSummaryAsJSON` function in | |
| [test_results_tracker.h]. | |
| - `json` for the JSON format described in [json_test_results_format.md]. Tests | |
| that can generate this type of JSON include Blink web tests and typ-supported | |
| tests. | |
| Though it eased the migration of results into ResultDB, using result_adapter has | |
| a few drawbacks: | |
| - result_adapter uploads all results at once after the test invocation exits. | |
| If the test execution crashes partway through, ResultDB will not track any of | |
| the results that were successfully produced. | |
| - result_adapter is limited by the JSON format of the test. It would be unable | |
| to use any new or advanced feature in ResultDB. | |
| Consequently, it's preferred when possible for new test types and new test | |
| harnesses to integrate with ResultSink directly rather than using | |
| result_adapter. But if circumstances make integration difficult (e.g. we don't | |
| have access to the test harness implementation) result_adapter can be needed. | |
| ### Specifying integration in //testing/buildbot/ | |
| The *.pyl spec files in [//testing/buildbot/] control what tests a given bot | |
| runs. These testing specs also control how result_adapter is used. By default, a | |
| `isolated_scripts` test will have result_adapter added using the `json` | |
| output format, and a `gtest_tests` test will have result_adapter added using the | |
| `gtest` output format. This can be overwritten using the | |
| `has_native_resultdb_integration` [mixin] which will disable result_adapter for | |
| that test. Additionally, a custom `result_format` can be specified for a test to | |
| overwrite the expected format of the JSON: [example]. | |
| ## Uses of the legacy JSON | |
| Although the JSON output of tests don't have an immediate effect on builds, | |
| they can still impact pass/fail decisions. This happens during the exoneration | |
| phase of the build which determines if a test failure was introduced by the | |
| CL-under-test or if the failure is also present on ToT. This is accomplished by | |
| an RPC to [FindIt], which is a service that tracks failures across bots. Rather | |
| than using ResultDB, FindIt uses the [legacy test results app] as its source of | |
| truth for test results, and this app uses the results in a test's JSON. | |
| Consequently, the JSON output of tests can still have an effect on builds, just | |
| in an indirect way. | |
| Migration/replacement [efforts] for this phase of the build are underway. | |
| [public source]: https://source.chromium.org/chromium/infra/infra/+/main:go/src/go.chromium.org/luci/resultdb/ | |
| [Google internal documentation]: http://shortn/_bTdqm8VDXz | |
| [internal documentation]: http://shortn/_zAbl5fa84c | |
| [ResultSink's API]: https://source.chromium.org/chromium/infra/infra/+/main:go/src/go.chromium.org/luci/resultdb/sink/proto/v1/sink.proto;drc=54f060e7452368ff982d9c66f2c1001bf4fa7394;l=24 | |
| [test_result_sink.py]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/tools/blinkpy/web_tests/controllers/test_result_sink.py | |
| [result_sink.py]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/typ/typ/result_sink.py | |
| [//build/util/lib/results/]: https://source.chromium.org/chromium/chromium/src/+/main:build/util/lib/results/ | |
| [result_sink_util.py]: https://source.chromium.org/chromium/chromium/src/+/main:ios/build/bots/scripts/result_sink_util.py | |
| [result_adapter]: https://source.chromium.org/chromium/infra/infra/+/main:go/src/infra/tools/result_adapter/ | |
| [test_results_tracker.h]: https://source.chromium.org/chromium/chromium/src/+/main:base/test/launcher/test_results_tracker.h;drc=96020cfd447cb285acfa1a96c37a67ed22fa2499;l=83 | |
| [json_test_results_format.md]: json_test_results_format.md | |
| [//testing/buildbot/]: https://source.chromium.org/chromium/chromium/src/+/main:testing/buildbot/ | |
| [mixin]: https://source.chromium.org/chromium/chromium/src/+/main:testing/buildbot/mixins.pyl;drc=d0985a69618056e95d64b48803ca90e3ae6a6c77;l=453 | |
| [example]: https://source.chromium.org/chromium/chromium/src/+/main:testing/buildbot/test_suites.pyl;drc=9ef43df31342fb0fc854de5233b7170039028bc1;l=1499 | |
| [FindIt]: https://chromium.googlesource.com/infra/infra/+/main/appengine/findit/README.md | |
| [efforts]: https://crbug.com/1243747 | |
| [legacy test results app]: https://test-results.appspot.com/ |