docs/testing/resultdb.md

[TOC]

# Chromium Integration with ResultDB

ResultDB is a LUCI service for storing and retrieving test results. See also
its [public source] and [Google internal documentation]. As of Q4 2021, all
tests on Chrome/Chromium builders have their test results uploaded to ResultDB,
and nearly all pass-fail decisions on the builders are based on these results.
Consequently, any JSON output from a test no longer has little to no impact on
the results of the bots. (For example, if a test's JSON say that there was a
failure but this isn't similarly reflected in ResultDB, then the build _will not
fail._

## ResultDB API & ResultSink

All test harnesses are responsible for uploading their results to ResultDB. This
is most easily done via ResultSink (see [internal documentation]). ResultSink is
a local HTTP server that proxies requests to ResultDB's backend. It can be
launched via the `rdb stream ...` command and listens on a port specified in a
file pointed to by the `LUCI_CONTEXT` environment variable. This server
currently runs in the background on all test bots.

On a machine with the server running, a test can report its results to ResultDB
by making JSON-formatted RPCs to the local HTTP ResultSink server. See
[ResultSink's API] for more details.

## ResultSink integration/wrappers within Chromium

There are several libraries used within Chromium that have ResultSink
integration:

- **Web Tests**: Blink's web tests upload their results through ResultSink via
  [test_result_sink.py].

- **typ**: Typ is a testing library used by performance benchmarks and GPU
  tests. It integrates with ResultSink in [result_sink.py].

- **//build/util/lib/results/**: [//build/util/lib/results/] contains a generic
  wrapper around ResultSink. This is used by both the Android and ChromeOS test
  runners to upload their results to ResultDB.

- **iOS test runner**: The iOS test runner has its own ResultSink integration in
  [result_sink_util.py].

- **result_adapter**: Most remaining tests use the `result_adapter` tool. See
  below for more details.

## result_adapter

[result_adapter] is a command-line tool that wraps a test invocation and will
automatically convert the test's output JSON to the ResultSink format and
uploads those results to ResultDB via ResultSink. Known JSON formats include:

- `gtest` for the JSON generated by running GTests using the support code in
  //base/test/launcher/, specifically the `SaveSummaryAsJSON` function in
  [test_results_tracker.h].

- `json` for the JSON format described in [json_test_results_format.md]. Tests
  that can generate this type of JSON include Blink web tests and typ-supported
  tests.

Though it eased the migration of results into ResultDB, using result_adapter has
a few drawbacks:

- result_adapter uploads all results at once after the test invocation exits.
  If the test execution crashes partway through, ResultDB will not track any of
  the results that were successfully produced.

- result_adapter is limited by the JSON format of the test. It would be unable
  to use any new or advanced feature in ResultDB.

Consequently, it's preferred when possible for new test types and new test
harnesses to integrate with ResultSink directly rather than using
result_adapter. But if circumstances make integration difficult (e.g. we don't
have access to the test harness implementation) result_adapter can be needed.

### Specifying integration in //testing/buildbot/

The *.pyl spec files in [//testing/buildbot/] control what tests a given bot
runs. These testing specs also control how result_adapter is used. By default, a
`isolated_scripts` test will have result_adapter added using the `json`
output format, and a `gtest_tests` test will have result_adapter added using the
`gtest` output format. This can be overwritten using the
`has_native_resultdb_integration` [mixin] which will disable result_adapter for
that test. Additionally, a custom `result_format` can be specified for a test to
overwrite the expected format of the JSON: [example].

[public source]: https://source.chromium.org/chromium/infra/infra/+/main:go/src/go.chromium.org/luci/resultdb/
[Google internal documentation]: http://shortn/_bTdqm8VDXz
[internal documentation]: http://shortn/_zAbl5fa84c
[ResultSink's API]: https://source.chromium.org/chromium/infra/infra/+/main:go/src/go.chromium.org/luci/resultdb/sink/proto/v1/sink.proto;drc=54f060e7452368ff982d9c66f2c1001bf4fa7394;l=24
[test_result_sink.py]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/tools/blinkpy/web_tests/controllers/test_result_sink.py
[result_sink.py]: https://source.chromium.org/chromium/chromium/src/+/main:third_party/catapult/third_party/typ/typ/result_sink.py
[//build/util/lib/results/]: https://source.chromium.org/chromium/chromium/src/+/main:build/util/lib/results/
[result_sink_util.py]: https://source.chromium.org/chromium/chromium/src/+/main:ios/build/bots/scripts/result_sink_util.py
[result_adapter]: https://source.chromium.org/chromium/infra/infra/+/main:go/src/infra/tools/result_adapter/
[test_results_tracker.h]: https://source.chromium.org/chromium/chromium/src/+/main:base/test/launcher/test_results_tracker.h;drc=96020cfd447cb285acfa1a96c37a67ed22fa2499;l=83
[json_test_results_format.md]: json_test_results_format.md
[//testing/buildbot/]: https://source.chromium.org/chromium/chromium/src/+/main:testing/buildbot/
[mixin]: https://source.chromium.org/chromium/chromium/src/+/main:testing/buildbot/mixins.pyl;drc=d0985a69618056e95d64b48803ca90e3ae6a6c77;l=453
[example]: https://source.chromium.org/chromium/chromium/src/+/main:testing/buildbot/test_suites.pyl;drc=9ef43df31342fb0fc854de5233b7170039028bc1;l=1499