docs/fuchsia/gtests.md

# Deploying and running gtests on Fuchsia

[TOC]

Fuchsia gtest binaries are deployed and executed via scripts that are
automatically generated by the `test()` GN target. For each test, a wrapper
scripts is created named `run_<test_target_name>` for running the Fuchsia
package on the device. These executables are found in `${OUTPUT_DIR}/bin`.

The aforementioned devices can be either emulators started by the scripts
themselves, an existing emulator instance, or a physical device. To build a
gtest binary, check this [documentation](build_instructions.md).

For the sake of this example, we will be using `base_unittests` as the package
we wish to install and/or execute.

If using anything other than the default Fuchsia boot image, see
[Supported Fuchsia Product configurations](#supported-fuchsia-product-configurations).

For details of the implementation of `run_<test_target_name>` scripts, see
[here](test_scripts.md).

## Hermetic emulation

The test script brings up an emulator, runs the tests on it, and
shuts the emulator down when finished.
```bash
$ out/fuchsia/bin/run_base_unittests
```

The flag `--custom-image` can be used to specify the Fuchsia boot image used
by the emulator. For instance, setting
`--custom-image=workstation.qemu-x64-release` would run the test on a
workstation image.

## Run on a persistent emulator

You can also run tests on physical devices or start a persistent emulator
from the Chromium tree. To start a persistent emulator, run:

```bash
$ ./build/fuchsia/test/start_emulator.py
```

Part of the output should be like the following:

```
INFO:root:Emulator successfully started. You can now run Chrome Fuchsia tests with --target-id=fuchsia-emulator-198 to target this emulator.
```

You can run tests on persistent devices by adding the command line
arguments indicated above. For example, to run `base_unittests` you
would run the following command:

```bash
$ out/fuchsia/bin/run_base_unittests --target-id fuchsia-emulator-198
```

If only one Fuchsia device is connected, you can use the following command:

```bash
$ out/fuchsia/bin/run_base_unittests -d
```

## Run on a device paved with Fuchsia built from source

Make sure that the CPU architecture of your Chromium output directory matches
the architecture of the Fuchsia output directory (x64==x64, arm64==arm64, etc.).

```bash
$ out/fuchsia/bin/run_base_unittests -d --repo [FUCHSIA_OUT_DIR]/amber-files --no-repo-init
```

Note that you should not have `fx serve` running as Chromium will handle serving
the repository.

## Run on a device the host is connected to remotely via ssh

```bash
$ out/fuchsia/bin/run_base_unittests --target-id=[::1]:8022
```

## Troubleshooting a test

To troubleshoot a specific test, consider a combination of the following
arguments to the test runner script:

* `--gtest_filter="[TestSuite.TestName]"` to only run a specific test, or a
  comma-separated list to run a set of tests. Wildcards can also be used to run
  a set of tests or an entire test suite, e.g. `--gtest_filter="[TestSuite.*]"`.
* `--test-launcher-jobs=1` to only have one batch of tests running at a time.
  This bypasses the test launcher buffering of test log output, making it
  possible to access the log output from successful test runs.
* `--single-process-tests` to run all the tests in the same process. Unlike the
  above option, this will run the tests directly in the test launcher process,
  making it easier to attach a debugger.
* "--logs-dir=[/path/to/log/directory]` to specify the directory to write logs
  to. By default, Chromium logs are written to the "system_log" file in that
  directory.
* `--gtest_repeat=[number] --gtest_break_on_failure` to run a test or test suite
  a certain number of times until it fails. This is useful to investigate flaky
  tests.

## Supported Fuchsia Product configurations

The prebuilt Terminal and Workstation images downloaded by `gclient sync` are
specifically configured to support running Chromium tests. It is also possible
to run Chromium tests on other Product configurations, such as Core, using a
custom build.

**Use the default unless you have a specific reason to run on a different
Product.** For example, if you need to reproduce an issue occurring on a bot
using that Product.

### Terminal
Chromium gtests are primarily supported (i.e., pass all tests) on the Terminal
Fuchsia Product configuration. This is the default image used for
[hermetic emulation](#hermetic-emulation) and when starting a
[persistent emulator](#run-on-a-persistent-emulator).

### Workstation
Workstation configurations also generally support most tests.

### Core

By default, Fuchsia Core Product configurations do **not** support running
Chromium tests. Similarly, the Core images provided with the Fuchsia SDK are
unable to run Chromium tests. (In the future, it may be possible to run them
on such images [with a corresponding TUF repo](https://crbug.com/1033794).)

When working with a local Fuchsia Core build, there are have two options for
running Chromium tests.
1. **Combined repo**: Follow the instructions in
[Run on a device paved with Fuchsia built from source](#run-on-a-device-paved-with-fuchsia-built-from-source).
2. **Packages in base**: Add the
[additional required packages](#additional-required-packages) to the
[base packages](https://fuchsia.dev/fuchsia-src/concepts/packages/package?hl=en#base-packages).
<!-- Resolving needed packages before running the tests does not work. -->

[Using method (1)](#combined-repo) with the default configuration for
`core.qemu-x64`, all `base_unittests` pass.
The same tests pass using method (2) with
[modifications](#chromium-only-repo-packages-in-base) to `core.qemu-x64`'s
configuration.

Most other test suites will require additional packages used (indirectly) by
those tests. Some of these packages may not even be in Core's default set of
[universe packages](https://fuchsia.dev/fuchsia-src/concepts/packages/package?hl=en#universe-packages)
and need to be explicitly added as described in
[Adding packages](#adding-packages).

#### Additional required packages
At a minimum, `test_manager` is required in order to run tests. This is already
available when using a combined repo as in option (1).

<!-- TODO(crbug.com/42050571): Remove this paragraph when the packages for fakes
     are subpackaged with the tests.
-->
Another required package is
`//src/developer/build_info/testing:fake-build-info`, which is not available in
Core by default.

Other useful packages include:
* `intl_property_manager` - avoids many warnings when the test shard tries
  to launch it.
* `audio_core` - avoids warnings related to `fuchsia.media.ProfileProvider`.
  <!-- TODO(crbug.com/42050523): Replace `audio_core` with the appropriate
       Package name and path here and below, respectively, when switching to
       `fuchsia.scheduler.ProfileProvider`.
  -->

This is sufficient to pass `base_unittests` and eliminate most warnings.

#### Adding packages
To add packages, find the path to the package's build target in the Fuchsia code
and add `--with[-base]` followed by `//path_from_fuchsia_root:target_name` to
the `fx set` command for the Fuchsia build.

##### Combined repo

When using a combined repo as in option (1), packages that are already part of
the Core configuration (including
[cached](https://fuchsia.dev/fuchsia-src/concepts/packages/package?hl=en#cached-packages)
and
[universe](https://fuchsia.dev/fuchsia-src/concepts/packages/package?hl=en#universe-packages)
packages), are available. Thus, only packages _not_ included in Core's
[base, cached, or universe packages](https://fuchsia.dev/fuchsia-src/concepts/packages/package?hl=en#types_of_packages),
need to be added. Similarly, `--with`, `--with-cache`, and `--with-base` are all
acceptable ways to add such packages.

The following makes all
[additional required packages](#additional-required-packages) available to the
Chromium tests.
```bash
$ fx set core.qemu-x64 --auto-dir --release --with //src/developer/build_info/testing:fake-build-info --with //src/testing/fidl/intl_property_manager --with //src/media/audio/audio_core
```

##### Chromium-only repo: packages in base
By default - and specifically when not using option (1) - Chromium's
[test scripts](test_scripts.md) only serve packages from the Chromium workspace,
meaning that Fuchsia provided packages not in Core's
[base packages](https://fuchsia.dev/fuchsia-src/concepts/packages/package?hl=en#base-packages)
cannot be resolved. Thus, all Fuchsia packages needed during the tests must be
added to base.

In this case `--with-base` is the only acceptable method of specifying packages,
and `test_manager` must be explicitly added to base packages by including
`--with-base //src/sys/test_manager`. The following is the minimum `fx set`
command line for this case.
```bash
$ fx set core.qemu-x64 --auto-dir --release --with-base //src/sys/test_manager
```

For the purposes of successfully running Chromium tests, the following is
equivalent to the command line in [Combined repo](#combined-repo). It adds all
[additional required packages](#additional-required-packages) to
[base packages](https://fuchsia.dev/fuchsia-src/concepts/packages/package?hl=en#base-packages).
It also adds `//sdk/lib/sys/cpp` to base packages to enable `base_unittests`'s
`TestComponentContextForProcessTest.ProvideSystemService` test to pass.
```bash
$ fx set core.qemu-x64 --auto-dir --release  --with-base //src/sys/test_manager  --with-base //src/developer/build_info/testing:fake-build-info --with-base //src/testing/fidl/intl_property_manager --with-base //src/media/audio/audio_core --with-base //sdk/lib/sys/cpp
```

### Other Products
Other Fuchsia Product configurations may not support running Chromium tests by
default and require some subset of the
[additional required packages](#additional-required-packages) needed for
[Core](#core) as well as others for specific test suites.