doc/fixtures.md

# Test Fixtures

If your fuzz test requires setup/teardown logic that is too expensive to be
embedded in your [property function](fuzz-test-macro.md#the-property-function)
and executed on each fuzz test iteration, you can use a
test fixture.
In FuzzTest, any default-constructible class can serve as a test fixture.
To instantiate it, use the macro 
FUZZ_TEST_F,
which corresponds to [`FUZZ_TEST`](fuzz-test-macro.md) in the same way that
`TEST_F` corresponds to `TEST` in GoogleTest. Furthermore, you can easily adapt
and reuse the existing
[GoogleTest fixtures](https://google.github.io/googletest/primer.html#same-data-multiple-tests)
in your fuzz tests.

[TOC]

## When to use fixtures in fuzz tests

Most best practices for writing unit tests
(https://abseil.io/tips/122)
apply equally well when writing fuzz tests. In view of that, you should
generally avoid fixtures and prefer standalone helper functions for your fuzz
test's initialization. However, in some situations test fixtures are
appropriate.

1.  If your fuzz test requires expensive setup, e.g., initializing a server,
    doing it in the property function may be prohibitively expensive. The
    fuzzing engine calls the property function once per fuzz test iteration, and
    running the setup each time may slow down fuzzing to a point where it
    becomes ineffective.

2.  If you want to add a fuzz test to an existing code base that uses test
    fixtures for unit tests, refactoring the code to eliminate fixtures may be
    impractical or even impossible.

## Using a fixture for expensive setup

Suppose we wanted to test that a certain HTTP server called EchoServer echoes
back the string it receives. We could do this with a fuzz test that doesn't use
fixtures.

```c++
// A standalone property function that tests the property of interest.
void ReturnsTheSameString(const std::string& request) {
  // Initialize the server.
  EchoServer server;
  server.Start("localhost:9999");

  // Test the property.
  std::string response;
  SendRequest("localhost:9999", request, &response);
  EXPECT_EQ(response, request);

  // Tear down the server.
  server.Stop();
}

// Instantiate the fuzz test using the FUZZ_TEST macro with a chosen suite name
// and the property function as the test name.
FUZZ_TEST(EchoServerFuzzTest, ReturnsTheSameString)
    .WithDomains(fuzztest::Arbitrary<std::string>())
    .WithSeeds({"Hello"});
```

The issue with this fuzz test is that the server initialization and teardown
happen in each fuzz test iteration, that is, in each call to the property
function. If starting and stopping the server takes non-trivial time, the fuzz
test will be slow and ineffective.

To improve the test, we would like to do the initialization and teardown once
and reuse the same server across all fuzz test iterations. We can do this using
a test fixture:

```c++
// In FuzzTest, any default-constructible class can be a test fixture.
class EchoServerFuzzTest {
 public:
  // The constructor initializes the fixture.
  EchoServerFuzzTest() { server_.Start("localhost:9999"); }

  // The destructor tears down the fixture.
  ~EchoServerFuzzTest() { server_.Stop(); }

  // The fuzz test's property function must be a public member of the fixture.
  void ReturnsTheSameString(const std::string& request) {
    std::string response;
    SendRequest("localhost:9999", request, &response);
    EXPECT_EQ(response, request);
  }

 private:
  EchoServer server_;
};

// Instantiate the fuzz test using the FUZZ_TEST_F macro, with the fixture as
// the suite name and the property function as the test name.
FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString)
    // The macro supports the FUZZ_TEST clauses for specifying domains and seeds.
    .WithDomains(fuzztest::Arbitrary<std::string>())
    .WithSeeds({"Hello"});
```

FuzzTest will instantiate the fixture once and call the property function
on the same object in each fuzz test iteration.

### Note on mutable test fixtures

While persisting state across fuzz test iterations can be effective, it can also
have unwanted consequences. Consider the following example.

```c++
class MyFuzzTest {
 public:
  void MyProperty(int n) {
    MyApi(n, flag_);
    flag_ = !flag_;
  }

 private:
  bool flag_ = false;
};
FUZZ_TEST_F(MyFuzzTest, MyProperty);
```

By mutating `flag_` and using it in the call to `MyApi()`, the test no longer
depends only on the input generated by the fuzzer. This can potentially break
the fuzzer's assumptions about how the inputs affect coverage and make fuzzing
ineffective. Additionally, a crashing input found by the fuzzer may not be
sufficient for reproducing the crash, since the crash also depends on a
particular value of `flag_`.

BEST PRACTICE: Make sure that the property function resets the mutated fixture
so that each fuzz test iteration starts in the same state.

## Using existing GoogleTest fixtures in fuzz tests

Note: Only use GoogleTest fixtures if they are shared with existing unit tests,
or if you need static per-fixture setup and teardown (via `SetUpTestSuite` and
`TearDownTestSuite`). Otherwise, prefer fixtureless code with
[standalone initialization functions](https://abseil.io/tips/122).

If you already use a GoogleTest fixture derived from `::testing::Test` in your
unit tests and you want to reuse it in fuzz tests, you first need to choose when
you want the fixture to be instantiated and destroyed. GoogleTest fixtures are
typically built around a guarantee that the same fixture object will never be
reused for multiple tests or multiple runs of the same test. In a fuzz test,
this guarantee requires instantiating the fixture *once per fuzz test
iteration*, i.e., calling the property function each time on a fresh fixture
object. However, for increased performance, it is possible to instantiate a
fixture *once per fuzz test* (if the fixture supports such persistence) and
reuse the same fixture object in all calls to the property function. As noted
earlier, this comes with the risk of state mutation, which can lead to
non-reproducible crashes, so design your fixture carefully.

### Semantics 1: Fixture instantiated once per fuzz test iteration

Adapt a fixture called `Fixture` by extending
`::fuzztest::PerIterationFixtureAdapter<Fixture>`. This case should be the more
common one, since it observes the GoogleTest invariant of not using a fixture
object more than once.

As an example, suppose we have a function `SumVec()` that returns the sum of a
vector of integers and we have the following fixture.

```c++
class SumVecTest : public testing::Test {
 public:
  SumVecTest() : vec_{1, 2, 3} {}

 protected:
  std::vector<int> vec_;
};
```

A fuzz test that uses the fixture would look like the following:

```c++
// Adapts the fixture using the "per-iteration" semantics.
class SumVecFuzzTest : public fuzztest::PerIterationFixtureAdapter<SumVecTest> {
 public:
  void SumsLastEntry(int last_entry) {
    int previous_sum = SumVec(vec_);
    vec_.push_back(last_entry);
    EXPECT_EQ(SumVec(vec_), previous_sum + last_entry);
  }
};
FUZZ_TEST_F(SumVecFuzzTest, SumsLastEntry);
```

The "per-iteration" semantics is appropriate for this fixture because the test
mutates `vec_`.

### Semantics 2: Fixture instantiated once per fuzz test

Adapt a fixture called `Fixture` by extending
`::fuzztest::PerFuzzTestFixtureAdapter<Fixture>`. This case should be the less
common one: it relies on the fixture being easily resettable, which is not a
typical concern when designing GoogleTest fixtures.

As an example, let's return to EchoServer. Suppose we already have a GoogleTest
fixture that sets up the server.

```c++
class EchoServerTest : public testing::Test {
 public:
  EchoServerTest() { server_.Start("localhost:9999"); }
  ~EchoServerTest() override { server_.Stop(); }

 private:
  EchoServer server_;
};
```

We would write a test using this fixture as in the following code:

```c++
// Adapts the fixture using the "per-fuzz-test" semantics.
class EchoServerFuzzTest
    : public fuzztest::PerFuzzTestFixtureAdapter<EchoServerTest> {
 public:
  void ReturnsTheSameString(const std::string& request) {
    std::string response;
    SendRequest("localhost:9999", request, &response);
    EXPECT_EQ(response, request);
  }
};
FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);
```

Here, the "per-fuzz-test" semantics is appropriate because the test doesn't
mutate the server in any way that would be relevant for the test, and the server
initialization is too expensive to be performed in each iteration.

## Advanced fixtures with runner functions

When using non-GoogleTest fixtures, FuzzTest can make use of *runner functions*
on the fixture to specify per-test or per-iteration setups in a more flexible
way. FuzzTest provides the following runner interfaces for a fixture to inherit:

-   `::fuzztest::FuzzTestRunnerFixture`, which declares the runner function
    `void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test)`

    If the interface is inherited, instead of directly running the fuzz test,
    FuzzTest will call `FuzzTestRunner`, expecting that it will in turn:

    1.  Set up everything needed for running the test.
    2.  Invoke `run_test`, through which FuzzTest will essentially pass the
        whole fuzzing loop with multiple invocations to the property function.
        The function `run_test` will return once FuzzTest is done with the test.
    3.  Tear down everything no longer needed after the test.

-   `::fuzztest::IterationRunnerFixture`, which declares the runner function
    `void FuzzTestIterationRunner(absl::AnyInvocable<void() &&> run_iteration)`

    If the interface is inherited, instead of directly running a fuzz test
    iteration, FuzzTest will call `FuzzTestIterationRunner`, expecting that it
    will in turn:

    1.  Set up everything needed for running a test iteration.
    2.  Invoke `run_iteration`, through which FuzzTest will invoke the property
        function once. The function `run_iteration` will return once the
        iteration is done.
    3.  Tear down everything no longer needed after the test iteration.

Note that the type `absl::AnyInvocable<void() &&>` ensures that the functions
`run_test` and `run_iteration` can be called only once. To call them, you must
first move them, for example `std::move(run_test)()`. Attempting to call these
functions more than once will result in a runtime error.

The traditional SetUp/TearDown functions of a fixture can be trivially
transformed into a runner function. For example, the `EchoServerFuzzTest`
fixture could be defined as:

```c++
class EchoServerFuzzTest : public fuzztest::FuzzTestRunnerFixture {
 public:

  void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test) override {
   server_.Start("localhost:9999");
   std::move(run_test)();
   server_.Stop();
  }

  // The fuzz test's property function
  void ReturnsTheSameString(const std::string& request) {
    std::string response;
    SendRequest("localhost:9999", request, &response);
    EXPECT_EQ(response, request);
  }

 private:
  EchoServer server_;
};

FUZZ_TEST_F(EchoServerFuzzTest, ReturnsTheSameString);
```

To create a new `EchoServer` instance for each fuzz test iteration, define
`FuzzTestIterationRunner` instead:

```c++
class EchoServerFuzzTest : public fuzztest::IterationRunnerFixture {
 public:

  void FuzzTestIterationRunner(absl::AnyInvocable<void() &&> run_iteration) override {
   server_.Start("localhost:9999");
   std::move(run_iteration)();
   server_.Stop();
  }

  ...
```

Although converting SetUp/TearDown to runner functions is trivial, the other way
around may not even be possible. For example, suppose there is a fuzz test whose
property function must run within an SUT created by an SUT method
`RunWithinSut(sut_ready_cb)`, which starts the SUT and calls the callback when
the SUT is ready, and shuts down the SUT when the callback returns. Suppose
additionally that `RunWithinSut` must be run in the main thread of the test,
which is also the thread where FuzzTest runs the test fixture. Given these
constraints, it is impossible to run such test with a SetUp/TearDown-style
fixture without calling `RunWithinSut` for every iteration, which may be too
heavy and impact the fuzzing performance. With runner functions, the fixture can
be defined as:

```c++
class SutBasedFuzzTest : public fuzztest::FuzzTestRunnerFixture {
 public:

  void FuzzTestRunner(absl::AnyInvocable<void() &&> run_test) {
   sut_.RunWithinSut([&] { std::move(run_test)(); });`.
  }

 private:
  SUT sut_;
}
```