William Schor: From Unit Tests to Synthetic Canaries at Netflix

I work on Netflix’s key-value team, which owns Cassandra databases and layers that provide a key-value interface to the company. I want to trace how our testing evolved and how we tried to make those techniques useful to other online data-storage teams.

Tests clarify our thinking, express intent and preserve a contract, but I think their central purpose is confidence: enough confidence to change code and deploy it. Production experience can also give confidence because the code has survived actual runtime conditions.

A test and the implementation can share the same mistaken assumption. Code review, real dependencies and eventually production can disagree with a passing build. I want to bring some of the independence and fidelity of production experience earlier into the testing process.

Unit tests remain useful as contracts, communication and exercises in thinking through a design. But hand-picked cases can leave gaps. In our key-value store, scans of all records and all keys worked; scans of one record and one key worked too. Scanning all records for a particular key, however, triggered a failure that remained hidden for years.

Property-based testing helps explore combinations without enumerating each example by hand. In the scan example, a two-by-two matrix of record selection and key selection makes the missing combinations visible. Generating cases across that space could have found the bug.

We also compare different record-store implementations under generated inputs. An established implementation can act as an oracle for a new one, giving us more confidence as we add backends. This still assumes the implementations do not all share the same error.

Property testing is not a silver bullet. In one case, an external producer generated filenames with a prefix while our code expected no prefix. Extensive internal tests shared the wrong assumption. Integrating the real producer’s code into a test hardened the contract between teams.

Integration tests still cover slices of a larger system. Different providers, caches and databases create a combinatorial space. One combination of store and cache violated read-your-writes consistency, but our tests had not exercised that combination. Adding a test only after each incident keeps us behind the expanding system.

A smoke test can start the whole service and exercise reads and writes, improving fidelity. Yet if workloads are still handwritten and fixed, it becomes a large unit test. The cost of creating each new workload discouraged complete coverage over time.

We considered copying production traffic to a candidate cluster. For stateful services, sending writes to real data can risk corruption. Sending them to a separate data source avoids that risk but creates a fidelity gap: a missing read might reflect a bug or simply data absent from the copy.

Our goal became combining production-like infrastructure with the broad generated inputs of property testing. Synthetic canaries use a service specification, typically gRPC at Netflix, to drive a generic traffic generator. Each service supplies its specification and workload definitions rather than hand-coding every test.

The tooling provisions the infrastructure variants to test. A workload can specify the domain of fields, such as a range of IDs with a Zipfian distribution, as well as item counts and sizes. This gives us a way to model the shapes of traffic we expect rather than a single fixed example.

Requests can refer back to earlier steps. For example, after a put, a get can reuse the same ID and assert that the returned value matches. The library combines a gRPC specification with YAML generator descriptions, sends the resulting requests and checks relationships between them.

A canary can compare existing and candidate code, including performance. Integrated with resilience tooling, the approach can provision infrastructure and generate very large workloads, potentially millions of requests per second, approximating traffic shapes observed in production.

There is an important limitation: when a correctness violation occurs amid millions of requests, reproducing the exact database state and sequence can be very difficult. Developers must reduce the failing case, and other tools may help. High-load testing can reveal a problem without making it easy to diagnose.

The broader vision is to combine generic traffic generation with infrastructure provisioning: run multiple versions or configurations of a service, exercise relevant traffic patterns across them, and check the results centrally. The online data-stores group has been pioneering this approach for wider use.