Cameron Wolfe on Evaluating AI When Answers Are Subjective

I’m Cameron Wolfe, a staff research scientist at Netflix. This talk is an overview of how to evaluate highly subjective tasks, such as creative writing, where there may be no single verifiable right answer.

A primary difficulty in evaluation is noise. Agents have nondeterministic behavior, so repeated runs can produce different trajectories and answers. When the benchmark contains only a limited number of tasks, that variability makes performance differences harder to interpret.

In a simple evaluation trial, a suite supplies tasks, an agent produces a rollout and an outcome, and a grader assigns a result. Some benchmarks are expensive to run, limiting the number of tasks or repeated runs a team can afford.

Tasks lie on a spectrum of verifiability. A math answer may permit a deterministic check, while creative writing requires judgment about open-ended qualities. Replacing a deterministic grader with a model judge adds another potential source of variability.

My framework combines human evaluation, offline automatic evaluation and production monitoring. These components inform one another: humans define quality, automated proxies enable faster iteration, and production provides evidence about real use.

Start with human evaluation as the reference for a subjective task. Work with subject-matter experts to define a small set of criteria, rating scales and detailed examples, so reviewers have a shared understanding of what they are evaluating.

Good guidelines alone do not guarantee agreement. Ask multiple reviewers to assess the same examples, study where they disagree, discuss ambiguity and revise the guidelines. Repeat this calibration until agreement is appropriate for the task.

Sampling matters too. Power analysis can estimate the number of examples needed to detect an effect of a chosen size, given assumptions about variability, significance level and desired statistical power. It is a planning tool, not a guarantee that every conclusion will be correct.

The evaluation set should reflect real production tasks, include diverse cases and be difficult enough to distinguish models. Profiling rollouts can reveal what a model solves or misses, while human and model-based review can improve data quality over time.

Human review of every model change quickly becomes a bottleneck. Offline automatic evaluation uses an imperfect proxy—often a language-model judge or a rubric—to make a larger number of experiments practical.

In the distinction I use here, an LLM judge applies high-level quality guidelines across tasks. A rubric breaks a particular task into a checklist of criteria that can be assessed separately. Both approaches still require validation against the intended quality standard.

I show a Dr. Tulu example involving an adaptation of a book for Netflix. Rather than relying on one general impression, the rubric lists specific features to check. Making criteria explicit can improve consistency, though it does not eliminate judgment or bias.

Meta-evaluation checks whether the automatic judge agrees with trustworthy human-labeled “golden” examples. Depending on the task, teams can examine correlation, classification accuracy and particular patterns of disagreement rather than relying on a single headline number.

In a Netflix project I worked on, we linked an LLM judge’s outputs to streaming outcomes such as abandonment. I describe this as evidence relevant to that project; the talk does not provide enough detail to generalize its causal claim to all judges or viewing contexts.

How much agreement is enough depends on the task. One practical reference is agreement among human reviewers: compare the judge’s agreement with people against the level people achieve with one another, while still examining important errors.

With both a stochastic model and a stochastic judge, uncertainty needs explicit treatment. I recommend planning sample size, reporting error bars and using repeated samples or resampling where appropriate. The design must account for the sources of dependence and variability in the experiment.

Use automatic evaluation to explore many changes, then bring promising candidates back to human reviewers. Statistical intervals help quantify sampling uncertainty, but they do not remove systematic bias or prove that one model is better in every setting.

Before a full deployment, monitor production signals. These can be explicit ratings, such as thumbs up or down, or implicit measures, such as session length and abandonment. Each is a proxy whose meaning depends on the product and user context.

An A/B test compares a candidate with the existing experience and can serve as a controlled, limited rollout. Power analysis can help plan duration and sample size; guardrails and ongoing monitoring remain important after a full release.

Production can reveal new failure cases and use patterns missing from the offline set. Feed those examples back into the evaluation data, improve guidelines and recalibrate reviewers or judges as the product evolves.

The complete loop is to use offline evaluation for breadth, human evaluation for a higher-fidelity view, and A/B tests plus monitoring to check real-world behavior. At each stage, be skeptical of noisy results and communicate the uncertainty alongside the score.