How to Build Better AI Evaluations: Shreya Shankar and Hamel Husain on Finding Real Failure Modes

RESUMEN

En 30 segundos

Shreya Shankar and Hamel Husain review Peter Yang's podcast-writing evaluations and demonstrate a five-step error-discovery workflow. They distinguish general task requirements from failures found in real examples, show how human annotations become reusable criteria, and explain why automated evaluators still need checks for missed errors and false positives.

1Two sources of evaluation criteriaTop-down criteria follow from the task; bottom-up criteria emerge when people inspect examples and notice failures.
2Five steps to error discoveryUnderstand the data, design its visual representation, build a review app, choose diverse samples and iterate on human feedback.
3Review the reviewer's flagsIn the demo, an overly broad short-sentence rule generated many noisy suggestions.
4Automatic evaluation has blind spotsHusain's comparison found that tools caught obvious errors but missed some failures requiring knowledge of the product and sales workflow.

Conclusiones

Separate top-down task requirements from bottom-up failure modes discovered by reviewing real examples.
Let agents build a suitable interface, select diverse samples and organize human feedback into themes.
Turn reviewed failure modes into focused checks and validate those checks against annotated examples.
Inspect automatic suggestions for false positives and missing context; evaluation tools do not replace product judgment.
Avoid overfitting every editorial preference to a single interview or treating every rule as equally important.

TemasEvaluaciones AI Strategy Agentes de programación

TEXTO COMPLETO

Lee el texto editado por capítulos

Ver transcripción

Solo la transcripción

Reviewing Peter's existing evals

The guests distinguish task-level criteria from failures found in actual outputs.

AI evaluation starts with looking at actual outputs. In this interview, evaluation educators Shreya Shankar and Hamel Husain show Peter Yang how an AI agent can help organize that review while a person supplies the judgment. The episode is sponsored by Wispr Flow; its description also contains an affiliate link to the guests' course.

Yang shows a skill that turns podcast transcripts into takeaways and drafts. His current checks include measurable requirements, such as length, and subjective questions, such as whether a takeaway makes sense on its own. He still gives the model feedback and edits its work across multiple rounds.

Shankar distinguishes top-down criteria, inferred from understanding the task, from bottom-up criteria, discovered by reviewing many real outputs. A model can help propose the former, she says, but the latter depends on people expressing judgments they discover in the data.

Yang worries that adding a new rule after every interview could overfit one episode. Shankar suggests separating the two kinds of criteria, using focused agents to check groups of rules, and presenting pass/fail results in a table. A person can then decide which failures actually matter for this particular piece.

One way to discover recurring errors is to compare generated takeaways across several transcripts with the final human-edited versions. Husain adds that writing is especially subjective: criteria can conflict, and a rule that helps one piece may be unnecessary in another.

Building an error-discovery interface

Shankar demonstrates five steps using synthetic writing samples.

Shankar demonstrates her error-discovery skill on synthetic AI-written articles. It works in five steps: understand the data's content; design a suitable visual encoding; build a review interface; cluster and select diverse examples; and run an interactive loop that turns human annotations into themes and suggests more samples.

DIAPOSITIVA 1 / 5Five-step error discoveryThe skill reads data, designs and builds an interface, selects samples and monitors feedback.

The interface depends on what is being reviewed. Articles can be displayed as readable prose; an agent trace or support conversation would call for a different view. The agent selects examples from clusters so the reviewer can encounter a range of outputs rather than only similar ones.

From annotations to reusable criteria

Human feedback becomes themes, suggestions and evaluators that still require validation.

DIAPOSITIVA 2 / 5Reviewing an articleThe generated interface presents a sample alongside an agent monitoring annotations.

In the demonstration, Shankar reads samples and marks passages she dislikes, including formulaic contrasts and repetitive lists. She can give open-ended feedback without naming a polished category. An agent monitors the annotations, groups them into possible failure modes and scans other records for similar examples.

DIAPOSITIVA 3 / 5Grouping failure modesThe progress view organizes reviewer notes into possible patterns.

The scan can itself produce noise. The demo briefly produces hundreds of suggestions, including many short sentences matched by an overly broad rule. Reviewers should inspect suggestions and refine the criteria rather than treating every automatic flag as a genuine error.

DIAPOSITIVA 4 / 5Checking automated suggestionsThe demo shows why broad matching rules can produce noisy flags.

The resulting failure modes can become individual rubric criteria, model judges or monitoring checks. The annotated examples also form a dataset for checking how well those judges match human decisions. Shankar recommends reviewing enough examples to gain confidence rather than drawing conclusions from only a handful.

Husain notes that the evaluation workflow can also inform the product interface: a writing tool might show specific suggestions and allow a writer to accept or reject them instead of silently rewriting a draft. The right interface depends on the task and the people using it.

Where automated evaluation falls short

Husain discusses a trace-based comparison and the continuing need for human judgment.

Husain describes his team's comparison of automated evaluation tools and coding agents against human-annotated traces from a real-estate sales assistant. He says they caught many obvious failures, but often missed subtler issues that required product knowledge, such as failing to offer alternatives to a prospect or putting Markdown in text messages.

DIAPOSITIVA 5 / 5A comparison of evaluation toolsHusain presents his team's results for one annotated dataset, including precision and recall.

The guests argue that these tools are useful as a baseline, but their findings need review. Shankar says the tested systems' best precision was roughly 80–90%, so even detected issues can include false positives. That number comes from their comparison, not a universal rate; both missed failures and incorrect flags matter.

To apply the process to his podcast workflow, Yang could collect transcripts, generated takeaways and approved human-edited versions, then review diverse cases and turn recurring problems into checks. The guests conclude that AI can accelerate discovery and apply criteria at scale, while people remain responsible for inspecting data and defining quality.