30초 요약
Shreya Shankar and Hamel Husain review Peter Yang's podcast-writing evaluations and demonstrate a five-step error-discovery workflow. They distinguish general task requirements from failures found in real examples, show how human annotations become reusable criteria, and explain why automated evaluators still need checks for missed errors and false positives.
- 1Two sources of evaluation criteriaTop-down criteria follow from the task; bottom-up criteria emerge when people inspect examples and notice failures.
- 2Five steps to error discoveryUnderstand the data, design its visual representation, build a review app, choose diverse samples and iterate on human feedback.
- 3Review the reviewer's flagsIn the demo, an overly broad short-sentence rule generated many noisy suggestions.
- 4Automatic evaluation has blind spotsHusain's comparison found that tools caught obvious errors but missed some failures requiring knowledge of the product and sales workflow.
핵심 정리
- Separate top-down task requirements from bottom-up failure modes discovered by reviewing real examples.
- Let agents build a suitable interface, select diverse samples and organize human feedback into themes.
- Turn reviewed failure modes into focused checks and validate those checks against annotated examples.
- Inspect automatic suggestions for false positives and missing context; evaluation tools do not replace product judgment.
- Avoid overfitting every editorial preference to a single interview or treating every rule as equally important.
챕터별로 다듬은 전문 읽기
Reviewing Peter's existing evals
The guests distinguish task-level criteria from failures found in actual outputs.
AI evaluation starts with looking at actual outputs. In this interview, evaluation educators Shreya Shankar and Hamel Husain show Peter Yang how an AI agent can help organize that review while a person supplies the judgment. The episode is sponsored by Wispr Flow; its description also contains an affiliate link to the guests' course.
Yang shows a skill that turns podcast transcripts into takeaways and drafts. His current checks include measurable requirements, such as length, and subjective questions, such as whether a takeaway makes sense on its own. He still gives the model feedback and edits its work across multiple rounds.
Shankar distinguishes top-down criteria, inferred from understanding the task, from bottom-up criteria, discovered by reviewing many real outputs. A model can help propose the former, she says, but the latter depends on people expressing judgments they discover in the data.
Yang worries that adding a new rule after every interview could overfit one episode. Shankar suggests separating the two kinds of criteria, using focused agents to check groups of rules, and presenting pass/fail results in a table. A person can then decide which failures actually matter for this particular piece.
One way to discover recurring errors is to compare generated takeaways across several transcripts with the final human-edited versions. Husain adds that writing is especially subjective: criteria can conflict, and a rule that helps one piece may be unnecessary in another.
Building an error-discovery interface
Shankar demonstrates five steps using synthetic writing samples.
Shankar demonstrates her error-discovery skill on synthetic AI-written articles. It works in five steps: understand the data's content; design a suitable visual encoding; build a review interface; cluster and select diverse examples; and run an interactive loop that turns human annotations into themes and suggests more samples.
The interface depends on what is being reviewed. Articles can be displayed as readable prose; an agent trace or support conversation would call for a different view. The agent selects examples from clusters so the reviewer can encounter a range of outputs rather than only similar ones.
From annotations to reusable criteria
Human feedback becomes themes, suggestions and evaluators that still require validation.
In the demonstration, Shankar reads samples and marks passages she dislikes, including formulaic contrasts and repetitive lists. She can give open-ended feedback without naming a polished category. An agent monitors the annotations, groups them into possible failure modes and scans other records for similar examples.
The scan can itself produce noise. The demo briefly produces hundreds of suggestions, including many short sentences matched by an overly broad rule. Reviewers should inspect suggestions and refine the criteria rather than treating every automatic flag as a genuine error.
The resulting failure modes can become individual rubric criteria, model judges or monitoring checks. The annotated examples also form a dataset for checking how well those judges match human decisions. Shankar recommends reviewing enough examples to gain confidence rather than drawing conclusions from only a handful.
Husain notes that the evaluation workflow can also inform the product interface: a writing tool might show specific suggestions and allow a writer to accept or reject them instead of silently rewriting a draft. The right interface depends on the task and the people using it.
Where automated evaluation falls short
Husain discusses a trace-based comparison and the continuing need for human judgment.
Husain describes his team's comparison of automated evaluation tools and coding agents against human-annotated traces from a real-estate sales assistant. He says they caught many obvious failures, but often missed subtler issues that required product knowledge, such as failing to offer alternatives to a prospect or putting Markdown in text messages.
The guests argue that these tools are useful as a baseline, but their findings need review. Shankar says the tested systems' best precision was roughly 80–90%, so even detected issues can include false positives. That number comes from their comparison, not a universal rate; both missed failures and incorrect flags matter.
To apply the process to his podcast workflow, Yang could collect transcripts, generated takeaways and approved human-edited versions, then review diverse cases and turn recurring problems into checks. The guests conclude that AI can accelerate discovery and apply criteria at scale, while people remain responsible for inspecting data and defining quality.
같은 채널의 다른 글
46:40EN
Peter Yang· 2026년 11월 8일팟캐스트 · 2명Ashley Alexander: How AI Helps Me Care for My Parents With Cancer게스트 Ashley Alexander · 진행자 Peter YangA caregiver uses AI to prepare for appointments, organize records and ask better questions—while keeping doctors at the center of treatment.
17:50ENPeter Yang· 2026년 11월 4일팟캐스트Peter Yang’s Five Steps for Using AI to Help Care for His MotherYang uses a shared care brief and organized records to navigate appointments, while making time for family.
15:03ENPeter Yang· 2026년 11월 3일팟캐스트How Peter Yang Used an AI Phone Agent to Negotiate His Internet BillAn AI call negotiated a lower price and faster service, but Yang still had to approve the change himself.
33:09ENPeter Yang· 2026년 11월 1일팟캐스트 · 2명Jaclyn Konzelmann: When AI Can Build More, Choose What Deserves to Exist게스트 Jaclyn Konzelmann · 진행자 Peter YangGoogle Labs pairs short strategy documents with prototypes, user co-creation and lessons from experiments that end.
13:05ENPeter Yang· 2026년 10월 28일팟캐스트Peter Yang Tests Genie 4: Convincing Worlds, Unfinished GameplayFive live demonstrations show scenes that respond to prompts and sound, alongside inconsistent actions and limited game mechanics.
1:06ENPeter Yang· 2026년 10월 26일팟캐스트Peter Yang on Setting Up Payroll for a One-Person BusinessIn a paid partnership, Yang describes the paperwork he encountered and a payroll product integrated with a business account.