30초 요약
Linear engineer Jacob describes growing an AI-agent evaluation dataset through real use, combining deterministic checks with a language-model judge for more subjective output quality.
- 1Grow the test setUnexpected real-world use becomes new evaluation data over time.
- 2Check objective behaviorA request for an in-progress status should result in the corresponding status.
- 3Judge subjective outputThe team uses a language-model judge for questions such as whether the title and description capture the right information.
핵심 정리
- Add unexpected user interactions to an evaluation dataset as the product is used.
- Check deterministic requirements, such as setting a requested status, directly.
- Use a model judge for more subjective criteria, such as extracting a useful title.
챕터별로 다듬은 전문 읽기
Evaluation from usage and feedback
Jacob explains how Linear combines test cases and model judgment.
Linear engineer Jacob says the team grows its AI-agent evaluation dataset through actual use. When a user behaves in an unexpected way, the team adds that case for future testing. Some checks are objective: if a user asks for an issue to be in progress, the agent should set that status. Other questions, such as whether a description is well organized or the right information became the title, are more subjective. For those, he describes using a language model as a judge against the expected information.
같은 채널의 다른 글
46:40EN
Peter Yang· 2026년 11월 8일팟캐스트 · 2명Ashley Alexander: How AI Helps Me Care for My Parents With Cancer게스트 Ashley Alexander · 진행자 Peter YangA caregiver uses AI to prepare for appointments, organize records and ask better questions—while keeping doctors at the center of treatment.
17:50ENPeter Yang· 2026년 11월 4일팟캐스트Peter Yang’s Five Steps for Using AI to Help Care for His MotherYang uses a shared care brief and organized records to navigate appointments, while making time for family.
15:03ENPeter Yang· 2026년 11월 3일팟캐스트How Peter Yang Used an AI Phone Agent to Negotiate His Internet BillAn AI call negotiated a lower price and faster service, but Yang still had to approve the change himself.
33:09ENPeter Yang· 2026년 11월 1일팟캐스트 · 2명Jaclyn Konzelmann: When AI Can Build More, Choose What Deserves to Exist게스트 Jaclyn Konzelmann · 진행자 Peter YangGoogle Labs pairs short strategy documents with prototypes, user co-creation and lessons from experiments that end.
13:05ENPeter Yang· 2026년 10월 28일팟캐스트Peter Yang Tests Genie 4: Convincing Worlds, Unfinished GameplayFive live demonstrations show scenes that respond to prompts and sound, alongside inconsistent actions and limited game mechanics.
1:06ENPeter Yang· 2026년 10월 26일팟캐스트Peter Yang on Setting Up Payroll for a One-Person BusinessIn a paid partnership, Yang describes the paperwork he encountered and a payroll product integrated with a business account.