En 30 segundos
Linear engineer Jacob describes growing an AI-agent evaluation dataset through real use, combining deterministic checks with a language-model judge for more subjective output quality.
- 1Grow the test setUnexpected real-world use becomes new evaluation data over time.
- 2Check objective behaviorA request for an in-progress status should result in the corresponding status.
- 3Judge subjective outputThe team uses a language-model judge for questions such as whether the title and description capture the right information.
Conclusiones
- Add unexpected user interactions to an evaluation dataset as the product is used.
- Check deterministic requirements, such as setting a requested status, directly.
- Use a model judge for more subjective criteria, such as extracting a useful title.
Lee el texto editado por capítulos
Evaluation from usage and feedback
Jacob explains how Linear combines test cases and model judgment.
Linear engineer Jacob says the team grows its AI-agent evaluation dataset through actual use. When a user behaves in an unexpected way, the team adds that case for future testing. Some checks are objective: if a user asks for an issue to be in progress, the agent should set that status. Other questions, such as whether a description is well organized or the right information became the title, are more subjective. For those, he describes using a language model as a judge against the expected information.
Más de este canal
46:40EN
Peter Yang· 8 nov 2026Pódcast · 2 vocesAshley Alexander: How AI Helps Me Care for My Parents With CancerInvitado: Ashley Alexander · Presenta: Peter YangA caregiver uses AI to prepare for appointments, organize records and ask better questions—while keeping doctors at the center of treatment.
17:50ENPeter Yang· 4 nov 2026PódcastPeter Yang’s Five Steps for Using AI to Help Care for His MotherYang uses a shared care brief and organized records to navigate appointments, while making time for family.
15:03ENPeter Yang· 3 nov 2026PódcastHow Peter Yang Used an AI Phone Agent to Negotiate His Internet BillAn AI call negotiated a lower price and faster service, but Yang still had to approve the change himself.
33:09ENPeter Yang· 1 nov 2026Pódcast · 2 vocesJaclyn Konzelmann: When AI Can Build More, Choose What Deserves to ExistInvitado: Jaclyn Konzelmann · Presenta: Peter YangGoogle Labs pairs short strategy documents with prototypes, user co-creation and lessons from experiments that end.
13:05ENPeter Yang· 28 oct 2026PódcastPeter Yang Tests Genie 4: Convincing Worlds, Unfinished GameplayFive live demonstrations show scenes that respond to prompts and sound, alongside inconsistent actions and limited game mechanics.
1:06ENPeter Yang· 26 oct 2026PódcastPeter Yang on Setting Up Payroll for a One-Person BusinessIn a paid partnership, Yang describes the paperwork he encountered and a payroll product integrated with a business account.