30 秒でわかる
Linear engineer Jacob describes growing an AI-agent evaluation dataset through real use, combining deterministic checks with a language-model judge for more subjective output quality.
- 1Grow the test setUnexpected real-world use becomes new evaluation data over time.
- 2Check objective behaviorA request for an in-progress status should result in the corresponding status.
- 3Judge subjective outputThe team uses a language-model judge for questions such as whether the title and description capture the right information.
ポイント
- Add unexpected user interactions to an evaluation dataset as the product is used.
- Check deterministic requirements, such as setting a requested status, directly.
- Use a model judge for more subjective criteria, such as extracting a useful title.
チャプターごとに整えた全文を読む
Evaluation from usage and feedback
Jacob explains how Linear combines test cases and model judgment.
Linear engineer Jacob says the team grows its AI-agent evaluation dataset through actual use. When a user behaves in an unexpected way, the team adds that case for future testing. Some checks are objective: if a user asks for an issue to be in progress, the agent should set that status. Other questions, such as whether a description is well organized or the right information became the title, are more subjective. For those, he describes using a language model as a judge against the expected information.
同じチャンネルの記事
46:40EN
Peter Yang· 2026年11月8日ポッドキャスト · 2 人Ashley Alexander: How AI Helps Me Care for My Parents With Cancerゲスト Ashley Alexander · ホスト Peter YangA caregiver uses AI to prepare for appointments, organize records and ask better questions—while keeping doctors at the center of treatment.
17:50ENPeter Yang· 2026年11月4日ポッドキャストPeter Yang’s Five Steps for Using AI to Help Care for His MotherYang uses a shared care brief and organized records to navigate appointments, while making time for family.
15:03ENPeter Yang· 2026年11月3日ポッドキャストHow Peter Yang Used an AI Phone Agent to Negotiate His Internet BillAn AI call negotiated a lower price and faster service, but Yang still had to approve the change himself.
33:09ENPeter Yang· 2026年11月1日ポッドキャスト · 2 人Jaclyn Konzelmann: When AI Can Build More, Choose What Deserves to Existゲスト Jaclyn Konzelmann · ホスト Peter YangGoogle Labs pairs short strategy documents with prototypes, user co-creation and lessons from experiments that end.
13:05ENPeter Yang· 2026年10月28日ポッドキャストPeter Yang Tests Genie 4: Convincing Worlds, Unfinished GameplayFive live demonstrations show scenes that respond to prompts and sound, alongside inconsistent actions and limited game mechanics.
1:06ENPeter Yang· 2026年10月26日ポッドキャストPeter Yang on Setting Up Payroll for a One-Person BusinessIn a paid partnership, Yang describes the paperwork he encountered and a payroll product integrated with a business account.