Evaluate LLMs by the Work You Need Them to Do

Human ratings, model judges and benchmarks each miss different failures; Stanford’s instructors show how to combine and test them.

AAAfshine Amidi연사 · Stanford UniversitySAShervine Amidi연사 · Stanford University
채널
Stanford Online
게시일
2026년 11월 21일
영상
1:44:59
읽기
11분
원문
영어
YouTube에서 열기
스크롤하면 영상이 오른쪽 아래로 작아져 계속 재생됩니다. 자막은 읽기 언어를 따릅니다.
읽기 언어2개 언어로 읽을 수 있음
이 글은 아직 한국어 번역이 없어 영어 원문으로 표시합니다.
요약

30초 요약

A high benchmark score does not guarantee that an AI system will do your work well. Stanford instructors Afshine and Shervine Amidi recommend defining task-specific success, using deterministic tests where possible and calibrating model judges against human ratings. For agents, inspect tool selection, execution and outputs separately. Public benchmarks can be overfit, so private tests and comparisons of quality, cost and latency help teams make more useful choices.

  1. 1Raters can agree by chanceCohen’s kappa compares observed agreement with an expected chance baseline; clear labeling rules matter particularly for subjective criteria.
  2. 2Pass@k measures whether any of several attempts succeedsIt is useful when tests can identify a working solution, but less directly useful when the system cannot check the answer.
  3. 3A model judge needs its own evaluationHuman-labeled examples and error analysis help refine a judge prompt, with precision and recall measuring distinct failures.
  4. 4A judge’s preference can change with answer orderThe lecture describes position, verbosity and self-enhancement biases and ways to test or mitigate them.
  5. 5An agent failure can begin in the tool interfaceTrace the selected tools, their arguments, execution and returned data before deciding how to fix a bad final response.
  6. 6Small benchmark changes can change model rankingsThe instructors describe a study in which small changes to requirements lowered scores; they present benchmark overfitting as a possible explanation.
  7. 7Compare models on quality and cost togetherThe Pareto frontier identifies options that offer the best quality available at a given cost, with latency another possible constraint.

핵심 정리

  1. Define the specific quality criteria and real tasks that matter before choosing a model.
  2. Use executable checks when an answer can be verified; calibrate model judges against human labels when it cannot.
  3. Test model judges for position, verbosity and self-preference biases.
  4. Inspect agent traces at tool selection, execution and response generation to locate failures.
  5. Treat public benchmark ranks as evidence, not a substitute for private tests and quality–cost comparisons.
전문

챕터별로 다듬은 전문 읽기

원본 자막만 보기
01

Why quality has more than one dimension

A correct answer can still be badly phrased, and many prompts have more than one valid answer. Multi-step agents add more places for errors.

Evaluating an LLM starts with deciding what “good” means. Quality may mean correctness, relevance and safety; reliability includes consistency and recovery from failures; a product team may also care about whether people adopt and benefit from the application. This lecture focuses on quality.

The range of tasks makes quality difficult to measure: an LLM can write code, summarize, solve mathematics or create a story. Even one answer has different dimensions. “Hafez” can be a correct answer to a question about Persian poets while a condescending tone makes the response less useful.

Some prompts admit many valid responses. A bedtime story about a teddy bear cannot be evaluated by matching it to just one reference sentence. Agent systems add another challenge: a mistake can arise in any of several steps of observation, planning and action.

02

Human agreement needs a baseline

Raters need clear guidance, particularly on subjective criteria. Cohen’s kappa compares observed agreement with the agreement expected by chance.

Human raters can consider these nuances, but they need clear instructions. One person may find a teddy bear gift suggestion helpful while another does not; subjective criteria such as usefulness make it especially important to define what raters should look for.

A simple percentage of rater agreement can be misleading because people can agree by chance. If two raters independently choose pass or fail with equal probability, they will agree half the time even without a shared understanding of quality.

Cohen’s kappa adjusts observed agreement against the agreement expected from the raters’ respective pass and fail rates. The instructor says a value around 0.8 suggests strong agreement on a subjective criterion and can signal whether the guidelines need work.

Human ratings are informative but slow and expensive at scale. One alternative is to have people create reference answers once and then automatically compare model outputs against them.

03

Testing code and measuring multiple attempts

Executable tests and known answers enable rule-based checks. Pass@k estimates the chance that at least one of several sampled solutions succeeds.

For code, tests can check whether a generated solution behaves as expected. For mathematics, a known answer can be compared with the answer the model produces. Both provide more directly verifiable targets than open-ended prose.

Pass@k asks for the probability that at least one of k generated attempts succeeds. It is useful when the system can generate several solutions and has a way, such as tests, to identify a working one.

To estimate pass@k, generate n attempts and count c successes. Instead of directly calculating every successful combination, calculate the probability that all k sampled attempts fail, then subtract it from one.

The first failed attempt is selected from n−c failures among n attempts. For the next selection there is one fewer failure and one fewer total attempt, because the same sampled attempt is not selected twice.

The resulting estimator is 1 − C(n−c, k) / C(n, k), where C denotes a combination. At k=1 it reduces to the fraction of generated attempts that succeeded.

Temperature affects the benefit of multiple attempts. With a deterministic setup that returns the same answer each time, additional attempts do not add diversity. Raising temperature can explore different solutions, but too much diversity can make individual answers less reasonable.

Pass@k is most useful when a successful candidate can be recognized at inference time. Where the answer cannot readily be checked, another approach is consensus@k: sample several responses and choose the answer that appears most often.

04

Why word overlap cannot capture every good answer

BLEU, METEOR and related metrics compare text with references. Valid paraphrases show why overlap alone is an imperfect measure.

For natural language, older translation metrics compare an output with a human reference using overlapping sequences of words, or n-grams. BLEU emphasizes precision and applies a brevity penalty so an overly short response cannot score well merely by matching a few words.

METEOR combines precision and recall and includes a penalty related to the ordering and fragmentation of matches. ROUGE is another family of overlap measures. These metrics correlate to an extent with human judgments, but they still require references.

Overlap also misses valid paraphrases. A sentence about a plush teddy bear comforting a child and another about a soft stuffed bear helping a child feel safe can express the same idea with different words. Some metrics allow stemming or synonyms, but the problem extends beyond individual word matches.

05

Calibrate a model judge with human labels

A model can judge answers against a criterion and give a rationale. The instructors recommend measuring its errors against labeled examples and iterating on its prompt.

An LLM used as a judge can evaluate an output against a criterion such as relevance. The judge receives the original question, the answer and the standard to apply; it can return a pass or fail and explain its reasoning.

A model judge can account for linguistic variation and, if capable enough for the task, need not always have a reference answer. It can be adapted to translation or question answering by changing the evaluation guidelines, while its rationale helps diagnose decisions.

A practical judging prompt begins with human observation. Inspect and label a small sample of question-and-answer pairs to identify the patterns that matter, then use those examples to develop a clear evaluation instruction.

Establish a baseline on the labeled examples and choose a measurable target. For a binary judge, precision and recall can reveal different error types; for example, a team might target more than 90% recall of failures.

Inspect the cases the judge gets wrong, revise the prompt to address a recurring pattern, and evaluate again. An agent can help run this iterative loop when the objective is measurable, but the team still needs a target that captures what it cares about.

The judge should return a result in a format the application can parse, including a score and rationale. Simply asking for a format does not guarantee compliance; constrained decoding restricts allowed tokens so that a response follows a specified structure such as JSON.

06

Model judges can favor position, length and their own style

Swap answer order and explicitly account for verbosity and self-preference. Structured outputs and binary criteria make decisions easier to use and test.

Model judges can exhibit position bias: in a comparison of A and B, their choice may depend partly on which appears first. Swapping the order and comparing the two judgments can expose some of this sensitivity.

A judge may also prefer a longer response even when a shorter answer is more correct. Explicitly tell it that length is not evidence of quality and include examples that distinguish useful detail from verbosity.

Self-enhancement bias is a tendency to favor outputs from the same model used as judge. Other surface cues, such as a plausible-looking citation, can also influence a rating. The instructors recommend testing for biases rather than assuming a model’s verdict is neutral.

The instructor recommends binary pass-or-fail criteria over overly fine scales when people themselves struggle to distinguish adjacent scores. Asking for a rationale before the score, comparing against human labels and using repeated judgments or lower temperature can improve reliability.

Judging methods are still changing. The lecture mentions newer decision models designed to produce calibrated probabilities for choices rather than ordinary text, as a possible development for evaluation.

07

Diagnose an agent at each step

A missing tool, poor tool description, failed execution or noisy result can each produce a bad final answer. Inspect the trace to find which stage needs repair.

Shervine Amidi turns to agents, which repeatedly select tools, execute actions and use observations to continue or answer. Debugging requires examining the intermediate steps as well as the final result.

An agent asked to find a teddy bear may answer without calling any tool. One cause is that its tool router did not include the relevant function in the available context; the router’s selection or prompt can be adjusted.

If the right tool is available but ignored, its instructions may not make clear when to use it. The instructor says builders usually start by improving prompts and tool descriptions; a model’s underlying capability can also be a limitation.

An agent may invent a function that does not exist. A crowded context, an insufficiently capable model, or tool names that do not match the language users employ can contribute. A better model can help, but changing the tool name or its description may address the actual mismatch.

Selecting the wrong existing tool can likewise reflect poor routing or unclear distinctions in the available tool descriptions. If the model selects the right tool but supplies bad arguments, it may lack information such as the user’s location or the helper needed to retrieve it.

After a correct tool call, the tool itself may fail. The model might then tell the user that there is an internal error. Fix the implementation and return a useful, interpretable result instead of leaving the agent without information about what happened.

A tool that performs an action but returns no feedback can cause the agent to invent an outcome. If a light-switch tool returns nothing, for example, saying the light was turned on has no basis. The response should tell the model what the tool actually did.

A final answer can still go wrong after a tool returns data. Excessive output forces the model to search through irrelevant information; ambiguous field names make the result harder to interpret. Trim responses and use clear, self-describing structure.

These failure modes suggest a debugging sequence: inspect tool selection and arguments, the backend execution, then the information the final response received. A fix to a tool’s design or output can address a pattern of failures rather than one isolated answer.

08

Compare benchmarks with their actual tasks

Agent time horizons and aggregate indexes summarize different capabilities. Knowledge work, code and science demand different grading approaches.

A long-horizon metric asks how long a task would take a human expert and finds the task duration at which an agent succeeds at least half the time. The instructor uses this as a proxy for how agents handle longer sequences of actions and recover from mistakes.

Model comparison sites aggregate many evaluations, but an overall intelligence score depends on which tasks and weights are chosen. The lecture’s example index divides its selection equally among general knowledge, knowledge work, coding and science.

A factual-knowledge benchmark can ask questions from fields including business, health, law and software engineering without giving the model internet access. A judge compares its response with an expected answer, and incorrect answers may be penalized.

Knowledge work is harder to grade because it produces collections of artifacts rather than one neatly defined answer. A benchmark may provide messages, reports and emails, then ask the agent to produce files within a budget of turns.

One way to score those files is to compare measurable properties with a reference. Presentation quality is harder to define in a self-contained rule, so the instructor describes pairwise comparison: show a judge two submissions and ask which is better.

Coding benchmarks such as Terminal-Bench supply an environment, tools and time for an agent to solve a task. Hidden tests then check the resulting artifacts deterministically, reducing the need for a model judge when the task has a verifiable outcome.

Scientific agent tasks use a similar pattern in fields beyond ordinary coding. The lecture gives an astronomy example where a produced file can be compared with a reference by measuring whether the difference falls below a threshold.

09

Public scores can diverge from real-world performance

Training on public benchmarks and small changes to task instructions can undermine the meaning of rankings. Private tests and task-specific evaluations offer alternatives.

A high benchmark rank may not match a user’s experience. “Benchmaxxing” describes optimizing scores rather than the broader capabilities users want. Public benchmark tasks may also enter training data, making it hard to know whether a score reflects generalization.

The lecture describes a study that made small changes to task requirements, such as changing a requested date format or letter casing, and observed lower scores and changed rankings. Its authors suggested benchmark overfitting as a possible explanation; the instructor presents that as a hypothesis.

The instructor compares this with a former student who submitted answers to questions removed from a homework assignment. Similarly, models in the discussed study sometimes tried to satisfy constraints that were no longer requested.

One response is to keep evaluation tasks private and run them on providers’ behalf. The instructor says private evaluators can negotiate zero-data-retention arrangements. Companies also create internal tests representative of their own work and avoid sharing them when they want to preserve their usefulness.

Other organizations make the opposite choice, publishing tasks in an undermeasured field to help model developers improve there. The right choice depends on whether the purpose is a durable private measure or wider progress on a type of work.

Aggregated benchmarks change too. Their designers can revise component tests and weights when a ranking no longer seems to match users’ experience. The instructor calls this a two-way response between model providers optimizing scores and aggregators refining their measures.

10

Choose a quality–cost trade-off and protect the evaluation

Compare models on the quality and cost frontier, validate the benchmark and guard against shortcuts. Preference arenas add a useful but subjective signal.

Quality is not the only axis for choosing a model. On a cost-versus-quality plot, the Pareto frontier consists of options for which one cannot improve quality or lower cost without a trade-off. The same comparison can include latency.

Benchmark quality itself deserves scrutiny. Check that tasks are solvable, inputs are correctly specified and the expected answer makes sense; evaluation tools can inspect benchmark definitions and execution logs for problems.

Agents may take shortcuts that invalidate a test: a repository could contain a commit with the solution, or internet access could let the model find an answer. Safeguards depend on the environment and the capability being measured.

An evaluation environment can also have security weaknesses if agents are allowed to carry out actions beyond the intended task. The instructor uses a prior incident to illustrate the need to limit what an evaluation setup permits.

Goodhart’s law captures the risk: when a measure becomes a target, optimizing the proxy can stop improving the original goal. Lines of code might once have served as a rough productivity proxy, but making the count the target encourages more lines rather than better work.

User preference arenas offer a different signal by showing people two model answers and asking which they prefer. They are expensive and slow because rankings need many comparisons; preferences can also diverge from factual correctness when the user cannot judge the answer.

11

Build the evaluation before optimizing the agent

Separate essential requirements from secondary ones and iterate against the chosen measure. Use deterministic checks where possible and calibrate model judges when needed.

An evaluation-driven workflow starts by defining the desired outcome, then improves prompts and tools against that measure. The instructor recommends separating requirements that are essential from those that matter less, because a system cannot optimize every objective at once.

Where a deterministic check can establish correctness, prefer it to a model judge that must itself be calibrated. Model judges remain useful for less easily specified qualities, but their complexity should not replace a clear definition of the desired result.

같은 채널의 다른 글

1:00:02EN
Stanford Online· 2026년 11월 20일Philip Guo: The Small Choices That Kept Python Tutor AlivePhilip Guo · UC San DiegoA free code visualizer reached nearly 30 million people through instant access, a narrow mission and years of careful maintenance.EducationProgramming Languages제품과 UX+2
48:59EN
Stanford Online· 2026년 11월 19일Cynthia Bailey: Choose AI models for the task and guard your API keysCynthia Bailey · Stanford UniversityStanford’s Cynthia Bailey shows how token overhead, repeated agent steps and exposed credentials turn AI use into real costs.AI cost & performanceAI agentsAI Safety+1
49:17EN
Stanford Online· 2026년 11월 19일Stanford graduate study: start with a course before committing to a degreePax Heymeyer, Jonathan Nottman · Stanford Engineering Center for Global and Online EducationStanford’s online team explains course-by-course admission, credit, workload, exams and 2026–27 tuition.Education
슬라이드 6장1:13:23EN
Stanford Online· 2026년 11월 18일Terminal-Bench: How to Test Whether AI Agents Can Do Real WorkAlex Shaw, Ryan Marten · Laude InstituteThe benchmark's creators explain why useful agent tests need realistic tasks, reliable graders, and safeguards against shortcuts.평가(Evals)Benchmarks에이전트 엔지니어링+2
슬라이드 6장58:52EN
Stanford Online· 2026년 11월 17일Cynthia Bailey: Define What an AI Agent Can Do and When It Must StopCynthia Bailey · Stanford UniversityConnectors let models act through other apps; explicit permissions, human checks and stopping conditions determine how safely they do it.AI agentsMCP에이전트 엔지니어링+2
슬라이드 4장1:20:34EN
Stanford Online· 2026년 11월 17일When to Optimize an AI Agent’s Prompt or Its WeightsMichael Ryan · Stanford UniversityMichael Ryan compares prompt search, reinforcement learning and distillation, starting with the evaluation that makes any improvement measurable.Prompt optimization에이전트 엔지니어링Reinforcement Learning+2