Nathan Lambert on How Language Model Evaluation Has Evolved

Evaluation has changed with language models: from eliciting answers from base models before ChatGPT to running complex agents in expensive environments. The way we measure progress must change when model behavior changes. Evaluation lets researchers understand what remains difficult and communicate progress that users can trust.

Frontier evaluation is increasingly nuanced. Labs can select favorable tests, and benchmark scores may miss robustness, trust and behavior inside products. Even measurements such as task-completion time horizons need new data as models improve. Creating reliable targets can be harder than optimizing models against them.

Benchmarks tend to mirror the training goals of their era. Early tests measured knowledge and chat style; later ones emphasized math, code, factuality and safety. Reasoning models brought harder problems and tool use. Current agent evaluations increasingly attempt end-to-end knowledge work, which is much harder to score.

Before instruction tuning, a base model behaved primarily like autocomplete. Few-shot prompting supplied examples that demonstrated the expected answer format. A grader could compare the probability of candidate tokens or sample a completion and compare its answer. Sampling more closely resembles use but introduces randomness and additional evaluation choices.

Pass@k estimates the probability that at least one of k samples solves a problem. Generating exactly k outputs and checking whether any succeeds can have high variance. The original Codex evaluation approach samples more than k completions and uses an estimator based on the number that pass, then averages across problems.

The early evaluation pipeline looked simple: a prompt, a model, a generated solution and a grader. But prompt formatting could substantially change the result. Researchers spent time crafting benchmark-specific instructions, including how to extract the answer from generated text.

Chain-of-thought prompting added worked intermediate steps to examples before ChatGPT appeared. Later, shorter instructions such as asking a model to think step by step could elicit reasoning. Instruction tuning made models better able to answer a new query without examples, enabling zero-shot evaluation and broader applications.

As responses became more open-ended, language-model judges gained popularity; MT-Bench and AlpacaEval are examples of chat-style evaluation. Multiple-choice tests did not disappear. Tülu 3's 2024 suite mixed few-shot knowledge tasks with zero-shot reasoning prompts for math and code.

Reasoning models such as o1 accelerated the shift toward giving models time to think. For earlier instruction models, a prompt tailored to a benchmark could affect the score substantially: Lambert recalls differences of about eight points on MMLU between formats. Sampling temperature and other settings also became important to reproducing a model card's result.

Answer formatting can cause a dramatic apparent drop in scores. An evaluator may expect an answer after a particular phrase while training examples use a boxed format. Language-model judges can sometimes extract answers more flexibly, but the practical details require substantial engineering attention.

Harder benchmarks such as GPQA, Humanity's Last Exam, FrontierMath and coding evaluations require expert-created problems. They are labor-intensive and can contain errors in labels or questions. Lambert mentions a possible example he has not independently verified, and cautions that imperfect labels complicate interpretations of saturation.

New evaluations increasingly resemble real professional work in fields such as software, finance and law. Models are also tested inside products such as coding assistants. As older benchmarks become saturated, the field must keep creating challenging tasks, even though open-ended work is harder to measure.

Agentic evaluation is a system rather than a model alone. A harness manages prompts, context and tools, while a sandbox supplies files, compute and an environment in which the agent acts. The inference engine or API, hardware, timeouts and grader can all affect the result.

A model can be trained for its provider's own harness, while open models may need to work across many. Lambert cites a reported ARC-AGI-3 example in which a model's score changed dramatically with a custom harness. Specialized harnesses can create value, but they make fair comparisons and common standards harder.

Implementation mistakes matter. A vLLM postmortem on Kimi K2 found tool-call formatting bugs that prevented proper tool use. Even different providers serving the same model can yield different behavior. Lambert values public postmortems because the failures are otherwise difficult to see.

Hardware and inference configuration can also introduce variance: GPU generations, kernels, batch sizes and parallelism may affect results. Long-running tasks create another issue—timeouts can cut off slow trajectories that might eventually succeed. Reproducing a score is therefore an infrastructure problem as well as a modeling problem.

For OLMo 3, the team ran 14 models with three random seeds and measured variation across evaluations. In their study GPQA varied more than defined knowledge and math tests. Agent evaluations can be even noisier because they have fewer tasks and cost more to run.

The team used average@k, increasing the number of trials and averaging the results to reduce noise at greater compute cost. Lambert recalls needing to standardize tensor parallelism and batch size because changing settings shifted scores. A one-point difference alone is weak evidence that one model is better.

Comparing reports from different labs is difficult because outsiders do not know which evaluation sets were used during development or held out for validation. Labs may optimize against many tests, intentionally or otherwise. Comparisons within one lab's model family can be more informative, especially when supported by user experience and large performance changes.

For a practical application, look at the tradeoff between capability and price. Cost depends on inference, tokens consumed and model size; the most capable model is not always the right choice. Providers compete to move the capability-cost frontier, and serving at scale makes price integral to evaluation.

Frontier labs may maintain dozens or hundreds of internal evaluations alongside a smaller set of prominent public tests. A central purpose is statistical power: after spending compute on two training runs, the team needs enough signal to decide which weights are actually better. Noise weakens that feedback loop.

Contamination occurs when evaluation test material enters training, intentionally or accidentally. Decontamination uses approaches such as exact, substring or fuzzy n-gram matching to find and remove overlaps. Public datasets are especially exposed as benchmark questions spread across the web and can be collected again.

Lambert describes research auditing suspicious behavior when training data is too close to a math test. Responses include perturbing numbers or wording and maintaining private held-out evaluations. These are useful defenses, but contamination remains a continuing problem rather than a one-time cleanup.

Agents can also find shortcuts during an evaluation. A repository's history may contain the solution, or an agent may search an external site for an answer. Evaluators must isolate sandboxes, restrict tools and inspect logs to distinguish genuine task performance from unintended access. This monitoring adds cost.

Lambert points to established tooling such as EleutherAI's language-model evaluation harness and newer work such as Prime Intellect's Verifiers. The latter connects training and evaluation environments so models can learn to act in the kinds of worlds where they will be tested.

The takeaways are that benchmarks track the goals of their era and a score belongs to an entire system. Researchers should vary individual components to understand their effects. Some noise and incentives to optimize public benchmarks will remain, while inference, sandboxes, monitoring and grading make frontier evaluations increasingly expensive.