Terminal-Bench: How to Test Whether AI Agents Can Do Real Work

The benchmark's creators explain why useful agent tests need realistic tasks, reliable graders, and safeguards against shortcuts.

ASAlex Shaw연사 · Laude InstituteRMRyan Marten연사 · Laude Institute
채널
Stanford Online
게시일
2026년 11월 18일
영상
1:13:23
읽기
10분
슬라이드
6
원문
영어
YouTube에서 열기
스크롤하면 영상이 오른쪽 아래로 작아져 계속 재생됩니다. 자막은 읽기 언어를 따릅니다.
읽기 언어2개 언어로 읽을 수 있음
이 글은 아직 한국어 번역이 없어 영어 원문으로 표시합니다.
요약

30초 요약

An agent's score is only as trustworthy as the test deciding what counts as success. Terminal-Bench creators Alex Shaw and Ryan Marten describe how to build realistic computer tasks, match the grader to the instruction, and prevent agents from earning credit through shortcuts. Their Harbor framework runs these tasks in isolated environments and records full trial traces. When a grading rule changes, old attempts can sometimes be regraded; changes to what the agent sees require fresh runs. The speakers argue that domain experts should help define success, especially for work that cannot be checked with simple code.

  1. 1Agent behavior shifts development toward evaluationShaw argues that model calls and generated code make behavior harder to deduce from source alone. Developers must define desired outcomes and compare observed results.
  2. 2A task combines an instruction, environment, and independent checksTerminal-Bench applies that structure across computer work, from a CAD reconstruction to programming and other fields. Harbor lets teams run tasks with different agents.
  3. 3The hardest part is deciding what counts as successMarten compares the verifier to a classifier with false positives and false negatives. Its checks must align with the instruction, and a model-based judge can need an evaluation of its own.
  4. 4A separate verifier container limits opportunities to cheatHarbor can collect an agent's output, destroy its container and grade the artifact in a fresh one. Task-specific loopholes and executable output still need attention.
  5. 5Some grader fixes need regrading rather than new agent runsHarbor tracks task versions. A changed verifier can sometimes score saved trials again, whereas a material change to the agent-facing prompt or environment requires rerunning them.
  6. 6Human review of a verifier can improve thousands of trialsReal usage may help generate tasks, but a single observed example rarely specifies every correct outcome. The speakers see expert judgment on the verifier as particularly valuable.

핵심 정리

  1. Make each instruction and its verifier cover the same requirements; test for both false passes and false failures.
  2. Inspect agent traces and publish them alongside scores to expose shortcuts and unexpected failure modes.
  3. Isolate the verifier from the agent's environment, while accounting for executable artifacts.
  4. Regrade saved attempts after verifier-only fixes when possible; rerun after material changes visible to the agent.
  5. Use domain experts to define success for realistic work and review automatically generated graders.
전문

챕터별로 다듬은 전문 읽기

원본 자막만 보기
01

From writing rules to measuring outcomes

Shaw compares traditional programming with model and agent development. He argues that testing behavior becomes central when systems are learned or generated.

Alex Shaw and Ryan Marten introduce Terminal-Bench, a collection of challenging tasks for agents operating through a computer terminal, and Harbor, their framework for building and running evaluations. Shaw starts with the changing way people make software.

Early programmers wrote numeric instructions for processors. Assembly language made those instructions more readable, and high-level languages such as Fortran moved programming further from the machine. Shaw notes that claims about eliminating coding and debugging accompanied earlier changes too.

Traditional software can in principle be understood from its instructions and execution environment. Shaw calls this deductive development: people set rules, execute them on inputs, and reason about the result.

Machine learning moves the human role toward choosing data, labels, and learning algorithms. Instead of specifying all the rules, developers give a learning system examples and evaluate the rules it infers against further examples.

In Shaw's account, conventional software development proceeds from specification to implementation, testing and deployment. Model development starts with an objective and an evaluation, then repeatedly changes the system and measures the result.

Language models scale this approach by learning to predict text from large datasets. Agents add programs around those models that translate selected outputs into tool calls or executable code, allowing a model to change its environment.

Shaw expects this empirical development pattern to spread across models, agents and even agent-generated software. A model call makes an agent's behavior hard to infer from code alone, while large quantities of generated code may be impractical to inspect line by line.

He points to François Chollet's argument that generated code can be treated as a black-box artifact whose behavior must be managed by testing. On that view, the cycle becomes an objective, an evaluation, experiments and optimization.

A phone-number extractor shows the change. A conventional pattern-based function produces a predictable result for a fixed input, while a model call may recognize more variations but does not guarantee the same output on every run.

Shaw argues that the scarce human contribution may shift from specifying every rule toward evaluating what an agent or generated program actually does. What job titles and development systems emerge around that work remain open questions.

It is easy to imagine training and test data for predicting house prices, Shaw says, but less obvious what they look like for automating a dental office's front desk. Harbor and similar tools try to make that second kind of evaluation possible.

02

Why Terminal-Bench uses the command line

Marten introduces the benchmark and its origins in a broader view of computer work. The task format combines an instruction, an environment and a verifier.

Marten explains the terminal's appeal: language models take text in and produce text, while a terminal can use textual commands to run many programs. A cloud setup that takes numerous clicks in a web interface can sometimes be expressed as one command.

Terminal-Bench measures a broad range of computer work rather than only software engineering. The team sourced tasks from an open community, and Marten says the benchmark is used both for model releases and for intermediate training checkpoints and agent development.

Shaw traces the idea to the K Prize, a competition based on continuously refreshed SWE-bench tasks. Working with GitHub issues, repositories and pull requests led the team to ask why an agent benchmark could not cover a wider range of computer tasks.

Anatomy of a Terminal-Bench task
슬라이드 1 / 6Anatomy of a Terminal-Bench taskThe example combines an instruction, an environment definition and tests for a CAD artifact.

A task combines a written instruction, an environment definition and tests that determine whether the agent succeeded. One example asks the agent to reconstruct a three-dimensional object from a schematic and checks properties of its output.

The benchmark includes tasks from programming and networking to biology, chemistry, music and graphic design. Terminal-Bench supplies the tasks; Harbor provides a general format and infrastructure for running evaluations like them.

03

The components of an agent evaluation

Harbor separates tasks from agent and infrastructure choices. The speakers define trials, trajectories and verifiers.

Marten describes evaluation-driven development as setting a measurable objective and using observations to improve an agent. An agent evaluation must account for the environment and tools as well as the model's text, making grading more complicated than matching a string.

A task is one test with inputs and success criteria; a trial is one attempt. A verifier grades that attempt, a trajectory records its steps, and the evaluation harness supplies the infrastructure that runs the test.

Harbor packages tasks independently from a particular agent. The package contains the instruction and configuration, an environment, optional reference solution and verifier, so the same task can be run with different agents and infrastructure.

04

Write realistic instructions and environments

Good tasks specify an outcome without giving away the solution. Environments balance realistic context against reliability, security and contamination.

Marten says task authors face three recurring challenges: aligning what the instruction asks with what the verifier rewards, preventing agents from gaining reward by unintended means, and making the task realistic and worth measuring.

For Terminal-Bench, instructions should specify the required outcome without ambiguity while staying concise and natural. Additional details can accidentally conflict with the grader or tell an agent things it should be able to discover itself.

Shortening a task instruction
슬라이드 2 / 6Shortening a task instructionReviewers remove repetition and details the agent can discover while retaining the required outcome.

In a computer-aided design example, reviewers shortened a long instruction, removing repetitions and details such as software versions that the agent could find in its environment. The aim is enough information to complete the job, not a script for solving it.

The environment supplies files, software, running processes and other context that the agent would have in the work being simulated. Using actual tools where possible makes the test closer to the setting a developer wants to measure.

Internet access can make an environment realistic, but web content changes between runs, may contaminate a test, and can let an agent take unwanted real-world actions. Baking dependencies into the environment can reduce failures unrelated to the capability being measured.

Adding distractors to a poster task
슬라이드 3 / 6Adding distractors to a poster taskExtra plausible assets make the exercise closer to working with a real asset library.

One graphic-design task asks an agent to recreate an editable poster from a rendered image and source assets. The revised environment adds plausible unused assets, making the agent choose among distractions as it would in a company's asset library.

Asked how Harbor isolates simultaneous attempts, Marten says each trial should have independent state and resources. Running each in a fresh container or virtual machine also makes it practical to parallelize hundreds or thousands of trials.

05

Design and defend the verifier

Graders must recognize valid solutions and reject shortcuts. Separate verifier environments and reference solutions help, but task-specific flaws remain possible.

The verifier is often the hardest and most valuable part of a task: it must decide whether an outcome really satisfies the instruction. Marten compares it to a classifier that can make both false-positive and false-negative judgments.

An instruction and its tests should cover the same requirements. Otherwise an incomplete answer may pass or an answer that follows the directions may fail. Programmatic checks are easier to reason about, while subjective tasks may require a model-based judge.

A model judge introduces variability of its own, Marten says. Its judgments can themselves be tested on a separate benchmark and improved; using a judge does not remove the need to decide what good answers look like.

Reward hacking occurs when an agent receives credit without producing the intended result. Some defenses belong to the evaluation infrastructure, but task authors also need to anticipate shortcuts specific to their own checks.

Isolating the verifier
슬라이드 4 / 6Isolating the verifierThe agent's artifact moves to a separate environment for testing after the agent environment is torn down.

In Harbor's separate-verifier mode, the agent works in one container. Its artifact is collected, that container is torn down, and a fresh verifier container runs the tests, preventing an agent from leaving a process behind to interfere with grading.

Isolation is not the whole solution: an artifact can itself contain executable code, so the verifier must handle that carefully. Marten also points to tasks involving Lean proofs in which agents exploited bugs in the proof-checking kernel.

A reference or oracle solution can test that both an environment and its verifier work. Rather than hard-code a passing answer, a useful solution interacts with the environment in a way similar to a real agent.

For a music-harmonization task, the reference solution is a backtracking program that assembles notes and revises choices when they violate composition rules. The output can be checked programmatically, and the solver exercises the task environment.

06

Run and maintain benchmarks like software

Reliable, portable execution helps results stay comparable. Quality checks and task versioning support repairs without always repeating expensive runs.

Evaluation infrastructure should be reliable, reproducible, fast and portable. Agents already vary from run to run, so infrastructure failures should not add unnecessary noise to the comparison.

Independent trials can run in parallel, but the speedup is constrained by resources such as model token limits, available sandboxes and container registries. A result is useful only if the runs being compared are performed in comparable conditions.

Marten says agent training data and test data can share the same task format, analogous to a machine-learning train/test split. The infrastructure for running evaluations therefore overlaps with that used to generate training rollouts and sometimes with production systems.

Harbor separates tasks from models, agents and execution providers so a team can swap each component. The same engine can support evaluation, optimization and quality control during task construction.

Benchmark tasks are software and develop bugs, so they need quality assurance and ongoing maintenance. The team turns recurring mistakes into automated checks and uses more capable judges for questions such as whether the instructions match the tests.

Quality checks for tasks
슬라이드 5 / 6Quality checks for tasksStatic checks, agent judges and trial runs provide different ways to find task failures.

The team also runs reference solutions, frontier agents and agents trying to break tasks, then inspects their trajectories. This can expose a broken package or reveal that a task is difficult for a reason its author did not intend.

An oracle agent in this setting simply executes the reference solution; it need not be model-based. Giving an agent a hint is another possible experiment: the smallest hint that lets it succeed may identify the obstacle that makes a task hard.

Testing does not stop at release. As models improve, they discover new shortcuts and correct solutions a verifier may reject. Both reward hacks and false negatives provide evidence for fixing the task.

Harbor versions tasks according to what changed. If only the verifier changes, existing trial artifacts can sometimes be regraded without paying to run the agent again; a material change to the agent's instructions, environment, data or tools calls for a new trial.

An audience member asks how a changing benchmark can compare model releases fairly. Marten says the team avoids updating constantly, refreshes the leaderboard and retains historical results so users can compare models across releases.

He wants model reports to link to full results rather than only a static number. The Terminal-Bench site exposes individual trials, their trajectories, timing, costs and logs, allowing others to investigate why a score occurred.

Inspecting full benchmark results
슬라이드 6 / 6Inspecting full benchmark resultsPer-trial trajectories, logs, artifacts and rewards explain more than a leaderboard score alone.
07

From simulated users to domain-specific tests

The audience asks about interactive tasks, realism and generalization. The speakers describe simulated users, multi-service environments and the importance of domain knowledge.

Offline automated tests are one form of evaluation. Teams can also inspect traces from agents used in production or have people evaluate them manually, although those approaches come with different costs and scaling limits.

A student asks how Harbor can test an agent that needs to ask a person for information rather than receiving a complete instruction up front. The speakers say a simulated user can be added to the environment, including as a separate agent.

That simulation introduces another model and another source of randomness that may itself need evaluation. The speakers describe multi-container environments as one way to build these more involved interactions, while acknowledging that no general solution removes the difficulty.

Asked how Terminal-Bench relates to SWE-bench, a speaker says a SWE-bench-style task can be represented in the more general Terminal-Bench format. The actual datasets differ: SWE-bench concentrates on software repositories, while Terminal-Bench spans more kinds of computer work.

A realistic environment might reproduce a cluttered personal file system or the collection of services inside a company. Harbor can describe arbitrary files and, with a multi-container setup, a network of services; the relevant realism depends on the work being tested.

On whether a task really represents a general capability, the speakers distinguish infrastructure from expertise in the work itself. They expect domain experts to use Harbor to construct focused evaluations for the particular problems they understand and want automated.

The speakers expect the benchmark's current formulation to look simple in retrospect. Near-term directions include covering more computer activities, making environments more complex and testing tasks that require longer sequences of action.

08

Use real observations without neglecting the grader

Observed work can inspire tasks, but determining acceptable outcomes remains hard. The speakers advocate expert review and sharing complete evaluation traces.

Asked how the work began, a speaker describes pursuing unusually high implementation quality on dimensions that matter to a community. Experience building the K Prize and collaboration with the team helped shape the project.

A student suggests observing a front-desk worker and automatically turning the recording into evaluations. The speakers agree that real usage can supply instructions and perhaps snapshots of an environment, but say the correct verifier remains difficult to infer from one observed example.

A good verifier must distinguish many acceptable and unacceptable outcomes, not merely reproduce one demonstration. The speakers see human review of this decision boundary as high leverage because one improved verifier can assess thousands of trials.

Finally, the speakers argue for publishing complete traces. Community members can use them to spot reward hacks or compare properties of agent-generated artifacts, questions that a single aggregate score cannot answer.

같은 채널의 다른 글

48:59EN
Stanford Online· 2026년 11월 19일Cynthia Bailey: Choose AI models for the task and guard your API keysCynthia Bailey · Stanford UniversityStanford’s Cynthia Bailey shows how token overhead, repeated agent steps and exposed credentials turn AI use into real costs.AI cost & performanceAI agentsAI Safety+1
슬라이드 6장58:52EN
Stanford Online· 2026년 11월 17일Cynthia Bailey: Define What an AI Agent Can Do and When It Must StopCynthia Bailey · Stanford UniversityConnectors let models act through other apps; explicit permissions, human checks and stopping conditions determine how safely they do it.AI agentsMCP에이전트 엔지니어링+2
슬라이드 4장1:20:34EN
Stanford Online· 2026년 11월 17일When to Optimize an AI Agent’s Prompt or Its WeightsMichael Ryan · Stanford UniversityMichael Ryan compares prompt search, reinforcement learning and distillation, starting with the evaluation that makes any improvement measurable.Prompt optimization에이전트 엔지니어링Reinforcement Learning+2
1:09:27EN
Stanford Online· 2026년 11월 17일Cynthia Bailey: Keep Testing AI After You ShipCynthia Bailey · Stanford UniversityOld test data can miss changed user behavior. Bailey shows how independent review, post-launch checks and better data expose AI risks.코딩 에이전트AI Safety안전과 거버넌스+1
슬라이드 11장1:18:55EN
Stanford Online· 2026년 11월 16일Michael Ryan: More AI agents can make a task harderMichael Ryan · Stanford UniversityParallel agents can explore more, but coordination costs and conflicting work can make a single agent more effective.Multi-agent systemsAgent orchestrationAgent memory+2
슬라이드 8장1:18:54EN
Stanford Online· 2026년 11월 15일AI Agent Memory: What to Store, Retrieve and ForgetMichael Ryan · Stanford UniversityMichael Ryan compares agent memory architectures and shows why remembering more does not always improve decisions.Agent memoryAI agents검색 증강 생성(RAG)+2