30초 요약
An agent's score is only as trustworthy as the test deciding what counts as success. Terminal-Bench creators Alex Shaw and Ryan Marten describe how to build realistic computer tasks, match the grader to the instruction, and prevent agents from earning credit through shortcuts. Their Harbor framework runs these tasks in isolated environments and records full trial traces. When a grading rule changes, old attempts can sometimes be regraded; changes to what the agent sees require fresh runs. The speakers argue that domain experts should help define success, especially for work that cannot be checked with simple code.
- 1Agent behavior shifts development toward evaluationShaw argues that model calls and generated code make behavior harder to deduce from source alone. Developers must define desired outcomes and compare observed results.
- 2A task combines an instruction, environment, and independent checksTerminal-Bench applies that structure across computer work, from a CAD reconstruction to programming and other fields. Harbor lets teams run tasks with different agents.
- 3The hardest part is deciding what counts as successMarten compares the verifier to a classifier with false positives and false negatives. Its checks must align with the instruction, and a model-based judge can need an evaluation of its own.
- 4A separate verifier container limits opportunities to cheatHarbor can collect an agent's output, destroy its container and grade the artifact in a fresh one. Task-specific loopholes and executable output still need attention.
- 5Some grader fixes need regrading rather than new agent runsHarbor tracks task versions. A changed verifier can sometimes score saved trials again, whereas a material change to the agent-facing prompt or environment requires rerunning them.
- 6Human review of a verifier can improve thousands of trialsReal usage may help generate tasks, but a single observed example rarely specifies every correct outcome. The speakers see expert judgment on the verifier as particularly valuable.
핵심 정리
- Make each instruction and its verifier cover the same requirements; test for both false passes and false failures.
- Inspect agent traces and publish them alongside scores to expose shortcuts and unexpected failure modes.
- Isolate the verifier from the agent's environment, while accounting for executable artifacts.
- Regrade saved attempts after verifier-only fixes when possible; rerun after material changes visible to the agent.
- Use domain experts to define success for realistic work and review automatically generated graders.
챕터별로 다듬은 전문 읽기
From writing rules to measuring outcomes
Shaw compares traditional programming with model and agent development. He argues that testing behavior becomes central when systems are learned or generated.
Alex Shaw and Ryan Marten introduce Terminal-Bench, a collection of challenging tasks for agents operating through a computer terminal, and Harbor, their framework for building and running evaluations. Shaw starts with the changing way people make software.
Early programmers wrote numeric instructions for processors. Assembly language made those instructions more readable, and high-level languages such as Fortran moved programming further from the machine. Shaw notes that claims about eliminating coding and debugging accompanied earlier changes too.
Traditional software can in principle be understood from its instructions and execution environment. Shaw calls this deductive development: people set rules, execute them on inputs, and reason about the result.
Machine learning moves the human role toward choosing data, labels, and learning algorithms. Instead of specifying all the rules, developers give a learning system examples and evaluate the rules it infers against further examples.
In Shaw's account, conventional software development proceeds from specification to implementation, testing and deployment. Model development starts with an objective and an evaluation, then repeatedly changes the system and measures the result.
Language models scale this approach by learning to predict text from large datasets. Agents add programs around those models that translate selected outputs into tool calls or executable code, allowing a model to change its environment.
Shaw expects this empirical development pattern to spread across models, agents and even agent-generated software. A model call makes an agent's behavior hard to infer from code alone, while large quantities of generated code may be impractical to inspect line by line.
He points to François Chollet's argument that generated code can be treated as a black-box artifact whose behavior must be managed by testing. On that view, the cycle becomes an objective, an evaluation, experiments and optimization.
A phone-number extractor shows the change. A conventional pattern-based function produces a predictable result for a fixed input, while a model call may recognize more variations but does not guarantee the same output on every run.
Shaw argues that the scarce human contribution may shift from specifying every rule toward evaluating what an agent or generated program actually does. What job titles and development systems emerge around that work remain open questions.
It is easy to imagine training and test data for predicting house prices, Shaw says, but less obvious what they look like for automating a dental office's front desk. Harbor and similar tools try to make that second kind of evaluation possible.
Why Terminal-Bench uses the command line
Marten introduces the benchmark and its origins in a broader view of computer work. The task format combines an instruction, an environment and a verifier.
Marten explains the terminal's appeal: language models take text in and produce text, while a terminal can use textual commands to run many programs. A cloud setup that takes numerous clicks in a web interface can sometimes be expressed as one command.
Terminal-Bench measures a broad range of computer work rather than only software engineering. The team sourced tasks from an open community, and Marten says the benchmark is used both for model releases and for intermediate training checkpoints and agent development.
Shaw traces the idea to the K Prize, a competition based on continuously refreshed SWE-bench tasks. Working with GitHub issues, repositories and pull requests led the team to ask why an agent benchmark could not cover a wider range of computer tasks.

A task combines a written instruction, an environment definition and tests that determine whether the agent succeeded. One example asks the agent to reconstruct a three-dimensional object from a schematic and checks properties of its output.
The benchmark includes tasks from programming and networking to biology, chemistry, music and graphic design. Terminal-Bench supplies the tasks; Harbor provides a general format and infrastructure for running evaluations like them.
The components of an agent evaluation
Harbor separates tasks from agent and infrastructure choices. The speakers define trials, trajectories and verifiers.
Marten describes evaluation-driven development as setting a measurable objective and using observations to improve an agent. An agent evaluation must account for the environment and tools as well as the model's text, making grading more complicated than matching a string.
A task is one test with inputs and success criteria; a trial is one attempt. A verifier grades that attempt, a trajectory records its steps, and the evaluation harness supplies the infrastructure that runs the test.
Harbor packages tasks independently from a particular agent. The package contains the instruction and configuration, an environment, optional reference solution and verifier, so the same task can be run with different agents and infrastructure.
Write realistic instructions and environments
Good tasks specify an outcome without giving away the solution. Environments balance realistic context against reliability, security and contamination.
Marten says task authors face three recurring challenges: aligning what the instruction asks with what the verifier rewards, preventing agents from gaining reward by unintended means, and making the task realistic and worth measuring.
For Terminal-Bench, instructions should specify the required outcome without ambiguity while staying concise and natural. Additional details can accidentally conflict with the grader or tell an agent things it should be able to discover itself.

In a computer-aided design example, reviewers shortened a long instruction, removing repetitions and details such as software versions that the agent could find in its environment. The aim is enough information to complete the job, not a script for solving it.
The environment supplies files, software, running processes and other context that the agent would have in the work being simulated. Using actual tools where possible makes the test closer to the setting a developer wants to measure.
Internet access can make an environment realistic, but web content changes between runs, may contaminate a test, and can let an agent take unwanted real-world actions. Baking dependencies into the environment can reduce failures unrelated to the capability being measured.

One graphic-design task asks an agent to recreate an editable poster from a rendered image and source assets. The revised environment adds plausible unused assets, making the agent choose among distractions as it would in a company's asset library.
Asked how Harbor isolates simultaneous attempts, Marten says each trial should have independent state and resources. Running each in a fresh container or virtual machine also makes it practical to parallelize hundreds or thousands of trials.
Design and defend the verifier
Graders must recognize valid solutions and reject shortcuts. Separate verifier environments and reference solutions help, but task-specific flaws remain possible.
The verifier is often the hardest and most valuable part of a task: it must decide whether an outcome really satisfies the instruction. Marten compares it to a classifier that can make both false-positive and false-negative judgments.
An instruction and its tests should cover the same requirements. Otherwise an incomplete answer may pass or an answer that follows the directions may fail. Programmatic checks are easier to reason about, while subjective tasks may require a model-based judge.
A model judge introduces variability of its own, Marten says. Its judgments can themselves be tested on a separate benchmark and improved; using a judge does not remove the need to decide what good answers look like.
Reward hacking occurs when an agent receives credit without producing the intended result. Some defenses belong to the evaluation infrastructure, but task authors also need to anticipate shortcuts specific to their own checks.

In Harbor's separate-verifier mode, the agent works in one container. Its artifact is collected, that container is torn down, and a fresh verifier container runs the tests, preventing an agent from leaving a process behind to interfere with grading.
Isolation is not the whole solution: an artifact can itself contain executable code, so the verifier must handle that carefully. Marten also points to tasks involving Lean proofs in which agents exploited bugs in the proof-checking kernel.
A reference or oracle solution can test that both an environment and its verifier work. Rather than hard-code a passing answer, a useful solution interacts with the environment in a way similar to a real agent.
For a music-harmonization task, the reference solution is a backtracking program that assembles notes and revises choices when they violate composition rules. The output can be checked programmatically, and the solver exercises the task environment.
Run and maintain benchmarks like software
Reliable, portable execution helps results stay comparable. Quality checks and task versioning support repairs without always repeating expensive runs.
Evaluation infrastructure should be reliable, reproducible, fast and portable. Agents already vary from run to run, so infrastructure failures should not add unnecessary noise to the comparison.
Independent trials can run in parallel, but the speedup is constrained by resources such as model token limits, available sandboxes and container registries. A result is useful only if the runs being compared are performed in comparable conditions.
Marten says agent training data and test data can share the same task format, analogous to a machine-learning train/test split. The infrastructure for running evaluations therefore overlaps with that used to generate training rollouts and sometimes with production systems.
Harbor separates tasks from models, agents and execution providers so a team can swap each component. The same engine can support evaluation, optimization and quality control during task construction.
Benchmark tasks are software and develop bugs, so they need quality assurance and ongoing maintenance. The team turns recurring mistakes into automated checks and uses more capable judges for questions such as whether the instructions match the tests.

The team also runs reference solutions, frontier agents and agents trying to break tasks, then inspects their trajectories. This can expose a broken package or reveal that a task is difficult for a reason its author did not intend.
An oracle agent in this setting simply executes the reference solution; it need not be model-based. Giving an agent a hint is another possible experiment: the smallest hint that lets it succeed may identify the obstacle that makes a task hard.
Testing does not stop at release. As models improve, they discover new shortcuts and correct solutions a verifier may reject. Both reward hacks and false negatives provide evidence for fixing the task.
Harbor versions tasks according to what changed. If only the verifier changes, existing trial artifacts can sometimes be regraded without paying to run the agent again; a material change to the agent's instructions, environment, data or tools calls for a new trial.
An audience member asks how a changing benchmark can compare model releases fairly. Marten says the team avoids updating constantly, refreshes the leaderboard and retains historical results so users can compare models across releases.
He wants model reports to link to full results rather than only a static number. The Terminal-Bench site exposes individual trials, their trajectories, timing, costs and logs, allowing others to investigate why a score occurred.

From simulated users to domain-specific tests
The audience asks about interactive tasks, realism and generalization. The speakers describe simulated users, multi-service environments and the importance of domain knowledge.
Offline automated tests are one form of evaluation. Teams can also inspect traces from agents used in production or have people evaluate them manually, although those approaches come with different costs and scaling limits.
A student asks how Harbor can test an agent that needs to ask a person for information rather than receiving a complete instruction up front. The speakers say a simulated user can be added to the environment, including as a separate agent.
That simulation introduces another model and another source of randomness that may itself need evaluation. The speakers describe multi-container environments as one way to build these more involved interactions, while acknowledging that no general solution removes the difficulty.
Asked how Terminal-Bench relates to SWE-bench, a speaker says a SWE-bench-style task can be represented in the more general Terminal-Bench format. The actual datasets differ: SWE-bench concentrates on software repositories, while Terminal-Bench spans more kinds of computer work.
A realistic environment might reproduce a cluttered personal file system or the collection of services inside a company. Harbor can describe arbitrary files and, with a multi-container setup, a network of services; the relevant realism depends on the work being tested.
On whether a task really represents a general capability, the speakers distinguish infrastructure from expertise in the work itself. They expect domain experts to use Harbor to construct focused evaluations for the particular problems they understand and want automated.
The speakers expect the benchmark's current formulation to look simple in retrospect. Near-term directions include covering more computer activities, making environments more complex and testing tasks that require longer sequences of action.
Use real observations without neglecting the grader
Observed work can inspire tasks, but determining acceptable outcomes remains hard. The speakers advocate expert review and sharing complete evaluation traces.
Asked how the work began, a speaker describes pursuing unusually high implementation quality on dimensions that matter to a community. Experience building the K Prize and collaboration with the team helped shape the project.
A student suggests observing a front-desk worker and automatically turning the recording into evaluations. The speakers agree that real usage can supply instructions and perhaps snapshots of an environment, but say the correct verifier remains difficult to infer from one observed example.
A good verifier must distinguish many acceptable and unacceptable outcomes, not merely reproduce one demonstration. The speakers see human review of this decision boundary as high leverage because one improved verifier can assess thousands of trials.
Finally, the speakers argue for publishing complete traces. Community members can use them to spot reward hacks or compare properties of agent-generated artifacts, questions that a single aggregate score cannot answer.
같은 채널의 다른 글
48:59EN
58:52EN
1:20:34EN
1:09:27EN
1:18:55EN
1:18:54EN