Xander Song on Evaluating Agents with Harbor and Phoenix

I'm Xander Song, an engineer at Arize AI working on Phoenix, our open-source platform for observability, experimentation, and evaluation. Today I want to introduce Harbor, an open-source framework for evaluating agents on real tasks and real data in isolated environments. The theme is shipping on evidence rather than vibes.

My aim is to explain how Harbor works and how our team uses it to iterate more confidently on agents, harnesses, skills, prompts, and tools. This is a discussion rather than a hands-on workshop. One disclaimer: I am not a Harbor maintainer, though I use it extensively.

At a high level, a Harbor task is a natural-language instruction describing something you want an agent to accomplish. Harbor gives that instruction to the agent in a fresh sandbox seeded with relevant data. You define task-specific verifiers that check the side effects left in the environment, producing rewards such as a zero-or-one pass or fail.

If the task and data are realistic and the verifier captures success accurately, you can hold those fixed while changing the agent. Running the same pipeline tells you whether a change improved or regressed performance on that task. A collection of tasks covering the agent's capabilities gives you a stronger basis for iteration than intuition alone.

Each task contains instructions, typically in a Markdown file, and an environment, commonly described by a Dockerfile. The environment can hold whatever data the task needs: for example, a SQLite database or a Git repository at a particular commit. Harbor is not limited to one kind of data.

A task also includes verifier scripts in a test directory. After the agent runs, the scripts examine the environment and check whether the intended changes occurred. Several tasks can be collected into a dataset, and each task can have a configurable number of independent trials.

Trials matter because language models and agents are stochastic: they do not behave identically on every run. Each trial starts a fresh container with the same seed data, places the chosen agent inside it, and gives it the task instructions. The agent might change source files or a database; verifiers inspect what it left behind and write rewards.

You can optionally configure an agent to emit a trajectory in ATIF, the Agent Trajectory Interchange Format. This JSON-based tracing format records the steps the agent took and helps with debugging and inspection. A full run across a dataset, its trials, verifiers, rewards, and trajectories is called a Harbor job.

Harbor also supports an oracle: a script that solves the task. Running that known solution lets you work out problems in the data, environment, and verifiers before spending tokens on an agent. It is a way to establish that the evaluation pipeline itself works.

Consider a coding agent whose task is to fix a failing database migration. The instruction might simply say that the migration fails and ask the agent to repair it. The environment could contain a database, schema definitions, and source code with an up migration deliberately set up to fail.

The verifier could run the migration, check for a zero exit status, and inspect the expected tables, columns, and indexes. The optional oracle is a script that applies the actual fix. If the oracle passes the verifier, I gain some confidence that the task and checks are configured properly before I test an agent.

After the agent edits the broken migration, the verifier checks the resulting filesystem and database. In Harbor's default local Docker setup, the verifier is typically mounted only after the agent finishes, so the agent cannot inspect the tests or solution while carrying out the task.

The canonical verifier is agnostic about how the agent is implemented. It treats the agent as a black box and grades the result and side effects rather than the exact path taken. You can inspect a trajectory when you need to, but that is a different kind of check.

This resembles a well-written integration test for an API: the test expresses the behavior expected at the interface without coupling itself to the implementation behind it. You can refactor an agent, change tools, or alter a harness while holding the task constant and quickly find out whether the outcome still works.

An attendee asks about the kinds of systems that fit in the environment. It is a Docker container, so the agent can be built in any framework or language, and the environment can contain whatever executable code is needed. Our team has used this flexibility while changing tools, prompts, and skills.

For example, we are consolidating skills shared across several agent experiences: an MCP-based agent, a CLI-based agent, and our own server-side agent. Tests tied too closely to a particular tool broke when we changed it, even if the user's actual functionality still worked. Implementation-agnostic tests are easier to maintain through those changes.

Each task and trial is independent and runs in its own container, so the pipeline can scale horizontally. Harbor works with sandbox providers such as Daytona, Modal, and E2B. We use Daytona and can run around 200 workers concurrently, which lets us work through a large test suite quickly.

Imagine four tasks, two agent configurations, and three trials for each task and agent. Each box in the diagram is a separate, initially identical container; once the agent starts, the trials can take different paths. Running them independently in parallel makes repeated evaluation practical.

From those trials, you can compute metrics such as pass at three: did the agent succeed within three attempts? You can calculate that for each task and aggregate across the dataset. A good dataset gives you a quick picture of how a change affects the capabilities you care about.

Harbor also plugs into Phoenix, which I maintain. Phoenix maps Harbor datasets, jobs, trials, and rewards to its own datasets, experiments, runs, and evaluations. It can convert ATIF trajectories into OpenInference traces so we can inspect the steps of a run.

OpenInference is a set of semantic conventions built on OpenTelemetry, the framework for telemetry such as traces, logs, and metrics. Those conventions describe AI application events in a form downstream tools such as Phoenix can ingest. I am also a maintainer of OpenInference, and its specification has been donated to the OpenTelemetry working group.

In the migration example, the ATIF record can contain an agent step, a model call, and a tool call. After conversion, Phoenix can display those as a trace with interfaces tailored to agents, language models, and tools. That lets us investigate what happened inside a trial rather than only seeing its final reward.

Phoenix also versions tasks and datasets, records experiments, and charts rewards and other metrics over time. Skills, a CLI, and an MCP server make that information accessible to coding agents. Harbor has its own local open-source UI and a hosted service, so Phoenix is optional; these features are reasons you might choose to add it.

To explain our own use of Harbor, start with Phoenix's normal architecture. An instrumented AI application emits traces over OTLP to the Phoenix server. The server stores them in SQLite or Postgres and exposes services for traces, spans, sessions, datasets, experiments, and evaluations through REST and GraphQL APIs.

We support several agent experiences around those APIs. A coding agent can use our CLI or our MCP server, and our server-side agent, Pixie, can use the MCP server as well. Pixie also has a bash emulator. These are separate surfaces that our team needs to support and test.

Our architecture gives us an advantage for testing: Phoenix runs on a single node and is easy to put in a Docker container. We seed it with a SQLite file containing realistic data and replace the browser with a test client. Once the whole application is in a container, we can run it as a Harbor environment.

A realistic user task might ask an agent to find hallucinated model calls from the last week of traces and add them to a dataset. We can give the task to any one of the three agent configurations. At the end, a verifier can call a REST endpoint or query the SQLite database to check that the expected dataset change occurred.

An attendee asks how Harbor controls that setup. In any one trial, we test one agent configuration, not all three at once. Harbor itself is a CLI orchestrator running on a host such as a laptop or a GitHub Actions runner; the environment may be a local container or a remote sandbox.

Harbor starts the agent and the other steps through scripts. A task supplies scripts for starting work, running verifiers, and running the oracle, placed where Harbor expects them. The contract is largely a set of conventions about the scripts and their locations.

The environment is flexible. A common first example is a coding agent working on a Git repository, but our example includes a SQLite database, the Phoenix server and APIs, a CLI, an agent, and skills on the filesystem. If you can package what you need in a Docker image, Harbor can run that environment.

A question raises a harder case: what if the application is a distributed system with services such as S3 rather than one server and one SQLite file? I have not had to solve that yet. One possible approach is to isolate the agent behind a stable interface and supply data at that boundary, though mocking external services can reintroduce the maintenance problems we wanted to avoid.

Harbor can run multiple Docker images, so packaging several components may be another avenue to explore. I do not claim to have a complete answer for a distributed system; our single-node architecture made our setup easier.

Another question concerns dataset size. We currently use a smaller collection of data and sometimes ask Claude to clean it up for a particular task. Many tasks can share one dataset, but they do not have to. If a question needs only a small subset, you can include that subset instead of uploading everything.

If the goal is to test an agent against a full database or under heavier load, including much more data would make sense. We have not done that yet. As for Phoenix versus Harbor alone, you can use Harbor by itself; Phoenix adds dataset versioning, experiment views, visualizations, and ways to inspect results in detail.

Our concrete case study starts with aggregate questions. Users might ask which day had the most traces or which model cost the most. Our MCP server and CLI could use REST and GraphQL, but the available endpoints were not aggregate resolvers, so an agent might paginate through many results and calculate the aggregate itself.

That approach can require many tool calls as the data grows. We added a read-only SQL tool to the MCP server so the agent could query the database directly. But adding the tool introduced a choice: the agent now had REST, GraphQL, and SQL, and it did not always choose SQL when that was the better fit.

We changed a prompt that had previously emphasized GraphQL. The new guidance steered the agent toward the SQL tool for counts, groupings, and other aggregates. The question then became how to verify that this prompt change actually improved the behavior rather than merely assuming it did.

We seeded the environment with a semi-realistic SQLite file gathered as we developed and used Phoenix ourselves. Then I asked Claude to examine the product domains—traces, spans, datasets, and experiments—and generate user questions, some aggregate and some not, along with oracle solutions using REST calls or database queries.

I spot-checked the synthetic questions. The verifiers included SQL queries, REST calls, and some model-as-judge checks. Seeing the verifiers pass against the oracle gave me a degree of confidence that the evaluation setup worked before I used it to compare agent behavior.

The prompt change made the agent use SQL more frequently, as intended, and lowered the average number of tool calls in its trajectories. The test data was not especially large, so the reduction may understate the benefit when a real workload would require many more pages of results.

In Phoenix, the dataset view shows the task examples and their version history. One example asks for the three most expensive models by total cost. Its reference answer comes from an oracle's SQL query against the seeded data; the result is deterministic, although I still rely on the query itself being correct.

Phoenix calls the Harbor jobs experiments and shows runs with metrics such as cost, latency, rewards, tool calls, and SQL usage. In this case the reward was already high because the system generally found the correct answer before the change. The aim was to reduce unnecessary calls, not repair a broken answer.

The reward comes from the verifier you write for each task. You decide what success means and how to recognize it from the environment, such as the filesystem or a database. A verifier might emit zero or one, or another named reward; the experiment view can then aggregate those results.

An attendee notices that some checks look inside the trajectory to see whether SQL was used, even though I described verifiers as black-box checks. That is a fair observation. For a pure correctness regression test, we can inspect the final answer or side effect without depending on the path the agent took.

This experiment has a different goal: I am optimizing tool use, so I do care about the path. Harbor can support both end-to-end correctness tests and focused investigations into a specific behavior. Looking inside a trajectory can be useful, though it may make a test less robust to implementation changes.

Phoenix can also help teams share and curate data, add traces to datasets, and store or upsert data from external files. I cannot claim much firsthand experience with a large team, though, because only a handful of us are working on this evaluation setup.

The comparison view lets me place two jobs side by side at both the aggregate and individual-task level. In the example, I compare the same model before and after the prompt change. GraphQL calls and overall tool calls fall, SQL calls rise, and the reward stays roughly the same.

The table lets me examine the expected answer, the actual outputs, and the metrics for each task. Individual results can differ from the aggregate: for one example, the number of tools and turns fell even though its SQL call count also fell. Looking at task-level results prevents an average from hiding that detail.

I can filter the comparison to find cases where the second experiment reduced tool use. A demonstration of converting a natural-language filter to a query DSL with a local browser model does not work as expected, so I paste a known expression instead and narrow the table to the cases I want to inspect.

The converted ATIF traces show what changed for a particular task. Before the prompt change, the agent used bash, which called GraphQL under the hood; afterward, it issued a SQL query directly. I can inspect the traces visually, or ask an agent to examine the experiments and explain the difference.

That is part of our feedback loop: use the agent, record its traces and results, and then use those observations to improve the agent itself. The task-level comparison links the overall metric change to concrete runs.

Our regression testing is still early. We are starting with happy paths for important skills, which in our case describe workflows. The tests resemble end-to-end tests with an agent and language-model calls in the loop, and we want those paths to pass across the models and providers we support.

An attendee asks what happens if a new skill's front matter overlaps with an existing one and prevents the intended skill from firing. One approach is to care only about the final outcome: if the task succeeds, perhaps it does not matter which skill fired. A good end-to-end test can catch failures when that choice affects the result.

Alternatively, a verifier can inspect the trajectory and explicitly check which skill was called. We have written such tests. They are more coupled to implementation details, so renaming a skill can break them, but a focused one-turn test can be simple: ask a question and check whether the intended skill fired.

There is a spectrum between an implementation-independent end-to-end test and a test coupled to a particular internal step. Both can be useful. The important thing is to cover the behaviors that matter while understanding the maintenance cost of each kind of check.

In production, instrumented applications send traces to Phoenix, which writes them to its database. In our Harbor setup, we omit the end user's application and telemetry ingestion and seed the database directly. We assume the production application has already produced the data represented in the SQLite file.

Harbor is often used to evaluate a coding agent itself, but that is not exactly our unit of evaluation. We do not maintain Claude Code or Codex. We evaluate a hybrid system: a coding agent together with our MCP server, CLI, skills, or APIs, or our own server-side agent and its supporting infrastructure.

Those pieces need to change and be evaluated together because they determine how well the agent can use our product. It took us some time to arrive at that picture. The fact that Phoenix runs on one node and is easy to containerize helped; teams with distributed systems may face more setup work.

Harbor's conventions also require scripts in the correct directories, and there are not necessarily linters a coding agent can rely on to check every part of that contract. Harbor's skills can help an agent set up a project in the expected form.

For a SQLite file, our current process is relatively ad hoc: we keep a collection of data and reuse a file when it fits a task. If the file contains too much, I may ask Claude to remove what is unnecessary and put the result in cloud storage.

Getting the data is the hard part. The more realistic the data, the more effectively the evaluation can uncover realistic problems in the agent system. That remains the main challenge when building these tasks.