Building an Agent Regression Loop from Production Traces

I'm Sofia Jakovcevic, a forward deployed engineer at Arize. An agent can make building AI systems feel intuitive, but its behavior can also be a black box. Observability helps us see what the system actually does step by step; we can then evaluate its behavior and use that information to improve it.

The challenge is not just finding what broke. When we submit a fix, we also need to know whether it breaks other workflows. I want to show a regression process grounded in observed behavior and connected to continuous integration, before considering how agents might help close the loop.

Imagine a support agent and a security engineer who asks for one extra line: confirm before taking an action. It sounds harmless, and natural-language prompts can give us a false sense that we know exactly how a change will behave. Instead of relying on intuition alone, we need an evaluation suite covering the outcomes we care about.

In the example, a later evaluation shows action safety improving by 17 points. But task completion falls by 17 points: if the agent asks for confirmation at every step and the user leaves, a task may never finish. Averaging the two movements gives no net change and hides the tradeoff. The scores are from the demonstration, not a general effect of confirmation policies.

A reviewer may see a one-line change to a system prompt, but the behavioral effects occur when the agent runs. The change can alter steps, subagent handoffs and tool calls across many possible trajectories. A diff shows what changed; a trace records what happened in a particular run.

Jakovcevic argues that traces help expose silent failures that are difficult to infer from static changes alone. They are observations of actual runs, not a complete guarantee that every possible behavior has been covered. She uses them as the starting point for a regression loop.

The proposed loop has five steps: select a dataset from production traces, define evaluators, run a baseline experiment, compare a change against that baseline, and enforce chosen criteria in CI. The aim is to test a candidate before deploying it, rather than discover all of its consequences afterward.

For the dataset, use real inputs, tool calls and outcomes that represent normal workflows. Add known failures and edge cases, but also include successful workflows that must be protected. Testing only the failure being fixed may miss a regression in something that already worked.

Next, choose scoring mechanisms for the dimensions that matter: task completion, factuality, tool trajectory or efficiency, for example. Jakovcevic describes using a language model as a judge with a rubric to classify a trace or span. A score makes comparison possible, but a model judge can itself be wrong and needs validation against suitable human or other reference judgments.

Evaluations need not all use language models. Code-based checks can test deterministic properties, while model-based evaluators may help assess less deterministic behavior. The choice depends on what the team is trying to measure and on the evaluator's own reliability.

Run the agent against the selected dataset and score its outputs to establish a baseline. Then make one change—such as a policy, prompt, model, tool or handoff—and rerun the same tests. Comparing dimensions separately makes it easier to see an improvement in one area alongside a regression in another.

Jakovcevic gives model selection as an example: a more expensive model should be assessed against the quality, cost and latency that matter for the application. Experiments can inform that tradeoff, but a numerical score is evidence within the test design, not proof of return on investment in every deployment.

The fifth step is to put the experiment into CI. A pull request can run the regression suite, and the team can require its chosen criteria before merging. A failed candidate goes back for investigation and iteration; a passing gate is only as useful as its cases, thresholds and evaluators.

In the Arize demonstration, Jakovcevic shows a refund-agent experiment with several prompt variants. The display compares action-safety and task-completion results; one example shows the opposing 17-point movements, while another candidate has different improvements. These are demonstration results, not a controlled comparison of all agents or products.

She opens a prompt playground, selects a saved prompt and a 100-row dataset containing edge cases, known failures and successes to protect. She duplicates a candidate so two prompt versions can be tested side by side, then adds task-completion and action-safety evaluators.

The interface runs the evaluation over the records and presents metrics including evaluation scores, latency, cost and error rates. Jakovcevic then shows an agent playground that can call a remote agent over HTTP, allowing the same dataset and evaluators to exercise a more complex system with subagents.

Besides the user interface, she says experiments can be run programmatically through the SDK, API, CLI or CI. She highlights Arize's public agent-skills repository as a way to connect coding agents to the observability system and the context behind production behavior.

The human-led loop is observe, evaluate, change, test and improve. Engineers collect traces, score them, change a prompt, model, tool, skill or policy, and test the candidate. When new production failures appear, those cases can be added to the regression suite so future changes are checked against them.

Jakovcevic asks what happens if an agent investigates a failure and proposes a fix instead of leaving all dashboard review and triage to people. She envisions connecting coding agents to runtime traces so they can inspect behavior and suggest changes. This is a proposed direction for agent observability, not evidence that autonomous fixes are always correct or need no oversight.

Her closing argument is that code changes made without runtime context can be blind to production behavior. Giving agents access to traces, evaluation and feedback can help them investigate and improve their own systems. The value of that loop still depends on the data, tests and safeguards around the proposed changes.