Fuad Ali: Building a Feedback Loop to Improve AI Agents

I am Fuad Ali, a product manager at Arize. This workshop demonstrates a feedback loop for improving agents: collect traces, identify failures, turn them into tests, propose a fix and run an experiment. I will also discuss where human judgment belongs in that loop.

Agent failures are difficult to understand because an agent can take many nondeterministic paths, and organizations may operate thousands of agents across different functions. Traces, customer feedback and human annotations can supply evidence about what happened and what should improve.

Imagine a shopping assistant telling customers that the store sells no board games. A production failure like this might page an engineer at 4 a.m. The goal is to move toward a process that finds and investigates issues continuously, while keeping people involved where their judgment matters.

The demonstration uses Wonder Toys, a local chat-based shopping assistant with a deliberately planted bug. It can search for products and send traces to Arize AX. The workshop setup also requires an LLM provider key and the application’s development tools; following along is optional.

Teams often begin with tracing: they can see agent calls, tools and failures. The next stage is evaluation—asking whether the agent did the right thing. The harder missing pieces are reading traces at scale, saving failures as regression tests and proving that a fix did not break something else.

The proposed loop moves from traces to Signal, then a dataset, an evaluation, a fix and an experiment. Signal reviews traces and proposes issues and fixes; a person confirms the finding and reviews the change. New evidence and feedback can then inform later runs.

The toy-shopping agent uses FastAPI, Next.js, ChromaDB and agent SDKs. A planted filter bug causes search problems. Instrumentation sends the request path to the platform so the team can inspect model calls, tool calls, inputs, outputs and latency.

For a request such as “show me toys for nine-year-olds,” the trace tree shows the agent call, model calls and product search. Attributes include timing, metadata and returned database results. Those details are valuable, but reading thousands of individual traces manually is not practical.

The AX CLI and instrumentation can create spans for tool calls with their arguments and results, model calls and token counts. Related traces can be grouped into a session, making a multi-turn customer interaction easier to understand than isolated requests.

Signal is a scheduled agent that reads traces from a project. The default schedule in the demonstration is every six hours. Connected to a GitHub repository, it can compare production evidence with source code, cluster related issues and propose a change.

In the demo, Signal identifies redundant search retries and three sequential model calls, with latency around 14–17 seconds. The underlying bug is a mismatch: categories are stored as lowercase, hyphenated slugs, while the model supplies human-readable names with capital letters and spaces.

Without normalization, a category search returns no result and the agent searches more broadly. Signal’s proposed fix analyzes the repository and trace evidence. The issue can be used to create an evaluator, collect failure examples and open a pull request, but the finding still needs validation.

Managed-agent templates can investigate traces or examine cost, and teams can configure their own agent, task and skills. Ali describes connecting sources such as GitHub or application monitoring and triggering investigation on a schedule or an alert, such as a latency spike.

Signal groups related evidence into one issue rather than filing hundreds of duplicates. A reviewer can resolve or ignore findings. Adding confirmed failures to a dataset creates a growing set of fixtures for regression tests as new use cases and failure modes appear.

An evaluation codifies the expected behavior. A deterministic code check is cheap and useful for exact conditions such as valid JSON, a required field or a tool called with the correct arguments. A model judge can assess more qualitative results but adds model-call cost.

For more open-ended goals, an agent can act as a judge and inspect a broader trajectory, typically at higher cost. Ali also describes small decision models that return typed choices or scores as another relatively inexpensive way to classify traces. Sampling and evaluator choice are design decisions.

Trace evidence and evaluation results give a coding agent context for a proposed fix. Humans remain valuable in reviewing the pull request, deciding which failure cases belong in the dataset and defining what good output means. High-stakes work may require especially careful review before merge.

In the demo, Ali creates an evaluator from Signal’s redundant-search issue. The generated prompt inspects the trajectory and flags unnecessary repeated product searches. The evaluator can run on incoming traces, drive an alert and have its cost monitored.

An evaluator that has become redundant may later be retired, for example after a failure is reliably covered in development. The choice is not simply whether to run every evaluator on every production trace: the cost and remaining risk should guide that decision.

The experiment stage asks two questions: does the proposed fix address the failure, and did it break anything else? Even a small prompt change can have unexpected effects on other tools. Replay the agent against failure cases and other regression examples, then score the outputs.

Ali shows an experiment with an 88.9% result that he considers acceptable for this demo. The threshold is a decision for the team, not a universal guarantee. The agent can replay cases through a revised version and measure retry counts, latency and other checks.

A reviewer can inspect experiment outputs and annotate disagreements. Those annotations become additional context for future work. The same checks can be incorporated into CI/CD, but teams must choose models and sampling rates carefully to avoid spending excessively on validation.

After a fix, an online evaluator can continue checking new traces, and failed results can be added to datasets. Signal can run again on a schedule or alert. A person still needs to confirm that an issue is real, review the code and decide whether to merge and deploy the new version.

Confidence-based routing can direct uncertain judgments to an annotation queue. Ali cites a spam example in which routing 4.6% of cases with confidence between 0.3 and 0.7 to people left the rest at 99.5% accuracy. He cautions that thresholds depend on the particular use case.

The longer-term vision is a more automated loop, but running experiments safely raises questions about dependencies. A simple demo can start its own database; a real system may need mocks, tool restrictions, containers or other isolation. External dependencies make the test substantially more complicated.

Ali describes a goal of moving from an issue to an experiment with the necessary environment prepared. Findings and evidence can also be surfaced through tools such as Slack, local agents, APIs and the CLI. A pull request should carry the failed traces so a reviewer can verify the diagnosis.

Evaluators themselves need maintenance. A check that almost always passes may no longer justify its token cost, while an unhealthy evaluator may need improvement. Aligning a judge with human expectations is a separate process from simply obtaining a score.

Ali’s central recommendation is to use agents for the heavy work of reading data, while reserving human attention for the most consequential judgments. Traces, evaluation scores, human feedback and evidence should remain linked so the next iteration can learn from the same context.

During questions, he distinguishes evaluating Signal as a product from evaluating a customer’s agent. Quantitative issues such as retries and latency are relatively direct to measure; qualitative outcomes are harder to define. One example alone risks overfitting or failing to represent the broader use case.

For an agent that generates trace filters, the team can examine the whole user session: did the person ask again or create more filters? A product manager can also annotate whether the query and result were appropriate. Combining those observations with new cases helps improve a judge’s prompt and test set.

A rare failure can still be catastrophic, so a high pass rate alone is not enough. Ali distinguishes regression checks in CI/CD from evaluations on live traces. He recommends catching what can be caught before deployment, then using a production sampling rate informed by cost and risk.

Evaluators and monitoring agents also produce traces. Arize can evaluate those traces, including using Signal to examine Signal, but the process has diminishing returns. Proxy metrics such as pull-request merge rate are useful only to a point; without customer ground truth, additional scores may not establish real value.

Ali is interested in low-latency decision models for checks during execution, rather than only after a run. Lower evaluation cost may enable more teams to classify traces. For difficult multimodal judgments, confidence scores can help route uncertain cases to people, though they do not eliminate the need for judgment.

Custom managed agents can combine trace data with application metrics and logs from sources such as Datadog. Teams should connect the sources they need, scope access appropriately and avoid flooding the agent with irrelevant context; more information is not always better for triage.

The cost of the improvement loop depends on the harness, provider, frequency and context supplied. Teams can compare the cost of improving an agent with the value of the improvement, adjust evaluation sampling and context, or use the available APIs with their own coding agents.