How Arize Rebuilt Its Alyx Agent: Planning, Memory and Evaluation

I'm Fuad Ali, an AI product manager at Arize. Today I want to share lessons from roughly two years of shipping our in-product production agent. The aim is to give other teams useful design questions as they build their own agents.

Our work gets a little recursive. Coding agents help build our products; Signal reads agent traces; Alyx is the in-product agent I am discussing; and we use Alyx to debug Alyx. We can use the same tools to evaluate our agent and examine design patterns that might transfer to other systems.

Alyx lives inside Arize X. It can read traces, turn findings into evaluations and datasets, run experiments, plan multistep work and generate synthetic data from traces. A staff engineer described the first late-2023 prototype, using models from the 3.5 era, as about half hallucination. That is an internal characterization of an early prototype, not a measured comparison across products.

The product moved from a simpler prompt-and-tools interface to a deeper workflow with planning and explanations of what it is doing. In one example, Alyx is asked to optimize its own system prompt and plans how to inspect resources in the application.

The prototype began in late 2023. The team moved mathematical work out of the model and into harness code, then launched a copilot organized as a router over skills. An August 2025 rewrite introduced iterative tool calling and a to-do tool; a February 2026 relaunch and a June consolidation followed. Long-term memory was added later.

We didn't train the frontier models behind these changes. The models improved on their own schedule, but our rebuilds were primarily harness code. We don't use an agent framework: we own the loop, plan, stopping rule and memory. That gives us control and a way to learn from the system.

The first design question is who owns the plan. Think of giving a child a list of chores: you could ask them to maintain the list, or you could keep it yourself and provide the next task. Each approach makes different demands on the person doing the work.

Initially we gave the model a to-do list. It spent too many calls checking off items instead of doing useful work, and could lose focus after many tool calls. We then moved the plan into the harness incrementally, beginning with an in-progress state.

Next, the harness rendered the plan as a message on every model call, placing the rules next to the relevant data. We later removed the in-progress marker: if unfinished items are ordered, the next unfinished item already identifies the next task. We are now testing a version without a separate to-do tool, keeping state and completion checks in the harness.

The next question is what ends a turn. Our first rule treated text without tool calls as completion. That broke when a model announced a step in one turn and made the tool call in the next. Model changes can expose this kind of assumption, and customers can also choose their own providers.

The current design uses an explicit finish call. If plan items remain open, the harness can send the model back up to three times. The slide also describes nudging plain-text tool calls, stopping repeated calls that make no progress, and warning before the iteration limit followed by a forced finish.

Memory creates several related problems. A large tool result is like a book that takes over your desk; several such books can fill the desk entirely. Rather than leave the whole result in context, we keep a reference and a high-level description so the agent can retrieve what it needs.

Trace analysis can return a large JSON object that would crowd out useful work in the context window. Alyx truncates the immediate result and can write code or use tools such as jq to inspect the full material. This makes use of models' ability to work with code rather than dumping everything into the prompt.

The agent's own history also grows. The harness asks the model to write a checkpoint and then drops the material the checkpoint replaces, subject to a fixed history token budget. Across sessions, a background job edits a small file for each user.

For long-term memory, we tried vector retrieval, episodic and semantic stores, and temporal knowledge graphs. We ultimately chose a Markdown file per user per space, capped at 8,000 characters and loaded in full on each request. A background job applies small edits after a turn, with a safety check on writes.

The small file is easy to load and audit without overwhelming context. A human labeler can inspect what was saved and adjust the background job's instructions. Ali presents this as a lesson from Arize's experience in avoiding unnecessary engineering complexity, not proof that files outperform other memory systems everywhere.

We began with one trace agent, then grew to ten page-specific specialists. A dataset page, for example, had its own agent for generating synthetic data and inspecting datasets. We later merged the main flow into one agent that loads skill instructions and tools as needed, letting it choose the tools to use.

The slide shows that scoped subagents remain for bounded jobs rather than planning. It also reports reductions in prompt tokens and in prompt-plus-tool-definition size across the redesign. Those are figures from this system's architecture, not a general prediction for every consolidation.

The newest model isn't automatically the best default. A general frontier ranking differs from what works for a particular product. We usually choose a model that is fast and good enough, then spend time on the harness; the team is also exploring task-specific routing and small post-trained models.

Some tasks do not require an agent loop at all. Ali gives AI-assisted filter generation as an example where a single call to a small model may be sufficient. Adding a full loop in that case could be another form of overengineering.

Tracing production sessions helps us turn real user questions and failures into regression datasets. These fixtures change over time as new cases appear. Session datasets can also retain a full session and replay a particular turn, while experiments help compare harness changes.

Agent bugs do not always appear as explicit errors. A run can report success while making 43 identical tool calls in one turn. Latency also needs context: 227 seconds may clearly be too long, but an acceptable response time depends on the application. Signal helps the team surface patterns in traces.

A fix can introduce the next bug, so regression datasets matter. Evaluator judges need a human baseline: the team aligns automated judgments with engineers' assessments and uses human-created golden labels and datasets. Automation reduces some trace reading without eliminating expert judgment.

Choosing the tradeoff between speed and quality remains a human decision. Metrics can show the latency and quality changes from switching models, but they do not decide which tradeoff is appropriate across every part of the product.

Six practices survived the rebuilds: tracing into Arize's own product; doing mathematical aggregations in code and having the model explain them; golden datasets in continuous integration; recoverable errors with an obvious next step; streaming and persistence; and skills. The meaning of skills changed from a router branch to instructions and tools the agent can load itself.

Ali closes with the questions he wants builders to ask: who owns the plan, what ends a turn, how to handle oversized tool output and long-term memory, whether to use one agent or several, which model is fast and capable enough, and who checks the agent's fixes. His team continues to use Alyx to build Alyx.