SallyAnn DeLucia: How to Write Your First AI Eval

I’m SallyAnn, head of product at Arize. I’ve been here for about four years, and one of my first projects was building Alex, our AI agent, into the platform. I’ve worked on both agents and our evaluation product. I think of evals as the primary signal for our agents: they tell us how those agents are actually performing.

My goal is for you to leave with a clear definition of an eval, a recipe for writing your own criteria, and a working judge. We’ll establish a foundation, explore data in Arize, define what we want to measure, and build an evaluation. Along the way, I’ll show how that evaluation fits into a life cycle of testing and improvement.

Here’s why vibe checks don’t scale. On Monday, you change a prompt and ship it because it feels better. A week later, the use case you thought you fixed breaks again. As the problem gets worse, nobody can prove whether the behavior was fixed or is now failing. You only have impressions to go on.

Without evals, you can’t reliably detect regressions after a prompt change, objectively compare prompts or model versions, or know whether a new model is better for your specific use case. You also can’t put meaningful quality gates into CI. It becomes hard to say how your agent is performing, even while you continue to ship changes.

People often ask whether an eval is just a unit test. Traditional software tests commonly have an input and one correct output, and you can read the logic in the code. An agent’s behavior is different: much of its logic becomes visible only afterward in traces. A travel planner can produce many valid itineraries, so we need to assess criteria such as the itinerary itself and its budget breakdown.

I like to think of traces as the logs for AI and evals as the tests. The uncomfortable part is that you need observability to do those tests well. A trace records all the steps an agent took to answer a user. Each span represents an individual piece of that work, allowing us to see what happened between the request and the final response.

An agent might call a research agent, use tools, retrieve information from a knowledge base, and compose several model calls. There’s a lot of back and forth. This is where you can see the agent’s logic. A single conventional test of the final result won’t necessarily capture all that complexity or tell you which part of the agent is performing poorly.

Today we’ll focus on LLM as a judge: taking a second model and asking it to grade some aspect of our agent. That might be question-answering correctness, summarization, or relevance. The result is a label and an explanation. The explanation matters because it gives us some understanding of why a judgment was made rather than asking us to accept a label at face value.

There are other types of evals. Decision models are an emerging option that can be cheap and fast: you ask a question and receive a probability and an output such as a true/false result or label. Unlike an LLM judge, they generally don’t give you an explanation. Code-based evals are useful for deterministic checks, such as whether an output contains a word or is valid JSON; ROUGE and BLEU are other examples.

Annotations are also a form of evaluation, using labeled examples or golden datasets, sometimes including synthetic data. An agent can judge another agent, which can help with complex success criteria. User feedback and business metrics add production signals. You want development evals to keep growing with the application instead of investing in tests you later discard when you move to production.

For the workshop, we’ll use Arize AX and a demo project. You’ll need access to a model provider so we can make calls while testing the evaluator. A custom endpoint is fine; it doesn’t have to be one particular provider. The important thing is having a working model connection that the judge can use.

In the product, go to Settings and then AI Providers. Choose the provider you want to use—Anthropic, OpenAI, Gemini, or another supported option—and add its API key. Use the test-integration button to make sure that connection works. The tracing page also offers a demo project, and older versions of the demo are still usable for this exercise.

Our demo is a travel-planning agent. It receives a request, can ask follow-up questions and call tools, and returns a plan. In product terms, good behavior means answering the user’s follow-ups and respecting the requested number of days and budget. A request includes details such as destination, trip length, budget, and general travel style.

Before writing an eval, inspect a few traces and take inventory. What are users asking for? What did the agent do? Which tools did it use, and what answer did it produce? Ask yourself whether this is a response you would actually share with a user. That exploration helps you decide what is worth measuring.

In one example, the user wants to visit Prague for three days on an $800 budget and asks for something more local and less touristy. The agent returns a detailed plan for days one, two, and three. We can read the final itinerary and open the attributes to see additional data associated with the request and response.

The trace tree shows every individual step. If you want to know what information went into a particular model call, select that span and read its input and output. Pay attention to where the data lives. Later, your evaluation will need the right evidence, and that evidence may be at the root of the trace or inside a specific child span.

As you explore, think like a stakeholder in this travel agent: a product person, an engineer, or anyone responsible for its results. Look at how people interact with it and how it responds. Start making a mental list of what you would measure to determine success. We’ll turn those observations into explicit criteria in the next part of the workshop.

A trace is the collection of steps taken to answer a user’s question. At the top is the root span, whose parent ID is null. It has no parent above it, and it usually contains the overall input and final output. Beneath it are individual spans, each representing a unit of work.

In this example, the larger agent calls a research agent as part of a multi-agent system. That research agent calls a tool to get external information, such as weather or details about a destination. The icons distinguish agent spans, tool spans, and model spans. Following them lets us see how information moves through the application.

A model call can take information returned by a tool and summarize it for a later step. You can inspect the actual inputs and outputs for that call, then move down the tree one step at a time. The connected units of work ultimately support an itinerary that matches the user’s requested trip, preferences, activities, and places to stay.

An audience question is why two model spans appear around a tool call. Sometimes instrumentation separates the model choosing a tool from the model processing the tool’s result. In this trace, one model call selects the tool; the tool retrieves information such as temperature and destination data; a subsequent model call summarizes that output for further work.

Does the root node make all the decisions? That depends on the architecture. You might have a supervisor or orchestrating agent that makes a plan for subagents, or a model with tools that chooses what to do step by step. This example appears to be a LangGraph agent with modular agents responsible for different tasks, rather than a supervisor architecture.

Arize is framework-agnostic. Our OpenInference tracing package provides auto-instrumenters for different frameworks, and the goal is to offer useful observability and evals regardless of which framework you choose. Once you’re familiar with the traces and have spotted behaviors worth measuring, you’re ready to define the evaluation.

The first step is defining what good means. Avoid stopping at words like ‘helpful,’ ‘accurate,’ or ‘good response.’ Be specific. For our travel agent, good could mean answering the user’s follow-up, matching the requested number of days, or keeping the total within the stated budget. Those are concrete criteria tied to the application’s actual job.

Judge one quality per eval. There are exceptions: decision models and agent judges can sometimes handle multiple criteria together. But for an LLM judge, it’s good practice to choose one criterion and add more evals over time. Evaluation is iterative. You don’t create a suite once and assume it covers the application for the rest of its life.

When I first built evals for Alex, the set was simple: question-answering correctness, user frustration, and safety checks such as jailbreaks and toxicity. Now we run 32 evals on Alex. Start small, ground your decisions in real production usage, and evolve the suite as you learn what else is necessary to define success.

Is a generic eval better than a specific one? Generally, an eval tailored to your use case is better, but some evaluation is better than none. Arize’s eval hub includes templates for correctness, hallucination, session coherence, and other common concerns. Those are useful starting points and can be shared across an organization, then aligned and made more specific over time.

For the next exercise, write down six things: the task, what success looks like, what failure looks like, the quality you’re judging, and up to three bullets each for passing and failing. Those notes will become your eval. The more concrete you make them, the easier it will be to turn them into a useful prompt.

Someone asks whether the request information visible in these traces comes out of the box or requires custom instrumentation. Automatic instrumentation captures common attributes, such as the input messages sent to a model. Manual or hybrid instrumentation lets you add application-specific information. The aim is to get the evidence your use case needs while making the common setup straightforward.

You can also let an agent help instrument your application using the available skills, then guide it as needed. In the rubric example, the task is responding to a user’s follow-up about a trip. Success is answering that follow-up directly with the trip’s details; failure is falling back to a generic budget response. That is much more useful than simply saying the answer should be good.

Why write pass/fail bullets instead of a single sentence? Because we’ll use them to build an LLM-judge prompt, and the model needs an explicit description of what to look for. Treat it like explaining the acceptance criteria to a coworker or another stakeholder. If your rubric tries to do too much, cut it back before moving on.

To build the eval, return to the traces and select examples for a dataset. If you’re evaluating whether the itinerary matches the user’s input, save the root spans, where those two pieces of evidence live. If you’re evaluating whether a tool call returned the right information, select the relevant tool-call spans instead. The dataset should match the question your evaluator is answering.

Filters help you find those examples. For root spans, filter on parent ID being null or use the default traces view. To isolate a particular tool, filter by its name. Select the relevant spans and add them to a new dataset. Once the examples are saved, we can move into the playground to build and test the evaluator.

The first pass is: inspect traces, identify the span to judge, create a filter, and add those spans to a dataset. Then open a new playground. Arize has a prompt playground for testing evaluator prompts or other model prompts, and an agent playground for testing an application end to end. We’ll use the prompt playground for this exercise.

If you’ve already created an eval in the eval hub, add it from the dropdown. Otherwise, start with a blank eval. Take the criteria you wrote and turn them into a prompt; much of the rubric can be pasted directly. Begin by explaining the evaluator’s job so the model understands exactly what it is supposed to judge.

For example, the judge’s job might be to evaluate whether an assistant’s output accurately captures details requested or provided in the input. Then provide the data. The relevant information is in the attributes we inspected earlier, but you don’t need to hard-code those exact attribute names into every prompt. Variable mapping connects the prompt’s inputs to the actual data.

Variable mapping keeps the evaluator flexible. A dataset and a trace might use different names for the same information; instead of rewriting every eval, you can change the mapping where it is used. The prompt can concentrate on the criterion: mark the result correct only if the output is consistent with the input, and incorrect if it gets a material detail such as destination or property type wrong.

Under parameters, configure the labels the evaluator may return. This example uses correct and incorrect, but you can include partially correct or not applicable when those distinctions matter. Some Alex use cases benefit from knowing that a response is only partly right. The evaluator returns one label, and you also specify whether a higher or lower result represents better performance.

Enable explanations so you can understand why the eval assigned its label. Load the dataset built from your selected spans and map its inputs and outputs to the prompt variables. With the evidence connected, run the eval. In the demonstration I use GPT-4.1 mini, and the results appear directly in the playground.

Now iterate on the template. Read the response, decide whether you agree with the judgment, adjust the prompt, and run again. When the evaluator is in a useful state, save it. It will then be available elsewhere in the platform, both for the demo project and for applications you build and bring into Arize.

We’ve written the criteria and tested the evaluator ourselves, but Alex can help after that first pass. It can run the playground, inspect results, revise the prompt, and repeat. The broader improvement loop also includes Signal, our managed agent, which can inspect traces, find issues, and make fixes. Evaluation results become a signal that those improvement tools can consume.

The next step is alignment: annotate the examples in your dataset, run the eval, and check whether its judgments match the human feedback. Online evals put that evaluator into a task that runs continuously over incoming traces. As traces arrive, the configured evals score them automatically, giving you a way to monitor the application in production.

Every stakeholder evaluates the agent. This isn’t the responsibility of just one role. Bring together the people with a stake in its outcomes—product, engineering, support, domain experts, and security—and have them define success from their perspectives. Metrics and evals work together: a metric can signal that something may be broken, while an eval helps identify the specific failure mode.

Another useful experiment is converting an LLM-judge criterion into a decision-judge question and comparing the two. See which performs better for that task. Above all, start small: perhaps 20 traces, one dimension, and one judge. Read the explanations, iterate, and then expand to more evals. There are documentation and benchmarking resources to help you continue beyond the workshop.