From Chaos to a Failure Map: A Hands-On Primer on AI Evals

Hamel Husain defines evals as a systematic way to measure and improve an AI application. At its core, he says, it is data analytics on your LLM application, with metrics created where necessary so you can run experiments and iterate. It does not have to be scary or unapproachable.

Without evals, builders are left guessing: tweak a prompt, hope nothing else breaks, and rely on vibe checks. Husain says vibe checks are fine at the start, but they become unmanageable as an application grows and you start to feel lost. Evals give you a feedback signal you can iterate against with confidence.

Host Lenny Rachitsky offers the analogy of unit tests for code, but Shreya Shankar gently widens the frame. Unit tests can guard non-negotiable behaviors, yet an AI assistant also faces open-ended requests and new groups of users nobody anticipated. Evals can also mean regularly looking at data to spot new cohorts, or tracking simple metrics such as thumbs-up rates that feed the product-improvement flywheel.

Husain warns against the common trap of jumping straight to writing tests. LLM applications have far more surface area and are far more stochastic than traditional software, so you should ground what you test in data analysis first. His demo uses anonymized production data from Nurture Boss, an AI assistant for apartment property managers that handles leads, customer service and booking across chat, text and voice, with many tool calls and retrieval.

Loaded into an observability tool (he happens to use Braintrust, but notes Phoenix and LangSmith work as well), each interaction is a trace: a full log of the system prompt, tool calls and replies. In the first example a prospect asks about a one-bedroom with a study and the bot simply says none is listed and wishes them a nice day. Husain notes that with a product hat on this is not ideal, and he writes a one-line note: it should have handed off to a human.

The method is deliberately casual. Write down only the first, most upstream thing that went wrong in each trace, then move on. Other traces reveal garbled text-message conversations that the system cannot parse, and a bot that promises a virtual tour that does not exist. Husain says everyone who tries this gets addicted, because you learn so much so fast, and product people need to be in the room because developers are often poorly placed to judge these conversations.

A frequent question is whether an LLM can do this first pass. Shankar says no: asked to review a trace, a model will usually say it looks good because it lacks the context to smell a bad product experience. She would bet money that ChatGPT would approve the hallucinated virtual tour, because only Husain knew the feature does not exist. Free-form note-taking, which they call open coding, is not the place for automation.

To keep the process cheap, Husain coined the term "benevolent dictator". Teams often get bogged down trying to have a committee review every trace, but one person whose taste you trust is usually enough, especially at small and mid-sized companies. Simplify across as many dimensions as you can, he argues, or the process becomes too expensive to run.

That person should be the domain expert: a lawyer for legal products, a psychiatrist or similar expert for mental health, an apartment-leasing specialist here. Oftentimes it turns out to be the product manager. Notes should be specific enough to be useful later, such as "did not confirm call transfer with user", not just "janky", and perfection is not the goal.

Husain and Shankar recommend reviewing at least 100 traces, but the number is mostly a psychological device. Once you have done around 20, he says, the exercise is so useful you will carry on anyway. The honest answer is to keep going until you are no longer learning anything new.

Shankar names this idea theoretical saturation, a term from qualitative data analysis: you stop when new notes no longer surface new concepts or change the next step of your process. It takes some intuition, which develops after two or three rounds. Depending on the application and your skill, you might find that 15, 40 or 60 traces are enough.

Once you have notes, this is where AI can help. Husain pastes a CSV of his open codes into Claude, ChatGPT or similar and asks it to propose axial codes. Shankar explains that an axial code is simply a failure-mode label, and that many of your 100 notes will be repeats phrased differently. The goal is clusters of failure modes so you can find the most prevalent one.

The model's first attempt is a decent start, but Husain finds some categories, like "capability limitation", too generic to act on. He rewrites them into actionable ones such as tour-scheduling and rescheduling issues, human handoff problems, formatting errors, conversational flow, and follow-up promises not kept. Shankar stresses that the prompt can be as detailed as you like, and that the LLM synthesizes but does not propose fixes; that remains your job.

Husain then uses a spreadsheet formula with an AI call to label each note with one of the final categories, and Shankar suggests adding a "none of the above" option as a signal that the taxonomy is incomplete. A pivot table finishes the job: in his sample there were 17 conversational flow issues. Shankar adds that none of this was invented by them, since error analysis and open and axial coding come from machine learning and social science, and Husain shows an eight-year-old Andrew Ng video on the same technique.

Rachitsky observes that you do not have to do this often. Shankar says you do it once, then build on it, and that a weekly pass can take about 30 minutes and leave the product far better than if you never knew about the problems.

With counts in hand, Husain says the count is not the only criterion, since something very bad may deserve a fix regardless. Many problems are plain engineering errors. If the model simply was never told how to format output, fix the prompt; you may still add a cheap code-based check that tests the string without calling an LLM. There is a cost-benefit tradeoff, and you should ground yourself in actual errors.

Shankar clarifies the two kinds of automated evaluator. Code-based ones work for checks like whether output is JSON, markdown or short. For subjective failure modes such as whether the bot should have handed off to a human, you use an LLM judge scoped to that one failure mode. Because the judge does a single, tightly scoped task and returns pass or fail, it is far easier than building the original agent.

Judges are not just for pre-release unit tests or CI. Shankar points out that you can sample, say, 1,000 production traces a day, run the judge, and watch the failure rate online. Shankar says she typically ends up with four to seven judges, reserved for the pesky failure modes that persist even after you describe the ideal behavior in your prompt. The prompt is a one-time cost that can then run on your application indefinitely.

Husain insists that each judge check one failure and output a binary result. A one-to-five rating is, in his words, a weasel way of not making a decision, and nobody knows what 3.2 versus 3.7 means. Shankar adds that she sees expert-curated one-to-seven scales online and groans, because people then come back reporting a 4.2 average.

His example handoff judge outputs true or false and lists when a handoff is required: explicit human requests ignored, policy-mandated transfers, sensitive resident issues, missing tool data, and same-day walk-in or tour requests. An LLM can help write the prompt, but you must read and edit it. Most important, you must check that the judge agrees with your own labels before releasing it, because when people lose trust in your evals they lose trust in you.

Raw agreement is a dangerous metric, Husain warns. If the error occurs only 10% of the time, a judge that always says pass scores 90% agreement. He and Shankar recommend a confusion matrix of human versus judge labels, then iterating on the prompt until the off-diagonal cells shrink. If a colleague reports only that the judge agrees 75% of the time, Shankar says, that is a bad smell.

Rachitsky notes that other guests have said evals are the new PRDs, and the judge prompt seems to prove it: a precise statement of how the product should behave, running continuously. Shankar agrees but adds that the prompt is derived from your own data, so it contains expectations you could not have imagined before looking at traces.

She points to her research paper, "Who Validates the Validated?", from a user study that began in late 2023. The surprising finding was criteria drift: people cannot fix their rubrics up front, because their opinions of good and bad change as they review outputs, and even experienced builders think of failure modes only after seeing ten of them. You should still write PRDs, she says, but expect them to change, since with LLMs you do not know what you want until you see it.

After the judge exists, teams use it everywhere: example failing traces become unit tests, and online dashboards track failure rates. Shankar notes that teams doing this well rarely talk about it publicly because it is their moat. Husain adds that this is simply the same data science mindset applied to AI products.

Shankar thinks everyone in the evals fight is largely on the same side. People hold rigid definitions, such as evals being just unit tests, and many have been burned by badly done evals, like a Likert-scale judge that did not align with their expectations. She sympathizes: you should be anti Likert-scale judge. Social media then flattens it all into slogans such as "we don't do evals".

On the claim that Claude Code runs on vibes, Shankar says it stands on the shoulders of foundation models measured on many coding benchmarks, and that the team is almost certainly monitoring usage and dogfooding in ways that amount to error analysis. Husain adds that coding agents are unusual because the developer is the domain expert and uses the tool all day, so the process can collapse. Doctors, for example, are not dogfooding medical AI in that way, and he cautions that many teams claim dogfooding they do not really do.

On A/B tests, Shankar says they are just another form of eval, a metric compared across conditions, but teams often run them prematurely on hypothetical requirements before any error analysis. Husain says the A/B-versus-evals argument is the same data science thinking under a new label. Asked about OpenAI buying Statsig, they say they know little, but hope big labs will move beyond generic benchmarks and generic tools like cosine similarity or hallucination scores, which do not correlate with product-specific failures like handoffs.

Asked for common misconceptions, Husain lists first the belief that you can buy a tool, plug it in, and have AI do the eval for you. Second, people simply do not look at their data. In consulting he often opens individual traces with a client and finds the problem within minutes. Shankar adds a third: there is no one correct way to do evals, though there are many incorrect ones, and error analysis should always be part of it.

Her tips are not to be scared of the data and to remember that the goal is not doing evals perfectly but actionably improving your product. She also encourages using LLMs to organize your thinking, but not to replace yourself. Husain says to build your own annotation tools: AI makes simple web apps cheap, and the Nurture Boss team built one in a few hours that hides the system prompt, separates channels and shows error counts.

Husain reminds students that the point is not a beautiful eval suite but a better application: if something is obviously wrong, just fix it. On time, Shankar spends three to four days on the initial rounds of error analysis, labeling and a few judges, then about 30 minutes a week. Husain says the process is fun, recalling a client's recruiting email that opened with "given your background" and that he found generic and deletable.