Ramakrishnan Lokanathan: Evaluate the AI Agent’s Job, Not Just Its Answer

Evals are often broken because we overcomplicate them without asking what we are trying to measure. An evaluation is an exam for an AI system: a set of tasks, standards and evidence that helps us understand whether a model or agent performs as expected. The same basic idea underlies familiar human examinations.

An exam should answer a bounded question. The SAT, for example, offers one signal about college readiness; it is not a complete measure of a student's abilities. Likewise, an AI benchmark should be one signal, not a reason to assume that a model will do every part of a job well.

The practical question is whether performance matches the promise we make to the user. If we ship an agent for a particular job, the evaluation should tell us whether that product delivers the promised outcome in production, not merely whether its underlying model excels on a public benchmark.

I grew up in Cochin, India, and began coding and building machine-learning classifiers early. After seeing a GPT-3 product demo in 2021, I became convinced that new kinds of applications were possible. My team went full time shortly before the ChatGPT research preview appeared; the change both validated our direction and forced us to adapt.

We pivoted through several ideas and eventually built a pet-care AI assistant, monetized through e-commerce and recommendations, before the business was acquired in India. Building early applications made evaluation important to me. I went on to red-team models and work on AI safety research.

I suspect that if model capabilities continue to improve, evaluation may become less prominent a decade from now. That is a prediction, not a reason to wait: teams are putting systems into production today, and they need evidence that those systems actually deliver what they promise.

Start by defining the most ambitious version of success before designing the tests. Imagine the best outcome the agent could achieve and ask: what must happen, what must never happen, and when should the system stop? Those questions define the job and its boundaries before a score enters the picture.

Aim at the job you need, not the score that is easiest to obtain. My warning is that a perfect score can indicate an evaluation that is too easy or too narrow. As a system improves, challenge it with cases beyond its current capability rather than declaring victory because it passed a fixed set.

Test the whole workflow, not just its final answer. In one image-editing example, the request was only to change the background. The output had the requested cream background, but it also changed the object by adding a second handle. Checking only whether the background changed would miss the unintended alteration.

For a multi-step workflow, examine what enters each step, what comes out and what the expected result is. A final output can look superficially correct even when an intermediate action violates the task. The right unit of evaluation depends on the work the agent is actually performing.

One score is not an evaluation system. Persistent agents can write, call tools and make sequences of decisions, so use several forms of evidence. Put automated checks into the deployment pipeline and run the relevant evaluation questions repeatedly, rather than scoring a model once when choosing it.

Keep people in the loop to review whether results make sense. An AI reviewer can speed up parts of evaluation, but I would not rely on it as the sole reviewer. Add adversarial tests as well: deliberately place the workflow in difficult conditions and look for ways it can break.

Choose a test partly for the failures it can reveal. A static task with one answer does not necessarily show how an agent handles a sequence of decisions when circumstances change. I have been exploring games as one additional evaluation paradigm because an opponent can respond and force a system to adapt.

My open-source project eSlams lets agents and models play games through a Python package. Comparing an older and newer agent in chess, for example, is not a simple declaration that the winner is the better model. The moves, changes in strategy and recovery from mistakes can give us another view of their behavior.

The project supports different kinds of games and model providers. Chess and Go exercise planning with a visible board; poker introduces hidden information; cooperative games involve shared goals. I present these as complements to coding, writing and other evaluations, not as replacements for task-specific tests.

The eSlams website previews matches between models and also lets people play against them. The goal of the proposed AI games competition is to explore how models behave in interactive settings. A game result can be informative, but it cannot by itself establish that an agent is ready for a particular production job.

Keep testing what used to work. I used Google Assistant for reminders, and when my experience moved to Gemini I found that reminders stopped working in that setup. I use this as an example of the regression risk in a migration: a system can improve in many respects while losing a capability users already depended on.

Retain known tasks and previous failures in your regression suite, but also add fresh challenges. If a test has become easy for the system, improve it and push the frontier. Keeping both kinds of cases matters: new challenges should not cause you to forget the older behavior your users still need.

Turn real-world failures into future tests. Your customers are among the most important evaluators of the product, so create a route for their reports to enter the evaluation pipeline. Observability tools, a simple thumbs-up or thumbs-down, and other feedback channels can show where a workflow is failing.

Reproduce the reported problem as you would in ordinary software quality assurance. Work out where the failure happened—for example, in a tool call—and ask why the existing suite did not catch it. Review the case, add it to the pipeline and keep the feedback loop going as the system changes.

The central principle is to test the job, not just the answer. Define ambitious success, check the workflow continuously and design each test for the function of that step. If one part of the workflow calls tools, evaluate its tool use and accuracy rather than using an unrelated measure.

Gather evidence across different evaluation paradigms as agents evolve. Games are one avenue I am exploring; search performance is another example for products that need current information. No single arena covers every edge case, so keep looking for evidence that is relevant to the system you actually deploy.

Finally, evolve the tests and learn from reality. Harder cases help you see the next frontier, while regression cases and human review protect the capabilities people already use. Customer feedback shows whether the system is meeting its promise in production, which is the question the whole evaluation effort is meant to answer.