Build an Agent Evaluation That Checks Every Guest’s Needs

Annie Wang shows how to define success for an AI agent, choose an evaluation approach, build and test a custom evaluator, and run it during development and deployment. The example is an agent that recommends a restaurant for a group.

A highly rated nearby restaurant is not necessarily a successful recommendation. Some guests may be vegetarian, have allergies, face a budget limit or need to leave early. The actual requirement is a dinner plan that works for everyone.

The same request can produce different plans because language-model responses vary between runs. Both should pass if they satisfy the group. If an agent has tools to inspect menus or book a table, evaluation can also examine whether it checked dietary needs before booking.

The sequence of actions is the agent’s trajectory. In a multi-turn conversation, an evaluation may also need to check whether the agent remembers constraints introduced earlier. The first choice is whether to assess the final plan, the steps taken to produce it, or both.

Google’s Agent Development Kit (ADK) provides built-in metrics. A response can be compared with a checked reference by wording or by a language model’s assessment of meaning; tool calls can be compared with expected calls, their arguments and an adjustable order requirement.

These reference-based checks are comparators. A golden dataset pairs test requests with trusted answers, but a restaurant plan that works for one group may fail another, and even a single group may have several acceptable options.

A checker instead tests explicit requirements against a proposed plan. For the dinner agent, it can use the group’s needs and restaurant information to check menu, price and timing. Any plan meeting the requirements can pass without appearing in a list of approved answers.

The choice of what to compare against is separate from how to implement the check. A deterministic rule can fail a plan when a vegetarian guest has no suitable dish and return the same result for the same facts. A language model can instead judge the plan against a rubric and the supplied restaurant facts.

Simple structured yes-or-no facts are straightforward to check in code. A model judge may help with requirements requiring interpretation, such as clarity, but its judgments should also be tested against human-reviewed examples. ADK includes rubric-based judges for responses, tool use and multi-turn conversations.

For the demo, Wang uses Python because the guest and restaurant facts are structured. The agent receives fixed information up front and proposes one restaurant and time. Checks cover each person’s applicable dietary needs, allergies, budget, distance, spice preference and timing.

A plan counts as successful for an individual only when all that person’s requirements are met. The custom metric, everyone_ate, divides the number of people satisfied by group size: three of six gives 0.5; all six gives 1. The passing threshold is 1, and the evaluator records failure reasons.

ADK calls the custom Python metric with the recorded interaction and evaluation settings. The function reads a group ID, retrieves the corresponding requirements and parses the agent’s JSON proposal for restaurant and time.

Before scoring, the evaluator checks that both fields are present and valid. An unknown restaurant or missing time fails instead of allowing the evaluator to invent part of the agent’s plan. The function runs the checks, compares the score with the threshold, and returns overall and per-turn evaluation results; individual explanations go into a detail file.

An evaluation configuration JSON selects everyone_ate, sets its threshold and identifies the Python function. Before using the evaluator on an agent, Wang tests known answers: a three-of-six plan must score 0.5 and fail, two different valid plans must each score 1 and pass, and an incomplete answer must fail.

The adk eval command takes the agent, evaluation configuration and an evaluation-set JSON containing test requests. ADK runs the selected request and passes the recorded interaction to the checker.

In the deliberately flawed demo, the agent prioritizes ratings and chooses the highest-rated restaurant. The detail file identifies vegetarian, allergy and timing problems; the plan satisfies three of six people, scores 0.5 and fails the threshold of 1.

Wang recommends investigating why the agent favored ratings, changing its instructions or code, and rerunning the same case with the same success criteria to compare versions. Repeat important cases to assess consistency, add groups with different combinations of needs, and keep some cases separate for checking generalization.

Save newly discovered failures as future test cases. In the adk web browser interface, register the custom metric, select the dinner case and everyone_ate from the Eval tab, set the threshold to 1, and inspect both the result and the evaluated conversation.

A pytest test can reuse the evaluation-set JSON for the cases and the configuration JSON for scoring, including custom metric registration. It loads the full settings and calls ADK’s AgentEvaluator from Python; pytest then reports whether the evaluation passed.

The same pytest command can be a required CI/CD check before deployment, so a failing evaluation blocks that version. Wang’s four steps are to define success, select what and how to check, validate the evaluator against known answers, and run it repeatedly as the agent changes. She adds that such checks can later guide agent self-improvement.