How NatWest Tests Customer-Facing AI Agents Before Launch

I am Jonathan Osborne, a lead AI engineer at NatWest Group. Moving an agent from prototype to a customer-facing service in regulated banking is less about creating the agent than proving it is safe.

NatWest has about 19 million retail customers. Cora, its digital assistant, handles around 100,000 conversations a day, most of them historically scripted. The team is working to bring retrieval-backed generated answers and action-taking agents to roughly a third of its journeys.

The scale changes the risk. A wrong answer is not merely a bug report: it can affect a real customer and their money, invite regulatory scrutiny and damage trust. The work described here is about earning approval to go live and learning to trust the measurements.

A customer asking whether they can overpay a mortgage may reach a dead end in a scripted menu. An AI assistant can use a knowledge base to answer and handle a change of topic, but an incorrect answer about that customer’s mortgage could have real consequences.

Osborne identifies four failure modes: confident hallucinations, prompt injection, accurate but unhelpful loops, and nondeterministic responses. The regression assertions developed for a bounded scripted system do not directly cover that wider space.

The first defense is to shrink the surface exposed to models. A routing funnel filters routine events, applies deterministic pre-classification rules, then sends remaining utterances through relevance and rephrasing, an embedding-based intent classifier and policy rules.

This design supports access control, auditability, resilience and governance. Decisions carry codes; rules still work during a model outage; and versioned configuration can change without a full redeployment. The team first asks which problems actually need a model.

A single routing accuracy figure cannot establish whether an answer is grounded, resolves the customer’s problem or knows when to hand off. NatWest instead treats quality as a vector: safety, task success and experience are scored and monitored independently.

Before automating evaluation, the team used humans to review 111 conversations across 37 payment journeys and four brands, with three independent scorers per conversation. The overall pass rate was 83%.

That aggregate concealed a 54-point spread: happy-path cases passed at 97%, edge cases at 76%, and topic changes at 43%. Two of the largest failure causes were engineering issues rather than model issues, and human validation exposed them.

Human agreement also varied. Reviewers agreed 95% of the time on bias and fairness, but only 64% on intent understanding. Osborne asks what it means to train an automated judge to match human reviewers when the reviewers themselves do not always agree.

For evidence about an LLM judge, he turns to a different, already-live internal complaints assistant. It summarizes complaints for handlers; roughly 14,000 cases had scores from both humans and an LLM judge.

On faithfulness, the judge’s mean was about 0.97 with little spread, while the human mean was 0.83 with a much wider spread. The slide shows a correlation of 0.04: a judge that rates nearly everything highly cannot reliably distinguish strong cases from weak ones.

Changing the judge model alone moved its score relative to the pass threshold. The judge is therefore part of the evaluation configuration and can introduce breaking changes. Osborne says using a different model family from the agent helped bring the scores closer.

For Cora, conversations and guardrails are traced using open standards. The team samples 5% at random and additionally selects handoffs, failures, long conversations and angry users. A judge scores the samples across the chosen dimensions.

The design has bank constraints: the application runs without direct internet access, reaches the judge over a private path, and prevents personally identifiable information from leaving the network. Traces are captured continuously, offline scoring runs every four hours, and results feed the monitoring system.

Prompt or context changes can have outsized effects. A hard CI/CD regression gate compares changes with a known-good baseline. Osborne calls continuous evaluation production infrastructure, not a notebook used only at launch.

Generative agents and the traditional intent classifier need different instruments. Prompt changes can cause generative regressions; the classifier can drift as customer language changes. The team watches feature drift and human-labelled F1 scores to decide when retraining is needed.

The rollout begins in shadow mode: the new system classifies real customer utterances while the existing scripted system still responds. That lets the team observe production inputs before allowing the new system to answer customers.

The bank then plans to progress brand by brand and by percentage of traffic, using go/no-go indicators. Some decisions are automated and others require manual approval. Feature flags allow a global rollback or isolation of a backend or brand within seconds.

In the complaints project, feedback is a gate: a handler who wants to rerun an investigation must first explain why. That has produced a valuable set of roughly 14,000 colleague labels, which the projects can use to improve evaluation.

Osborne’s closing lesson is to compare distributions and correlation, not just averages, when calibrating judges. The evaluation system can be reused across agents. At the time of the talk, Cora’s new capability had not yet gone live; its launch was expected the following month.