Approve the Application, Not the AI Model: Building Evaluation Gates for a Bank

I lead AI engineering solutions at Langfuse by ClickHouse. We helped a top-five bank design safety gates and evaluations for shipping AI applications to production.

Public model benchmarks measure useful technical capabilities, but they do not necessarily predict performance on a bank’s internal data and use cases. Financial institutions are particularly protective of their data and the expertise embedded in their workflows.

Consider an analyst assistant that digests earnings-call transcripts for investment bankers. Summarizing the transcript alone is not enough: the application needs the firm’s context and expertise, and its output must be reliably evaluated against the bank’s requirements.

The bank’s principle was that it certifies applications, not models. A model may support many applications, each with different risks and standards for approval.

The bank also had deployment constraints: systems should run in its own environment, access should be gated, and sensitive workflows should be air-gapped. It wanted control over data even when outside providers offered retention guarantees.

The team needed to prove three things: that the application worked as intended, met compliance requirements, and was usable enough for people to adopt. A change to a system prompt, model or tool should trigger a decision about whether the application was still ready for production.

The anchor is a golden data set: representative test cases with inputs, expected outputs and metadata. Many banks already have test data; the missing part can be making evaluation against it repeatable and operational.

Each run produces traces that record what the application did when evaluated against the data set. Those traces become evidence, and scores across selected dimensions determine whether a given configuration passes its gates.

One tested model scored 97% on groundedness: its answers drew on bank data. But its numerical accuracy was only 60%. A response can be grounded in the right documents and still make mistakes when calculating financial figures.

The answer was to improve the surrounding application, not merely look for a different model. The team gave the agent a calculation tool, allowing deterministic code to handle arithmetic where the model had struggled.

The illustrated workflow starts with the golden data set and runs through planning, retrieval, calculation and answer composition. The application is then evaluated against separate thresholds for accuracy, groundedness, compliance and tool use.

Passing on average is not enough. Each dimension can have a different minimum, and any required gate can prevent promotion to production. The bank’s rules define what must pass for that particular use case.

A test case pairs an input with an expected output and metadata. Building this private benchmark makes it possible to assess the agent’s answer, behavior and quality on work that resembles the organization’s own tasks.

A trace of the improved agent shows the calculator receiving an expression and returning its result. De Jesus reports that giving the model this tool raised numerical accuracy from 60% to roughly 95% in the example.

Scoring should combine approaches. Deterministic checks use code to produce the same result for the same input; the team also evaluates the path the agent took, since an apparently reasonable final answer can conceal missing or inappropriate steps.

An LLM judge can assess more qualitative aspects of an answer, but De Jesus cautions against relying on model judges alone. Together, these evaluations measured performance across dimensions and helped the application clear the bank’s 85% accuracy threshold.

The evidence also ties outcomes to configuration changes: prompts, models and tools can be associated with the traces and scores produced by each run. That makes it possible to see which change improved or degraded results.

Domain experts must help set the thresholds. A risk team, for example, can decide what level of performance is acceptable for an internal or customer-facing risk application; the right bar depends on the use case.

De Jesus shows evaluation runs over data sets of 150 and 1,500 questions. An application can pass on groundedness, completeness and tool use but fail compliance. For a bank, a compliance failure can shut the entire gate regardless of the average score.

Failures can be routed into an annotation queue for a domain expert. The reviewer examines the trace, identifies what happened and supplies feedback that the team can use to improve the agent.

When a prompt or other configuration changes, the process reruns the evaluations. Reviewed examples can become candidates for the golden data set, so the benchmark improves as the team learns from actual failures.

De Jesus says the process helped reduce application evaluation from weeks to minutes, including use cases where risk and compliance review can otherwise take months. He stresses that the industry is still developing and there is no universal template.

His closing advice is to build a benchmark on your own data, gate the application rather than the model, and let domain experts establish the relevant thresholds. The resulting controls make model swaps easier to evaluate against the same application-specific standards.