Rayan Krishnan on Model Evaluation Beyond the Model Card

I am Rayan Krishnan, cofounder and CEO of Vals AI. I want to look beyond the model card and explain how evaluations measure the jagged frontier, with a focus on finance. Vals builds benchmarks used in model releases, and its index combines results across domains such as finance, law, coding and healthcare.

A single number gives a general impression of capability but conceals variation. Across individual benchmarks, different models can lead on different tasks. When we build custom evaluations for particular code repositories, model rankings become more fragmented across languages and libraries.

The model is only part of the system. Different agent harnesses, as well as cost and latency requirements, change what matters. A high-level model-card result is therefore not necessarily the best guide to the model a team should use in its own product.

I also discuss the gap between adoption and measured return. The figures I present show enterprise AI spending rising while relatively few respondents report established ROI. Teams that track their spending are better positioned to understand returns; difficulty measuring cost against benefit can also block broader adoption.

Consider a tax benchmark comparison from Vals. The displayed token rates for Claude Opus 5.5 are $4 input and $20 output, compared with $10 and $50 for Claude Fable 5.1. Yet on this task Opus averaged $15.09 versus $13.18 and scored 45.60% versus 49.22%, according to our slide.

I attribute the higher total cost to the model using more tokens to complete the work. The point is not that one model is always cheaper or better: the task, harness and measurement conditions matter. Looking only at token rates can lead a team to pay more for a weaker result on a particular workload.

Finance offers several examples, including research, Excel modeling and tax tasks. Rankings vary even within those categories. A cost–performance chart for our Excel benchmark compares the overall task set with its leveraged-buyout subset, and the latter has a lower performance frontier.

The frontier also changes when we isolate that subset: a model that looks attractive on the overall Excel benchmark can fall off the cost–accuracy frontier for leveraged-buyout work. A firm doing a specific kind of financial model should evaluate the subset that matches its own work.

Evaluation design starts with the metric. An average partial-credit score asks what fraction of available points a response tends to earn. The share of fully correct answers asks how often the entire result meets the criteria. A draft that is useful to refine may score well on the first measure while being unsuitable to send directly to a client.

We also look for validity, reliability and discriminability. Validity asks whether the tasks and rubrics reflect real work and its standard of excellence. Reliability needs enough tasks and samples for repeated runs to produce consistent results. Discriminability means the benchmark can distinguish models and still has room to measure future progress.

A recurring problem is that agents can solve the test without solving the user’s problem. In a sandbox they may try to call other models or tools; a finance model might give several hedged answers to satisfy a rubric when a user needs one actionable answer. Evaluations should reward the actual desired output and account for the available tools.

Running evaluations is also an infrastructure challenge. Long-running agents need a consistent environment and a way to resume after interruption. Vals reports running more than six trillion evaluation tokens and emphasizes parallel execution, but smaller teams can encounter the same reliability issues at a different scale.

Many benchmarks still use one prompt and one response, whereas work often takes several turns. A first specification may produce questions, and a completed draft may receive further feedback. Evaluations need to cover that iteration rather than assuming the first response is the end of the job.

For an internal case study, our team gave roughly 10 to 12 engineers access to tools with free usage and observed about $1.5 million worth of tokens in one month. We saw strong output, but wanted to retain it at a much lower cost. These figures are Vals’s account of its own experiment.

We built an evaluation from repository history, now offered as Vals Smith. A merged pull request represents an issue and an accepted human solution; by replaying earlier tasks, we can compare whether agents can do equivalent work and see which models are unnecessarily expensive for particular tasks.

In one internal comparison, the slide reports GPT-5.6 Sol at 74.32% accuracy and $2.51 per test versus Kimi K3 at 72.97% and $11.73, with lower latency for the former. This is a result from our setup, not proof that hosted closed models will always be cheaper or faster than open models.

Those measurements inform which tools we procure, their usage limits, and recommendations about which model to use for a given issue. I say our spending fell by about 90% while maintaining output. That is a company-reported outcome of this particular internal workflow.

The broader question is where an organization stores its history of work and standards of excellence. A law firm, for example, may have relevant material in client emails, SharePoint or data stores. Those records can help build evaluations that reflect the company’s own tasks.

Once constructed, evaluations can help right-size usage, negotiate pricing, route tasks and assess fine-tuning against an external evaluation set. My closing argument is that evaluations can make institutional knowledge and standards more explicit, helping organizations apply AI systematically.