What Drives AI Agent Costs: Cache Hits, Input Tokens and Turns

ZUSAMMENFASSUNG

In 30 Sekunden

George Cameron of Artificial Analysis explains why the price of a model call is only one part of agent economics. In one AA-Briefcase benchmark configuration, repeated calls and large inputs produced a roughly $21 task cost with caching; he estimates that no cache hits would push it above $200. He recommends monitoring input volume, cache-hit rates and turns, then evaluating the quality-versus-cost trade-off on the organization’s own tasks. The figures describe the benchmark and prices shown at the time of the talk.

1Track capability and cost over timeCameron reports a rapid historical decline in the cost of reaching fixed benchmark thresholds.
2A $21 agent taskThe AA-Briefcase example uses about 165 model calls and substantial repeated input.
3Caching changes the billHe estimates that zero cache hits would push this example above $200 per task.
4Effort changes the trajectoryReasoning settings affect turns and input volume as well as reasoning tokens.
5Match capability to the jobA bounded task may gain nothing from a more expensive model once it meets the required quality.

Das Wichtigste

Long agent trajectories can multiply a modest per-call cost across many turns.
Input tokens, cache-hit rate and number of turns are major cost drivers in the example.
Cameron estimates the illustrated task would exceed $200 without caching versus about $21 with it.
Reasoning effort can change both the number of turns and later input volume.
Choose models against task-specific quality requirements and budgets, then benchmark them on your own work.

ThemenAI cost & performance Inference KV caching & prefix reuse Evals Agent Engineering

VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Transkript zeigen

Nur das Transkript

Benchmarking the AI stack

Cameron explains the Intelligence Index and why individual evaluations matter for model selection.

I am George Cameron, co-founder of Artificial Analysis. We benchmark agents, language models, media models, hardware and inference providers because choosing an AI system involves trade-offs across the stack: capability, cost, speed and caching.

Before optimizing cost, the team uses an Intelligence Index that combines ten evaluations across knowledge work, scientific reasoning, coding, terminal use and other capabilities. Cameron describes it as a decision aid for current and near-term applications, not a measure of progress toward AGI.

An overall score does not replace task-specific results. Artificial Analysis also publishes individual evaluations, such as a finance and accounting agent benchmark, because a model’s strength on a particular workload may differ from its general ranking.

The changing cost landscape

He compares model costs and tracks the historical cost of reaching fixed capability thresholds.

Cameron says the pace of model releases makes the landscape difficult to follow. In his analysis, open-weight models have remained roughly three to nine months behind leading proprietary models and can offer attractive cost characteristics.

Costs vary by orders of magnitude. In the examples presented at the talk, some models cost several dollars per benchmark task while cheaper models cost cents. The cheaper models are less capable on the general index, but may still be sufficient for some jobs.

Artificial Analysis groups models by capability level and tracks the cheapest way to reach each threshold. Cameron reports that in the observed data, the cost of reaching a given index level fell by around an order of magnitude in roughly a year. This is a historical observation, not a guarantee of future pricing.

FOLIE 1 / 4Cost at fixed capability thresholdsThe chart tracks historical per-task cost for different Intelligence Index thresholds.

Breaking down an agent task

Repeated calls, input tokens and cache hits drive the illustrated task cost. Reasoning effort also changes the trajectory.

He breaks down an agent task from the AA-Briefcase benchmark, which asks agents to work with thousands of files and produce knowledge-work outputs. In the illustrated configuration, Claude Opus 5.5 at maximum effort averages about $21 per task.

FOLIE 2 / 4The cost of an agent taskThe breakdown shows how repeated calls, input tokens and prompt caching contribute to the illustrated $21 task.

The cost of one model call includes answer tokens, reasoning tokens and input tokens. The task uses about 165 calls, so a small per-call cost compounds across the trajectory. Long agent histories make input tokens particularly consequential.

Prompt caching changes the calculation: in the example, most input tokens are read from cache at a discounted rate. Cameron identifies input volume, the fraction of cache hits and the number of turns as key variables to monitor.

He estimates that if caching were not working at all for this example, the same task would cost over $200 rather than about $21. That comparison is specific to the benchmark and pricing assumptions; it illustrates why cache behavior matters when choosing an inference provider.

Reasoning effort affects more than reasoning-token count. It can change the number of turns and the size of later inputs as the conversation history grows. In the displayed benchmark, the same model costs roughly $1 per task at low effort and $21 at maximum effort.

FOLIE 3 / 4Reasoning effort and task costThe same model’s turns, input volume and cost differ across effort settings.

Comparing a leading model with a cheaper open-weight model, Cameron emphasizes that multiple drivers interact: input prices, cache-hit percentage, token consumption and turns. For long-horizon tasks, input costs and caching can dominate.

Choosing a model for the task

Cameron uses a quality-cost frontier and distinguishes bounded from open-ended tasks. He recommends benchmarking shortlisted models on actual workloads.

The difficult decision is not recognizing that a cheaper model costs less; it is understanding the quality and cost trade-off for a specific task. Cameron suggests starting with a Pareto frontier of general capability versus cost per task, then testing candidates.

FOLIE 4 / 4Capability versus costThe Pareto frontier illustrates the trade-off used to shortlist models.

He separates tasks into two conceptual categories. For open-ended analysis, more capability may continue to improve the result. For a bounded question, such as a correctly configured assistant totaling last month’s subscriptions, a cheaper model may return the same useful answer.

For a bounded task, ask what capability level is sufficient and find the least expensive way to reach it. For an open-ended task, ask how much additional capability is worth within the budget. Both choices depend on the actual workflow.

His suggested workflow is to understand the available metrics, compare models against priorities and create a shortlist. For production, he recommends building a benchmark on the organization’s own tasks rather than relying on a general index alone.

Cameron closes by describing Artificial Analysis’s Optima product, which helps generate task-specific benchmarks and quality-versus-cost curves. The purpose is to measure the trade-off in the user’s own context.