AI Cost per Completed Task: Models, Caching and Evaluation

At OpenAI DevDay 2026, three OpenAI presenters discuss measuring and reducing AI application costs while preserving task quality. They speak from the provider’s perspective; customer results are examples reported by OpenAI, not guarantees.

Mandeep argues that price per token alone can mislead. A cheaper model might use more tokens, fail more often or require human intervention, so he recommends measuring cost per successfully completed task, including human work.

The panel recommends defining the task and an acceptable quality threshold, then testing model, reasoning and harness configurations on representative examples. A cost–quality frontier helps identify configurations that meet the threshold efficiently.

They suggest trying a smaller model on simpler tasks and using more capable models where evaluations justify the cost. Their examples of customer model performance are specific to those workloads and should be checked against one’s own evaluation set.

Sapto introduces four levers: prompt caching, programmatic tool calling, reasoning effort, and batch or flexible processing. The goal is a successful result at acceptable cost and latency, not simply fewer tokens.

Prompt caching reuses processing for matching prompt prefixes while the model still handles new input and generates a fresh response. Keeping stable instructions at the beginning can improve reuse.

Programmatic tool calling can combine and filter intermediate tool results in code before returning a smaller report to the model. The panel reports a customer example with fewer input tokens and no measured quality loss; results vary by task.

Reasoning effort is a setting to evaluate separately from answer length. The panel suggests starting low for simple routing and increasing it when needed, then checking the result with evaluations.

Batch processing trades immediacy for lower pricing in the offering described at the event. Pricing, service terms and available controls can change, so users should check current documentation for their workload.

The panel returns to cache optimization, discussing stable prefixes, explicit boundaries in supported models and diagnostics for misses. It recommends changing one setting at a time and rerunning evaluations as models and applications evolve.

The closing recommendation is to define a representative evaluation set, compare configurations at the desired quality threshold, and keep iterating as the application changes.