Steven Latré on the Economics of Running Your Own AI Models

I'm Steven Latré, and I lead AI at imec, a Belgian research organization better known for semiconductor R&D than for AI. We work with partners on next-generation chips, but our work extends from transistors to systems and workloads. My role is to connect AI workloads with their infrastructure requirements.

Our own experience with AI agents has gone through stages that may sound familiar. In 2024 and 2025, we focused on using the newest frontier models. As pricing evolved toward token-based usage, we increasingly routed tasks to an appropriate model instead of always choosing the largest one.

Model routing is a useful first optimization, but as token consumption grows, some organizations begin taking more control of the stack. We started with closed APIs, then optimized costs, and are now exploring hosted open-weight models and leased infrastructure, with owning infrastructure as a possible later step.

An API is relatively straightforward to consume. Owning more of the infrastructure expands the design space: teams must consider model choice and trust, coding harnesses, hardware architecture, the CPU-to-GPU balance and KV-cache optimization, rather than task accuracy alone.

Our team ran thousands of benchmark runs to understand the move from API pricing to infrastructure and total cost of ownership for agentic AI. We asked what a given task performance costs and which harness, inference engine and hardware configuration make sense.

We chose long-horizon coding tasks from SWE-bench Pro. Long tasks may be an early fit for dedicated infrastructure; the economics of highly interactive use can differ. We varied coding harnesses, API and open-weight models, model sizes, and hardware configurations that could support concurrent users.

For each setup, we looked at task resolution rate, latency and cost, while increasing both model and hardware complexity. These findings come from one benchmark and workload, rather than a general ranking of all AI deployments.

The models fell into broad performance groups. In our runs, smaller models such as Qwen and DeepSeek resolved roughly 35–40% of tasks. A GLM model performed around the Opus range, while Kimi K3 reached 86%. I noted that some benchmark material may have been in its training data; that possibility is not an established finding.

Adding hardware costs changes the comparison. Qwen and DeepSeek achieved roughly similar application performance in our tests, but their hardware requirements could produce a fourfold difference in GPU rental costs. Moving from GLM to Kimi K3 gave about a 20% improvement in accuracy for around a 15% increase in rented GPU cost in the configurations compared.

Cost does not follow model quality in a smooth line. It often changes in steps: a slightly different model can require a very different hardware architecture. When choosing open-weight models, evaluate the actual hardware constraints and costs together with task performance.

Concurrency introduces another question: how many users can share a machine while keeping latency acceptable? With an API, token latency and throughput are part of the service experience; with your own infrastructure, you have to manage those limits yourself.

Many systems are constrained by memory. Both model parameters and each user's KV cache must fit, and cache requirements grow with the number of users. We therefore tested comfortable concurrency for each model on its own suitable hardware configuration.

Larger models supported markedly fewer concurrent users, even with larger hardware, and our setups were slower than the Claude Code baseline. For the Kimi K3 configuration on these long-horizon tasks, we found roughly four reliably concurrent users and latency about three times slower than that baseline.

We next compared API use with GPU rental and purchase from a total-cost-of-ownership perspective. Other considerations can also motivate a move to owned infrastructure; the thresholds I describe concern this particular economic analysis.

In our lower-end systems, moving away from an API required about 89% GPU utilization to break even: the GPU would need to be busy almost all the time. For a larger system such as eight B200 GPUs, the corresponding utilization was about 15%. The comparison depends on the workload and assumptions behind it.

For a GLM-class coding-assistant workload, our study identified a point where moving from API usage to infrastructure became attractive and estimated about 25% cost savings. This is an estimate for the evaluated application and setup, not a guarantee for every organization.

We also varied the coding harness. At first glance, task performance across the three harnesses did not change much for a given model. That might suggest the harness has little impact, but GPU wall-clock time becomes important when you pay for your own infrastructure.

Different harnesses consumed very different numbers of tokens. In the GLM case, Codex generated about half as many tokens and used about half the GPU wall time as the other harnesses in our comparison. Similar task performance therefore did not mean a similar GPU bill.

This is an initial study of one application. Real enterprises mix many workloads, which makes the decision more complex. We have not fully unpacked quantization or optimizations such as speculative decoding, and models and hardware are changing continuously.

We are expanding the benchmarking pipeline into a public data set to track that moving target. As a nonprofit research organization, imec is also forming an alliance to bring software practitioners together with chip and infrastructure providers, because the two sides need to work more closely.