Why Test-Time Compute Could Reshape AI Inference

Test-time compute may be one of the most consequential developments in AI since the transformer. Instead of always producing the first answer immediately, a system can spend more computation when responding: generating candidate answers, evaluating them, or working through a problem in steps. I think the market for inference could eventually be larger than the market for pretraining, though that is a prediction rather than an established result.

A Google DeepMind paper, [Scaling LLM Test-Time Compute Optimally](https://arxiv.org/abs/2408.03314), asks how much a language model can improve on a difficult prompt when given a fixed but meaningful inference-time budget. Its experiments show that how that budget is allocated matters, and in some settings spending it on a smaller model can be more effective than simply scaling model parameters. OpenAI's o1 and the o3 preview helped draw attention to this approach.

People often spend longer on difficult questions, considering alternatives before committing to an answer. That analogy helps explain why additional computation at test time can be useful for language models, though the mechanisms should not be confused with human thought. It also suggests possibilities for reasoning and agent tasks.

One method is best-of-N sampling: generate several different candidate responses and select one using a learned verifier or reward model. The selection mechanism matters because a system has to identify which candidate is best; generating more possibilities alone does not guarantee a correct result.

Outcome reward models assess whether the final answer is right or wrong. Process reward models instead assess intermediate steps. If the first four steps of a six-step solution are promising but the last two are wrong, a process-based approach can preserve the useful work and search for a better continuation rather than treating the whole attempt as equally bad.

There are several ways to search using these evaluations. Best-of-N generates complete answers and selects among them. Beam search expands possible next steps, scores them and retains the most promising partial paths before repeating the process. Lookahead search modifies the evaluation of each step by using rollouts into possible later steps, then returning to choose among the alternatives.

OpenAI's o1 showed why inference-time scaling attracted so much attention; the o3 preview pushed the idea further. ARC-AGI asks models to solve unfamiliar visual pattern tasks that many people can understand but earlier AI systems found hard. Its results illustrate both the potential of extra computation and the expense of using it.

On the semi-private ARC-AGI-1 evaluation, [ARC Prize reported](https://arcprize.org/blog/oai-o3-pub-breakthrough) 75.7% for the low-compute o3 preview and 87.5% for a high-compute configuration using roughly 172 times as much compute. I estimated the latter cost hundreds of thousands of dollars; cost estimates depend on pricing assumptions. The result is for this particular benchmark and preview system, not proof that extra computation always improves every task.

AMD CEO Lisa Su has argued that inference will become a larger market than training. In a clip, she describes AMD's MI300 as designed for large language model inference and says companies will move beyond deciding how to train their models toward running them. Her view is a forecast from a company selling hardware for that market.

At CES 2025, NVIDIA CEO Jensen Huang described three ways to scale AI: pretraining with data, model size and computation; post-training using techniques such as reinforcement learning and human feedback; and test-time scaling. The last concerns how much computation the system allocates when answering rather than simply changing its model parameters.

Huang's examples include breaking a problem into steps, generating several ideas and evaluating which is best. He also presents the need for more computation as a source of demand for NVIDIA's hardware. Both AMD and NVIDIA therefore have commercial interests in the inference market they are discussing.

Groq CEO Jonathan Ross also expects inference to become much bigger than pretraining. I am an investor in Groq, which focuses on inference chips, and I agree that the opportunity could be large. Longer computations can be expensive today, but investment in chips and other innovations may reduce the cost of producing a given amount of inference.

Would lower costs shrink the market? Ross invokes Jevons paradox: more efficient steam engines made more uses of coal economically viable, potentially increasing total consumption despite less coal being needed for each activity. Similarly, cheaper AI computation could make applications profitable that were previously too expensive.

Ross argues that compute has historically become much cheaper while total use and spending have increased. He predicts a thousandfold reduction in generative AI compute cost over the next decade and a hundredfold increase in spending. These are his broad historical claim and forecast, not guaranteed outcomes: whether aggregate spending rises depends on how strongly demand responds to lower costs.

The idea is not limited to language models. A January 2025 Google DeepMind paper, [Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps](https://arxiv.org/abs/2501.09732), investigates extra inference computation for diffusion models, including image generation. This extends the discussion of test-time allocation to a different kind of generative system.

Diffusion models usually begin with noise and progressively denoise it. Increasing the number of denoising steps can improve sample quality, but gains tend to level off. The paper explores ways to use extra computation beyond simply adding more denoising steps.

The proposed framework combines verifiers, which evaluate generated samples, with search algorithms that use verifier feedback to find better candidates. The authors report improved generation performance in their experiments; the outcome depends on the verifier and search method, rather than showing that image models literally think like people. For anyone deciding where to follow new AI research, I think inference-time methods deserve close attention.