Diffusion Will Win Inference: Stefano Ermon Bets Parallel Tokens Beat Custom Silicon

In 2024 Stefano Ermon’s Stanford lab published a result that still sits at the center of his company. Same transformer architecture, same data, same parameter count: train it as a diffusion model instead of next-token prediction, and at GPT-2 scale — under a billion parameters — you match the autoregressive model’s perplexity. The only difference that mattered commercially was speed. Because diffusion emits many tokens at once, they generated text about ten times faster.

Two years later that academic win has production customers. Sarah Guo hosts him on No Priors as co-founder and CEO of Inception, a company of about fifty people after two years, built to scale diffusion-based language models. One voice-agent customer had been running its LLM on Cerebras custom chips purely to crush latency; it switched to a diffusion model on ordinary NVIDIA GPUs. The episode title is the thesis: why diffusion will win AI inference.

Ermon’s résumé is short and load-bearing. He joined Stanford as an assistant professor in 2014 and became one of the people who set the modern diffusion path in 2019 with PhD student Yang Song. Stable Diffusion and Midjourney sit downstream of that lab. The question the whole conversation answers is why a fifty-person company would bet against every frontier lab’s autoregressive stack.

When Ermon started at Stanford, generative modeling was not hot. Models lived on MNIST; a grainy digit counted as a big success. Papers were hard to publish. Grant language had to pretend you were learning features from unlabeled data to help supervised learning, because that was what everyone cared about. He stuck with it for a simple reason: a generative model is the right way to make sure you understand structure in data.

The application in his head was a world model, not today’s chatbots. Stand up and walk out the door, and you can picture what will happen; decisions and model-predictive control need that generative capacity. He thought the problem was hard enough to occupy a career. He jokes that he was wrong — the field moved much faster than he expected.

On images he walked the usual path: autoregressive models that were slow and blurry, then VAEs, then GANs taking over. GANs produced good pictures and were unstable to train and hard to reproduce. In 2019 he and Yang Song tried something more principled: train a network to denoise images. If a network can restore a noisy picture, it understands enough structure to build a generative procedure from the denoiser — start from pure noise, refine step by step, finish with a clean image.

That procedure is the base of modern diffusion. The best models for image generation, video, music to some extent, and a lot of protein work still sit on it. Once images worked, Ermon started asking how to get the same idea onto text and code — discrete objects, more parallel, with built-in error correction. The 2024 paper was the breakthrough that made him start Inception to see what happens at commercial scale.

Guo’s hard question is the obvious one: diffusion already rules continuous media, but text and code are discrete, and every big lab is on autoregressive transformers. Why is this a bet? Ermon rewinds to 2017. RNNs had to process tokens one at a time, so training would not scale. Transformers let you process many tokens in parallel. That is what unlocked LLMs. The revolution, he says, happened on the training side only.

At inference, autoregressive models are still sequential. You cannot emit the tenth token until the first nine exist. That workload does not map to GPUs. It is extremely memory-bound: most of the time is spent shuttling weights through the memory hierarchy, with very little arithmetic. A GPU is built for pipelined matrix multiplies; autoregressive decode asks it to finish one part, then fetch the next.

Diffusion inference looks like training: many tokens at once, parallel matmuls, the thing GPUs actually do well. He pushes the analogy all the way. RNNs are to transformers as autoregressive models are to diffusion. Transformers won training parallelism; the next winner has to win inference parallelism. As models go multimodal, nobody knows whether a unified architecture will be autoregressive or diffusion. The jury, he says, is still out. Inception is betting diffusion because inference-time scaling is what will matter.

The economic argument points the same way. What dominates is intelligence per watt and intelligence per dollar. Reasoning models are mostly scaling test-time compute, so an architecture that scales better on that axis wins. RL post-training is bottlenecked on generating rollouts — let the model explore, score trajectories, improve. Faster inference automatically buys better RL scaling. He closes with the bitter lesson: the more parallel solution is the one that eventually wins.

Moving denoising onto discrete data was new science, not a port. Two pixel colors interpolate; two words do not. It took years of Stanford work to get to the 2024 result. As a company, Inception now claims Mercury models are on par with Claude Haiku, Gemini Flash, and OpenAI’s mini and nano class on benchmarks, while being significantly faster, and they are serving them in production. Model size and FLOPs are a trade secret. They are still pushing the frontier.

The unglamorous blocker is serving. You cannot run these models on vLLM or SGLang. The team built its own engine so it could handle real production complexity. The API is the other way: OpenAI-compatible, text in, text out, so customers keep their existing stack. The models follow instructions and emit structured JSON well enough — in the voice customer’s case, better than what they had been using.

Autoregressive generation of one token is basically the whole game. Diffusion has a lot of room to trade compute for quality at inference, the way image and video diffusion already do with distillation and higher-order differential-equation solvers. Figuring out how those tricks move to text is one reason Ermon thinks the ceiling is higher. The company stays scrappy: the backbone is still a transformer with attention, public datasets and evals stay in use, effort goes where they can differentiate. Right now that is speed. He does not believe anyone has found the best way to build these systems. There have to be alternatives.

Where does speed already win in September 2026? Anywhere latency matters, Ermon says — and more generally, if quality is the same, people always pick faster. Frontier labs’ own fast SKUs already command higher prices. Once you get used to a fast model, it is hard to go back. It is like broadband.

The public example is OpenCall, which builds voice agents. The pipeline is ASR, then an LLM that does tool calls and decides what to say next — typically a reasoning model for quality — then text-to-speech. Every hop eats latency. They had been serving the LLM on Cerebras to hit the speed the product needed. They switched to Inception’s diffusion LLMs and, Ermon says, got the same speed you would get running an autoregressive model on custom hardware, on NVIDIA GPUs: more availability, lower cost, higher quality. GPUs are scarce. There are still more of them than custom chips.

Guo asks how this interacts with a hardware landscape full of expensive high-SRAM parts. Hardware is one way to go fast, Ermon says; software on existing silicon can be better, and the two are complementary — gains multiply. Someday someone may even build chips that fit diffusion better. Software going first means you are not picky about the metal.

How big is the latency-sensitive market? He walked OpenRouter’s taxonomy — research, conversation, coding, software engineering, log processing — and estimates 20 to 30 percent of workloads really care about latency. That is a lower bound for models that deliver the highest quality inside a latency budget. Diffusion LLMs are not at frontier intelligence yet, and many jobs still need that, so the near-term slice is this one. Guo adds the industry backdrop: the last few years of industrial research treated efficiency as second-order to capability. Compute and power are tightening, so one of the most important decisions at an AI company is how to split every watt between training and the most valuable use. Ermon agrees nobody really knows how an “AI factory” will run. Exploring another way to manufacture intelligence is what excites him.

Guo voices the David-and-Goliath fear: it is hard to invest in new architectures because any method breakthrough gets absorbed by labs with more compute. Ermon does not dismiss it. The first layer of moat is IP and trade secrets — the recipes researchers use to train diffusion language models. The second is engineering: without a serving engine you cannot deploy, even if you train the weights. Forcing the company to ship end-to-end from day one taught them how to run these models. The third is customer feedback, production failures, and evals built from real scenes, which are harder to copy than a paper.

Two quieter claims may matter more at scale. Controllability: an autoregressive model has to finish the object before you can score a constraint — generate a molecule, then measure solubility. Diffusion is coarse-to-fine, so early in the process you can tell whether the object is the kind you want and steer with an external reward or a set of constraints. Academic literature already suggests diffusion is easier to control, with steering methods autoregression simply does not have. The team is thinking about product interfaces around those capabilities. Data efficiency is the other: denoising is data augmentation, the same example seen through many noisy views. There is academic evidence diffusion uses data more thriftily than autoregression. If that holds at scale, the card looks very different on the day data is the bottleneck. Whether larger diffusion LLMs grow new capabilities, he says honestly, is unknown. Emergence does not announce itself. Speed was the wedge because it is measurable, testable, and obviously valuable.

Guo presses on messy data. Images, video, and voice have physical structure; a lot of code does not look like the “correct” structure you want. Ermon unrolls what a generative model actually does. Autoregressive or diffusion, you are learning a compression scheme. The more you compress, the more patterns you have found. That is the quiet miracle of next-word prediction, and it is the same job under denoising. The 2024 metric was perplexity: how much structure you identified. Numbers do not lie. If you drive perplexity as low as the autoregressive baseline, the structure was there and the model found it. What remains is inductive bias — transformer plus next-token versus a denoising frame — and he says we do not even have the tools yet to answer which is the better structure detector.

He does not hide the costs. The whole stack had to be built in-house: serving engine, kernels, SFT, RLHF, RL. There is no mature ecosystem. Open-source diffusion LLMs exist and are not particularly good. That makes it harder to get customers to try the product, harder to adopt, harder to do on-prem. They chose not to open-source everything in order to keep IP close. The downside is less community contribution and slower adoption. Two years out he still thinks they are not at frontier intelligence, so many workloads stay on traditional models; the addressable slice is the latency-bound 20 to 30 percent.

Guo asks why researchers should come to a fifty-person company instead of a frontier lab. Training scale is not something Ermon will share. What he will say is that the field is still small and there is a lot to invent. People join Inception for ownership and green field. The team splits three ways: a product group on the platform and customer success, a group serving the current best model, and a group training the next one — training, RL, and inference.

On recursive self-improvement he is calm. They use frontier-lab models heavily; iteration is much faster. He does not think they are “there yet.” Human ingenuity is still super important: coming up with the right ideas, pruning the space, spotting the promising direction. Often the bottleneck is compute, not people. Six months or a year from now may look different. Right now, judgment still sits with the researchers.

The 2024 paper was not a one-off. His lab’s earlier diffusion work beat GANs at academic scale, more stably, and that line became Stable Diffusion and Midjourney. He was a co-advisor on Flash Attention, now in almost every large-model serving stack. DPO started as a rotation project in his group, a clever bit of mathematical structure that gave a more efficient alignment path than RLHF, now used on both LLMs and diffusion models. Gems still come from clever insights, not only from FLOPs.

Academia never has enough compute. What it does have is students who are not scared to take a bet nobody else believes. That, Ermon says, is why it has remained so impactful even in this era: a lot of the important ideas have roots, or were created, in universities. The commercial company is how you find out whether the next one scales. The research culture is how you allow yourself to work on it before the result is obvious.