Inference Costs Fell 100x. Tri Dao Says There's Another 10x — and NVIDIA's 90% Won't Hold

Jacob Effron has wanted Tri Dao on Unsupervised Learning since FlashAttention first made the rounds. Dao is chief scientist at Together AI, a Princeton professor, lead author of FlashAttention, and a co-creator of Mamba with Albert Gu. Effron starts with a classic VC question: after NVIDIA's run, will competitors show up on chips and packaged GPUs for the rest of the industry that does not own a TPU or Trainium?

NVIDIA's lead, as Dao tells it, is two stacked advantages: they design very good chips and they build very good software, and people build on top of that. Competitors are entering. AMD has been here for a while. As architectures coalesce around transformers, designing chips for a known workload gets easier. On inference, AMD's larger memory is already an advantage, and people are picking it up. Training is harder. Networking is a main bottleneck, and NVIDIA is ahead there. The challenges of building good training chips and good inference chips are now understood, he says. It comes down to execution. Right now he would put about 90% of workloads on NVIDIA. In the next couple of years, some of that work will become multi-silicon.

The day before the taping, SemiAnalysis tweeted, in essence, that a huge part of NVIDIA's moat is that Dao keeps doing this work in NVIDIA's ecosystem — so why wouldn't AMD or Google just pay him to do the same on their silicon? Effron hears an echo of the frontier-lab bidding war for people who can train a model from scratch. Dao already works with people at NVIDIA, AMD, Google, and Amazon. He spends most of his hours on NVIDIA for a boring reason: those chips are good, the software is good, and that combination lets him do interesting things. He and AMD have already integrated an AMD version of FlashAttention into the public repo. What he is thinking about now is bigger than a single vendor: abstractions that work across GPUs and accelerators, so new hires at Together can get productive and so that AI itself can take some of the kernel work.

Chip design is a two- or three-year bet on future training and inference. From a high level, Dao says, architecture has stabilized around the transformer. Look closer and a lot is still moving. Mixture of experts made models much larger in parameters and much sparser in compute: more memory, less uniform math. A chip designed for dense, even work suddenly has a sparsity problem. Attention is more than ten years old, and the details still bite. DeepSeek's multi-head latent attention uses a very large head dimension; if your systolic matrix-multiply engine is a certain size, it may not even fit.

Workloads are spreading too. The "traditional" chatbot is about three years old. Coding products such as Cursor and Windsurf, and agents that call tools, run a Python interpreter, and search the web, are different. A chip obsessed with running the model as fast as possible can neglect how it talks to the host to do a web search. High-level stability with a lot going on underneath means a constant race: how fast can you execute and adapt to the new workload.

On a two- to three-year view, he expects inference to diversify first. Cerebras, Groq, and SambaNova are already a serious challenge on very low latency, and some customers will pay a premium for that. Others want huge-batch, high-throughput jobs: massive data processing, synthetic data, RL training that rolls out as many trajectories as possible. Video generation wants yet another mix of compute and memory. If you are a startup, Dao's advice is not to optimize for the general case — incumbents will out-execute you. Place an outsized bet. Maybe chatbots fade and video, world models, or robotics become half the load. Design for that future and hope you are right. That, he says, is the role of a startup.

Are there abstractions that let you work across chips? Some, Dao says, and they come with the usual trade-off. Triton is really good: it supports NVIDIA, AMD, and Intel GPUs, with a shared front end and vendor backends. Meta is betting on it; PyTorch compile generates Triton and lets Triton lower to the device. Give up 5% performance and become 3x more productive and that is an acceptable trade. Give up too much in a competitive inference market and people will go lower.

Designing for humans is hard. "Hardware portability is kind of a myth." Even NVIDIA chips change a lot generation to generation. That is how they get more performance. A CPU might get 5 to 10% faster year to year, and old code still runs. NVIDIA essentially rewrites the software every two years, because more FLOPs come from more specialized components, lower precision, and new ways to synchronize pieces of the chip. Portability is not even there between generations of the same manufacturer. Abstractions would be valuable if they only smoothed that hop.

The field is crowded: Triton's newer, lower-level Gluon; Modular's Mojo, which Dao thinks has some of the right abstractions and will be judged, unfairly but inevitably, on how fast it is on NVIDIA; Stanford's ThunderKittens; TileLang; Google's Mosaic GPU. Training engineers to write performant kernels is painful, so people are iterating on DSLs quickly. In one or two years it should be clearer. Abstractions will come from two angles. From the ML side: inference is memory-bound, so you want primitives for moving memory and doing matrix multiplies fast. From the hardware side: NVIDIA's matrix engines are so fast that everything else is slow, chips are more asynchronous, and the abstraction has to express pipelines and synchronization. He is also asking how to design these languages for language models, which do not operate like humans.

Models have only recently become useful for this work, and it surprised him. Fully automatic GPU-kernel writing — describe the problem, get the kernel — is where simple Python, data analysis, and web front ends already are. For GPUs, we are very early. Models can write simple elementwise kernels, reductions, summing, normalization. A little more complication and they emit incorrect code. Training data is the bottleneck. Scrape the internet for kernel code and you get class projects and documentation meant for GPUs three generations ago, describing things you should not do now.

The path is a small amount of expert data, then synthetic data hooked to a compiler and a profiler. He does not think the expert code is mostly private; the hard part is the workflow. The GPU MODE Discord took the PyTorch compiler that lowers PyTorch to Triton and generated about 15,000 program pairs. You have to be creative because there is not that much data on the internet. Fully automatic generation, he thinks, can be solved in a year or two. It is still a difficult problem.

Working alongside humans is further along. Two milestones: o3 (and GPT-5) for high-level questions like how to optimize a function and what to watch; and Claude Code, which turned out pretty decent at writing Triton kernels. Dao loves the design work — what architecture takes advantage of the hardware — and finds implementation heavy. Claude Code took enough of that load that he is maybe 1.5x more productive. Effron is visibly surprised. The next milestone is more agentic models that know when they do not know: look up an API, read a compiler error, query a profiler. That, he says, is the vague but crucial test for new releases. Benchmarks for it are just starting.

Since ChatGPT debuted, Dao puts the drop in inference cost at maybe 100x, at least, and you can see it in API prices. Half of that is modeling: much better models for the same parameter count, from more data and better architecture, including more efficient attention. The other half is inference optimization. Early on the field did not understand the bottlenecks. Then people realized it is a lot about data movement: weights in and out of memory, and the KV cache that stores the entire history so attention can predict the next token. Most of the tricks reduce that movement.

Quantization was a sharp cut. Two or three years ago, 16 bits per parameter was typical. Eight bits is now common; new models use four; there is work at one or two bits. You often lose essentially no quality, which still requires sophisticated methods. OpenAI's gpt-oss release put most layers in 4-bit. The model is about 120 billion parameters and fits in something like 60 GB, which translates into really good inference. The other theme is co-design. FlashAttention itself was people who knew the hardware talking to people who knew the algorithm, seeing that memory access was the bottleneck, and rewriting attention to reduce it.

DeepSeek's MLA is a newer case: compress the KV cache with a latent projection onto a smaller space, and serving gets much more efficient. Mixture of experts keeps getting sparser. Mistral's first open MoE activated two of eight experts per layer, 25%. gpt-oss activates about four of 128, a factor of 32. Sparser is better when you serve lots of users. Have we picked the low-hanging fruit? A lot of it, he says, and there is still another 10x. Hardware can be 2–3x in a year now that chips can specialize for inference: native low precision, much better networking as MoE models span multiple GPUs. Architecture can be another 2–3x. Mamba-style state-space models compress history into a small state vector instead of a huge KV cache, which matters when test-time compute runs hundreds of chains at once — Gemini Deep Think's IMO gold-medal setup. Kernels, with more talented people flooding in, are maybe another 2x. Together, in about a year, another 10x.

Dao sees roughly three workload patterns, and every inference provider will learn them. Traditional chatbots need a mix of interactivity: not too slow, not creepy-fast. You want to feel that someone on the other end at least thought a little. Then there is really low latency. He spends a lot of time in Claude Code and would pay more if it were 3x or 5x faster, because speed is the difference between staying in flow and not. Other people run four Claude Codes at once; he prefers deep work with one, which his partner yells at him for. People will pay for that agentic, low-latency slice.

The third pattern is very large batch: latency does not matter, throughput does. Synthetic data is the example. An airline trying to train a complaints agent has a small amount of great human data and can generate tons of synthetic data from it. Models are "kind of crazy" at acting like humans — a customer from New York annoyed that a flight is delayed out of LaGuardia. Effron jokes that we taught them to be angry New Yorkers at an airport. RL training is the same shape: you sample tons of rollouts to evaluate a policy, so you need huge-batch inference as part of the training loop.

Running at scale is the extra card. Fleet-level optimization on thousands of GPUs lets you push batch jobs into the holes left by interactive queries. Together's batch API, like OpenAI's, is about 50% off; DeepSeek probably does something similar. It is the rideshare logic: people at peak, cargo at off-peak, one network, two businesses. Early on, a reasonable kernel and a reasonable engine beat what was out there. Open source — vLLM, SGLang — is now production quality, and Together contributes to it. The baseline is much higher. Customers still show up with prefix caching, low-latency needs, or video that wants a different throughput profile. Workloads evolve fast enough that there is always new work.

A year or two out, Dao thinks workloads will still coalesce rather than explode into dozens of profiles. Agentic work might be a killer use case, a step change the way ChatGPT was the first time users talked to a language model, debugged code, and synthesized information. The next applications let models take actions and collect information themselves. That is not just running the model fast on GPUs. It is connecting to the tools humans already use. Web search is the easy example. An engineer designing airplanes wants the model in the design software. A financial analyst wants it in the database.

That systems work around agents is a new set of problems, a natural evolution of what an inference company sells, especially for professional and enterprise users. On the consumer side, one of his bets is real-time video generation. It will change the consumer landscape the way TikTok did, for better or worse, because it is very engaging. Together already works with Pika Labs and Hedra on that bet. Video is extremely demanding on compute, which may fuel even more chips and even more inference optimization.

The question Dao is trying to answer for the next couple of years is how to get AI to expert level. Models are at median human, or a bit above, on tasks with lots of internet data — they are much better than he is at front-end programming. Economically valuable work still sits with experts: airplane designers, hardware designers, doctors, lawyers. Those people are experts because they spent time with specialized tools, not because the internet already contains their job. Getting models to work alongside humans at that level is where the economic value is.

Albert Gu has said transformers will not be the final solution by themselves. Dao loves Albert, and this may be where they disagree. To get to AGI or ASI, the current architecture might be sufficient. The issue is cost. A better architecture might get you there one or two years sooner, or at 10x less cost, which is what makes the result accessible. The industry is spending on the order of $500 billion a year on AI infrastructure — don't quote him on the figure, he says, but that is the ballpark. Spending 10x that seems somewhat unrealistic. Better architecture is how you get there on the current bill, or less. The current designs have the right ingredients; keep scaling and you could arrive at an astronomical price.

What he is paying attention to: MoE getting sparser and sparser, a direction DeepSeek and earlier DeepMind work made compelling, more intelligence for the same compute. He factors intelligence per dollar into intelligence per FLOP (architecture, data, algorithms) times FLOPs per dollar (hardware and kernels). State-space work with Albert is the other thread. NVIDIA has trained hybrid Transformer-plus-Mamba models that reach high quality at much smaller cost or much higher inference performance. He now talks about inference-first architecture design: most of the FLOPs are spent on inference anyway, so design for that.

Dao is still in kernels and state space, and he is exploring applications. Robotics is one. How far are we from good humanoid robots in homes? Five years, ten, he does not know, which usually means interesting research questions. You can initialize a robot with foundation models: a language model can plan "walk to the table and pick up the coffee." What is missing is acting in the world, because we do not have that data. Simulation and teleoperation are attempts. Robots also have to process information at multiple resolutions and timescales: joints need very fast, lightweight control; route planning can be slow and heavy. That maps naturally onto state-space thinking. The future system is probably a composite of language, vision, audio, and world models. How you stitch them together is the big question.

Why stay a Princeton professor and Together's chief scientist? It is personal. Startups move in days, weeks, months, with a team that executes. Academia thinks on a two- or three-year speculative horizon, with students, less compute, and evaluation by whether an idea is interesting rather than whether it runs fast. Macro-level, academia's role is exploration: government money to try tons of ideas so that 5 or 10% work, the same mode as a VC. Attention got famous from the Google paper and actually came out of Mila, with Dzmitry Bahdanau, Yoshua Bengio, and others. Adam and LayerNorm came from Jimmy Ba at Toronto. Lots of the current stack is academic exploration; companies take the ideas, execute, read the market, and apply financial power. The funding environment is distorting that split. Ilya's SSI said explicitly it would not build a product and still raised money because he is Ilya. In AI, some of those research bets have started to pay off, so investors are more willing to put money in.

Quickfire: the thing he changed his mind on this year is that models are surprisingly useful in his own advanced, expert daily work — math and coding — and 1.5x was way higher than Effron expected. Open-source models will be closer to closed source in a year, because scaling is now more on the RL side and that relies more on tooling than raw compute. What people are not paying enough attention to is data. Synthetic data and using models to rephrase have had a huge impact that is underhyped. His favorite Together application is Pika and Hedra generating viral TikTok videos on models trained and served there.