Inference Is the Last Market: Baseten’s 30x Year

Baseten sits in what Tuhin Srivastava calls one of the craziest markets in tech: the AI inference cloud. Over the last year the company grew about 30x. Sarah Guo puts on the record that it expects more than a billion dollars of revenue this year. He does not flinch. The last 24 months, he says, keep getting bigger, because people finally believe you can put AI everywhere.

Three conditions lined up. Closed-source and open-source options both got good; open-source models crossed a chasm in baseline capability; and reinforcement learning plus post-training for specialized models became mainstream enough that customers could see it working. Once that happened, they wanted to own more of their own inference. Long-tail models stopped being a slide and started being production. Baseten’s job is to index on that demand as the application layer grows.

Guo asks the existential question everyone keeps asking: does an independent application layer get to exist, or do the labs eat it? Srivastava’s first answer is honest. It would be sad if it did not exist. Sadness, he says, is not an investment thesis. The thesis is about where value sits. If the valuable thing is user signal that only the company can gather, and that signal is encoded into a model, a lot of the business is at risk. If it is encoded in workflows, the company can grow a node the labs cannot copy.

His example is Abridge, an ambient scribe used by physicians in almost every U.S. hospital. The scarce signal is not the first-draft note. It is how clinicians edit those notes, and what happens three steps down inside the electronic medical record. A frontier lab does not have that loop. The company that does can post-train on it and, over time, run longer-horizon agentic models against a workflow nobody else sees. Support is the same pattern. At a company like Decagon, a ticket is rarely one-shot; it is one, two, ten, twenty actions. That multi-step residue is soil for a specialized model.

Guo then splits the market: new AI-native companies selling to customers, versus enterprises building in-house. She asked the same question two years earlier. The answer, he says, is crazily consistent. By inference count it is still about 99% the former. That is the size of the remaining opportunity: most of the market has not come online. What has changed is the staircase. Two years ago it was not obvious enterprises would even use AI tools; that is obvious now. Closed-source model APIs are starting. Custom models are still ahead. The majority of Baseten’s customers today are the AI-natives at the top of that curve — Cursor, Abridge, OpenEvidence — and he expects the enterprise wave later.

How do you serve those AI-natives without becoming an enterprise company yourself? Two ways, he says. First, build for the highest scale, the way Stripe did: twelve years ago it was not an enterprise vendor; it rode the frontier and the enterprises arrived. Second, notice that your customers already sell to enterprises. Abridge, OpenEvidence, Decagon, Writer, Gamma, Clay — they translate the requirements. Data retention, where models may be deployed, which GPUs and latencies they will accept, what transparency a buyer wants. Serve Abridge and OpenEvidence well, and you are already most of the way to serving a health system, because they are selling into one.

The open-source mix on the platform has shifted in public. Two or three years ago the story was Mistral, then Meta’s Llama, then a wave of Chinese models. Baseten’s customers — among the fastest-growing AI companies in the world — buy capability first and optimize cost second. A small subset of tasks starts from price. The roster is whatever is at the frontier: GPT-OSS, Moonshot, DeepSeek, Canopy Labs’ Orpheus for text-to-speech. The difference now is that Baseten has more visibility into how to run those models well, and that they are actually good.

Gil asks about security worries — Trojan horses, agendas baked into Chinese weights — and whether the U.S. needs a counterweight. Srivastava’s split is careful. The models are fantastic; his team works with those labs. Network-bound a model and it cannot magically cross the boundary; he has never seen real evidence of a backdoor or agenda except in some early models the community spotted quickly. He still thinks it would be a massive loss if the U.S. cannot stand up its own open models. China has five labs shipping them. America is struggling to get one going. Necessary, he says, and inevitable.

Someone told him after the DeepSeek moment to forget it was a Chinese model and act as if it had come from Meta. Missing the forest for the trees, in his view: either America never produces good open weights, which is a fundamental problem, or it will, and the industry should be ready. Gil adds a colder reading. Chinese government subsidy of these models is, for now, being passed through to U.S. enterprises that adopt them. Guo notes the frontier is still closed — Anthropic, OpenAI, Google. Srivastava’s economics are still stark. Three months earlier DeepSeek was at or near the frontier; running it in production was perhaps 20% the cost of OpenAI and Anthropic models, with comparable or better latency and reliability. Lose that form of intelligence and a country innovates more slowly, because cheaper intelligence gets embedded in more places.

Asked to split traffic between custom models and vanilla open source, he does not hedge. It is basically all custom: 95% plus. Baseten now has three businesses. Dedicated inference, where the customer owns the SLA. Shared inference, shared endpoints and shared SLAs. And a training business. About 95% of tokens sit in the first bucket. Almost every customer is modifying the model with their own data for the use case. Some customize for quality, some compile differently for performance. Nobody is just running the vanilla open-source weights.

A few months earlier they acquired a research team to close a gap. Baseten was infrastructure and product people, strong on serving, thin on research. The acquired company was already a customer: it post-trained models and ran them on Baseten, and it was going to have to become an inference company anyway. Baseten needed to get closer to customers earlier. There were Australians on the team, which he jokes was a bit of alpha. The deeper lesson showed up once they started researching post-training instead of only performance. Quantization, when you do it, and how you trained the model are paired problems. Inference creates data; you eval; you post-train on the reward you found; that loop should produce more inference.

When should a customer invest in a custom model? Prove, with the best-in-class model, that you have something worth optimizing. Two years ago the meme was no GPUs before product-market fit. His update: no post-training before product-market fit. The people they work with already have unique user signal and have shown they can deliver value. Then you ask how to do it better, faster, cheaper. If the job is customer support, you may not need to be good at coding. A specialized model can be the better, faster, cheaper fit.

Everyone talks about a multi-year supply crunch. He does not think people realize how bad it really is. There is very little slack compute. Baseten runs large clusters of its own at uncomfortably high utilization — mid-90s, most of the time. The original architecture was a reliability play: one runtime fabric across many clouds, so customers would not care which provider held the GPUs, with failover and latency handled in software. That fabric now has a second job. They sit on 90 clusters across 18 clouds. Introduce a new provider in another country and they can have the full Baseten inference stack in the fabric in half a day, maybe less. Even then, growth is hard. There is a standing company meeting about how to allocate capacity against demand that is already here.

Capacity is only half the bottleneck. A lot of new suppliers, he says bluntly, are a bit grifty. They have not run data centers. They do not understand inference SLAs. Even when cards exist, operational quality does not. He would count about a dozen decent clouds and put three or four in a gold tier. The industry is not only supply-crunched. It is operator-crunched.

Can you buy January 2028? You can write the contract. You are also making a large bet in a market that moves every quarter. The H100 is four and a half years old and the price is still going up; he half-jokes it might be a nine-year chip, which helps the long-duration case and does not remove the bet. The change in the last six months is term length. A thousand and twenty-four B200s from a good cloud is not coming on less than a three-to-five-year contract, with something like 20 to 30% of total contract value prepaid. Acquiring capacity now requires enough demand to fill the cards and a low cost of capital. That mix is one reason they have thought seriously about going public sooner than a software company might, and why he has been learning a lot about debt. The working-capital shape of this business is unusual.

What makes a winner when everyone understands the problem? GPUs as a service, he says, are not sticky. Customers treat them as commodity. Inference with the software layer included is incredibly sticky. None of Baseten’s top 30 customers have ever churned. Annual net dollar retention is around 400%. The optimist in him wants to build the best software layer for inference that exists. The strategist in him has watched the labs: in a world of constrained compute, the number one thing to own is compute. If they have all of it, good luck running inference. You cannot make a good hot chocolate without milk. Guo jokes about oat milk. Nobody, he says, wants vegan inference.

Will there be a multi-chip world? He wants diversification the same way he waters many models, and he would be sad if it did not happen. Inference-specific chips, decode-specific chips — Groq’s LPU is the obvious example, and NVIDIA has said similar things. Do not underestimate NVIDIA’s supply chain, CUDA, or developer ecosystem. For an infrastructure company right now, speed of movement is the job, and you move fastest on NVIDIA. At their scale, a frontal competitor in the next couple of years is hard to see. The other chipmakers have a structural problem: if you tie most of your supply to one big lab, the ecosystem never forms around you. If you are that lab and you have 90% of a chip’s supply, it is in your interest to take 95%, so everything is built for you and nobody else can use it.

Workloads are pulling the runtime in new directions. Diffusion transformers, sandboxes for coding agents, new speculation techniques, KV-cache-aware routing, disentangling prefill and decode as separate problems — they are seeing large gains on the last of those. Above the runtime, the product thesis is the inference–post-training loop: native integration with an eval company such as Braintrust, a first-class sandbox experience, training APIs that make continual learning less of a discrete event. Around that loop sit the margin primitives, including async batch inference to drive utilization. Start with core inference, go down the stack to unblock supply, go up the stack to unlock value.

Scale makes edge cases real. The day before the taping they hit a kernel panic for the first time: a logging worker wrote too many logs onto one node, two processes at once. Hyperscalers are not infinite; at this size you meet their limits too. The surprising part, for him, is how immature LLM runtimes still are. Even KV-cache use is less sophisticated than outsiders assume. The next primitives — scale, security, performance — are hiding in those gaps, more often at the systems layer than in some LLM-specific mystery. What keeps him up is still capacity. There is no world, he thinks, in which there is enough compute in the next five to ten years to extract the value people want from LLMs, unless the field invents a lot of new things. The internal answer is always go bigger, go faster. Fun, and exhausting, and uncharted.

Organizationally, Baseten was extremely flat until 12 to 18 months ago. Elad Gil told him on a walk that he just needed leaders — advice that offends engineer-brain, where everything looks like overhead. He had once asked Guo why they should not just hire engineers instead of salespeople. What he learned: you want people you can hand whole problems. If you feel you have to be in everything, that is usually a hiring miss, not a founder duty. Be explicit about what you optimize for. “Smartest, hardest-working” is not a rubric. Theirs is first-principles thinking, kindness, low ego, no hero culture, and if you need a manager this is probably the wrong place. Once the rubric is clear, the fits and the misses become obvious. Core turnover has been low.

Operations is the other filter. Guo recalls Alyssa Henry, the longtime AWS executive, saying the hard thing about cloud is operations and that she slept with a pager under her pillow for a decade. Srivastava’s phone buzzes on air; he gets anxious. In a 45-minute meeting with senior AWS people, everyone’s pagers went off more than once. When his co-founder’s pager goes off, the co-founder’s seven-year-old asks if it is a P0. A P0 pulls the company onto a call. People joke there may as well be a siren in the office. The culture rejects engineers who hide from the pager, fast.

On Jevons: make intelligence cheaper and people consume more of it, not less. Consumers want the best answer, which is somewhat governed by more intelligence. Developers insert more intelligence when it is cheaper — a hell of a lot more. Agents run longer. Customers either start from the quality they need and back into how much inference that takes, or from a base model they can afford. Drive cost down and the conclusion is the same: more intelligence means better answers, more actions, more revenue. He does not see an “this answer is enough” ceiling. Inference is the last market. Even if there is AGI, all that is left is inference. The consumer picture is concierge everything: an agent for the doctor, for learning, for organizing a life, personalized education, more software rather than fewer software engineers. Baseten is not slowing engineer hiring. The extinction risk sits with workflow-software companies that do not insert intelligence into the product.