Speed Is the Thing You Can Buy: Baseten’s Bet on Inference

Tuhin Srivastava, CEO and co-founder of Baseten, describes the company as fast, scalable AI infrastructure for engineering teams working with large models, starting with inference and intending to do more later. The firm has been at it for about four and a half years, which is long in this industry. They started in 2019 because they thought machine learning was going somewhere and wanted a picks-and-shovels business that solved problems they themselves were hitting. A side note, he admits: he wanted to start a company with his friends.

Outsiders often file Baseten under no-code. Srivastava rejects the label. Early on the product had no-code-ish elements; three or four years of customers taught them that code is incredibly powerful and that engineers want to write it. Even at its best, you want tight abstractions, but the ability to turn knobs under the hood matters. No-code does not remove those knobs so much as make them harder to reach. Baseten’s job is to make the easy things super easy and keep the hard things possible, and to solve a “graduation problem”: a team should not have to switch tools between a weekend side project and large-scale production.

The workloads on the platform run from tiny weekend projects to AI-native companies and even foundation-model labs. Baseten supports Descript, where AI is core to the product, and powers AI features Patreon has shipped. Srivastava’s favorite examples sit at two extremes. Bland AI, building an SDK for call centers, collocates models and workloads to get sub-300-millisecond, sometimes sub-200-millisecond responses without months of infrastructure work. Picnic Health, a decade-old medical-data company, runs Picnic GPT to extract information from records. Domain skill and proprietary data belong to the customer. The infrastructure does not have to. That, he says, is where a lot of the next decade’s value will be released, without many person-months of serving work.

It has become fashionable to compare GPU-cluster sizes — 600,000 H100 equivalents, giant rounds raised to train or post-train. Elad Gil asks whether training and inference workloads are actually different. Srivastava says yes, down to the SLAs. Inference clusters want to sit near the work, the way a restaurant wants foot traffic. Training cares less about location as long as the GPUs are together. Networking on the racks is critical for training and much less so for inference, which spends more time on individual GPUs and less communicating across them.

The user workflow splits the same way. State of the art for training, in Srivastava’s telling, is still “give me SSH keys and let me go at it.” Inference is a repeated product: version management, deployment, CI/CD hooks, cold starts. Customers want similar things from the stack, which is why it can be packaged. Reliability inverts too. Downtime is unacceptable for inference; training nodes get terminated all the time and the SLA is more about throughput than uptime. Hardware classes keep moving as well. At the end of 2022 most Baseten customers were on T4s and A10Gs, then A100s, now H100s. Compute demand is not going down. It is going up as the services scale.

Asked what surprised him most, Srivastava answers at two altitudes. From the market: he felt, viscerally, Elad Gil’s old line that markets are all that matter. You can build cool stuff; when the market shows up it shoves both the customer and the product’s requirements forward. The acceleration through the end of 2022 and through 2023 took them by surprise. From 2019 to 2022, he says honestly, it was pretty quiet. They had happy customers. The demand was not really there.

From the practitioner side, the shock is how fast teams now move. In AI, at early-stage companies, and increasingly inside enterprises waking up to the same fact, speed is the number-one advantage. Things move so fast that if you are not competing on speed you get left behind. That has flipped the old hesitation to buy infrastructure. People will pay for technology they would once have insisted on building. Baseten talks to companies it assumes already have a stack, then finds they are less sophisticated than expected and handling more scale than expected. GPU hunger evolved on the same clock: T4s and A10Gs, then A100s, then H100s, always up.

Baseten had just been highlighted on independent Artificial Analysis benchmarks for highest throughput and lowest latency. Fast inference, Srivastava says, fails in three places: workflow headaches, scalability and reliability bottlenecks, and the time a single generation takes. The leaderboard is about the third. In the previous six months the research community poured out tricks such as speculative decoding, where a small model drafts and a large model verifies, plus continuous and dynamic batching that keep GPUs fuller. Baseten is tightly partnered with NVIDIA on TensorRT-LLM, has forked it, and has contributed back. The work is low-level. There is no clean abstraction. You either learn open source very well or rewrite kernels yourself.

What people complain about at OpenAI, he notes, is speed. That is a core performance advantage of open-source and smaller models: they can run fast. The benchmark numbers show the heat. State of the art was 90 tokens a second, then over 100, then over 200; some people are talking 300 to 400. He thinks language-model serving performance will eventually commoditize, and more of it should run locally. Staying attached to the state of the art is still existential. If Baseten falls off that curve, it is a business risk they cannot take.

Sarah Guo asks when users will stop waiting ten seconds, or two minutes, for a generation. Srivastava’s path has three layers. Hardware first: H100s are a step function over A100s, and H200s are next, hopefully cheaper and more available. Software next: continuous batching, dynamic batching, speculative decoding that parallelizes, batches, or offloads work to another model. Then smaller models — distilled, sharp, good at one thing. He finds Ollama interesting, and notes that Sourcegraph has announced Cody running locally for many customers. Guo calls it the unbundling of models. Srivastava admits that is not necessarily amazing for cloud providers who want everything to run on them. Image and speech are on the same clock. Gamma uses Stable Diffusion and cannot wait four or five seconds for a still-high-quality image. Whisper now exists as stock Whisper, faster-whisper, and a TensorRT build; everyone wants real time. Baseten is pushing those limits too, just not yet as far as it has pushed language.

When do customers deploy their own, open-source, or fine-tuned models instead of hitting a public API? Srivastava describes a ladder. You start on OpenAI or Anthropic because the API works. Then it is too slow, too expensive, or more powerful than you need, and you try open source. At the beginning you use a shared endpoint. Shared endpoints cannot give you your own SLAs: if two apps share machines and the neighbor gets slammed, your calls still have to stay fast. That is when you want dedicated compute. Data and privacy push the same way — maybe you do not want your traffic on the same pipes — and at some scale running it yourself is simply cheaper.

Large companies skip the shared rung. Enterprises often want something like Mistral 7B not off a shared endpoint, not even on dedicated cloud GPUs, but self-hosted in their own AWS or GCP. They already have marketplace spend commits and credit systems. Running inference against those commits solves procurement problems and has a massive cost advantage. So the three stages are shared, then dedicated in the cloud, then into your own cloud when even dedicated is not enough.

Gil’s enterprise prediction is that ChatGPT is only 15 or 16 months old and GPT-4 about a year old, so most companies are still in a planning cycle. Infrastructure vendors are on a wave that may get ten times bigger. Srivastava is wary of timescales; the industry has been wildly wrong on them. What he sees now is that copilots, especially for code, are already inside the enterprise. Ask a company how advanced its AI strategy is and the answer is often that everyone uses Copilot. The next wave, using OpenAI or Anthropic in production, is happening. A Pfizer-scale story of tens of millions of dollars of AI investment over 12 to 18 months frightens him a little. It is good for Baseten’s business and smells like the 2018–2020 ML trap, when CIOs bought software detached from real user value. His bet: we are overestimating enterprise over the next 12 to 18 months and underestimating it three to five years out. Ten-x would still be a conservative reading of the later period.

The volume is already stranger than traditional software intuition. One Baseten customer with four engineers is on track for mid-hundreds of thousands of dollars of annual inference spend by year-end — one use case, fewer than a thousand users, and the company is already cash-flow positive. Customer service and chatbots are the obvious efficiency battlefield; the call volume is far higher than a classic business, with harder latency requirements, which drives spend higher still. Srivastava’s brother is head of AI at Sunrun, the public solar company, another place where AI can eat processes at a scale that dwarfs a startup demo.

Guo describes the investor reset. Traditional SaaS at maturity was supposed to print 80 percent gross margins; data-processing COGS made software investors nervous. Training businesses have a big upfront capex bet. What is new is that you can spend heavily on inference and still end up with a valuable company that has fewer people. In her portfolio, headcount is more efficient and compute is a much larger line. For some Baseten customers, inference is already one of the largest items on the P&L. Srivastava tried to collect a year of compute upfront from one of them. The CTO came back: after payroll, this is the second-biggest expense of the year, and we are not prepaying it. That refusal is a market-size signal. The most efficient of these businesses, through markup and software optimization, can still run healthy margins on aggressive consumption contracts — a shape that is not normal SaaS and is hard to map onto traditional multiples.

Gil worries that ramps this fast on products a few months old look like the 1990s internet: startups selling to each other, telecom build-outs on a five-year cycle, then a glut. Three companies going from zero to five or ten million in the same year means enormous demand and an open question about defensibility. Srivastava points at healthcare. Nuance had a stranglehold on medical transcription for years and always looked like it was struggling; then Whisper arrived and medical note-taking exploded because the value was real. The moat, he and Gil agree, snaps back to the workflow. Gil adds that many markets are oligopolies, not monopolies — Stripe, Adyen, and PayPal in payments. Healthcare is deal-driven: multi-year contracts with big customers fragment into a few survivors more easily than annual-renewal software. Contract structure is an under-discussed determinant of whether a category can stay a business. Financial services, oddly, look lagged. Hedge funds were all over big data a decade ago; generative models are a different capability from old-school regression, and that muscle memory may be slowing the turn.

Guo has been asking the inverse screen: in an era when AI eats everything, which companies simply do not care? That indifference may be a moat. Demos from the past few weeks make the screen harder to use. “I thought my job was safe until yesterday,” she says. A partner asked how long venture capital would last against agent automation, having left software engineering at what he hoped was the right time. Her serious answer is that early-stage investing has an inputs problem. A lot of the signal is about people — who is leaving, who is a 100x engineer, who has product taste and velocity — and there is no digital trail. You are always out of distribution in a changing technology landscape. Gil’s punch line: everyone asked this question decides their own job is defensible. Guo insists she is not claiming that. She just does not think it is a tenable problem this year, and if it becomes one they will try to be first.

On hardware, the shortage has changed shape. A year earlier you could not get anything except a T4. With large, long-term commitments, Baseten can now acquire compute fairly quickly. Customers still struggle to get the most premium chips, H100s or A100s; even when supply exists, they spend three to six weeks negotiating with cloud providers, with sales reps calling in favors and conversations constantly escalated. Heterogeneity sounds good in principle. Srivastava thinks the industry overstates how easy it is to run CUDA-like workloads on AMD. Baseten already spends painful time debugging bad NVIDIA nodes. Customers fiddle with NVIDIA drivers, install new PyTorch versions, and ship custom Docker images. Abstracting that across untested chips, while they still touch the GPU, would add complexity and slow the company down. He hopes for more than one option. He will not spend the engineering to get there in the short term, not while customers are saying they need NVIDIA now.

Build versus buy, then, is not a philosophy. “Speed is the only thing that matters in this market.” Spend time fiddling with infrastructure, launch, and have the service go down, and you have hurt the end user. What is proprietary is models, data, and workflow. What is repeated is infrastructure. The last twelve months are full of teams that sounded like infrastructure engineers a decade ago — we will build this ourselves — and came back three months later with a Docker dumpster fire. Baseten’s sales qualifier is “have you built it yourself?” Pain makes a great customer. A four-person AI infra team that had been building for two years migrated all of its workloads in 36 hours and put those engineers on differentiated work. Customers who want to stay in their own environment can pay a license fee. The showcase is a six-person chatbot company pushing a billion tokens a day. Elastic, reliable, high-performance serving plus the product experience on top is impossible for six people. Give away the parts other people can do as well or better. Buy the speed. Skip the long build cycle.