A Hundred GPUs in Seconds: Why Modal Threw Out Docker and Kubernetes

Elad Gil introduces Erik Bernhardsson, founder and CEO of Modal, a serverless cloud for AI, machine learning, and data applications. Before that Bernhardsson spent seven years at Spotify, where he led machine-learning work and built the music recommender, then served as CTO of Better.com. Gil asks what he was trying to fix. The story starts in 2008. Spotify had almost no data infrastructure; Hadoop was the modern thing. Bernhardsson built a lot of it himself, including Luigi, a workflow scheduler he jokes almost nobody uses now, and Annoy, a vector-search library that had a moment and then faded.

At Better he thought constantly about developer productivity. During the pandemic he took time off, started hacking, and realized he had always wanted better infrastructure for data, AI, and machine learning. That was the genesis of Modal. The specific pain was the cloud itself. He has used it since about 2009 and loves the power; working with it is still frustrating. The idea in his head was to make cloud development feel almost as good as local development — fast feedback loops, write code and run it. Pretty quickly he concluded they could not use Docker and Kubernetes, so they threw them out.

The first two years were foundation: their own filesystem, scheduler, and container runtime, so containers could cold-start fast enough to run customer code safely. On one side Modal runs a large compute pool, thousands of GPUs and CPUs; if you need a hundred GPUs they can typically have them in seconds. Capacity planning is their problem, not the customer’s. On the other side a Python SDK turns functions into serverless cloud functions and hides the Kubernetes and Docker tax. They founded the company pre-generative-AI. The killer app arrived when Stable Diffusion shipped and people showed up because Modal had GPUs and no machine-provisioning ritual. Text-to-image is still a large slice; audio, music, and video are growing. Suno, the AI music company, runs all of its inference on Modal at large scale.

Gil cites an October post of Bernhardsson’s, roughly titled that the future of AI needs more flexible GPU capacity, and the industry complaint that a lot of GPU use is wasteful. Bernhardsson’s paradox is simple. GPUs are expensive, so the main way to get cloud capacity has been long-term commitments. That is not how startups should buy compute. The amazing thing about the cloud, for him, was always on-demand access to as many CPUs as you needed. Scarcity turned GPUs into contracts. His vision is that GPUs should work like CPUs: instant access to thousands if you need them.

Training can at least consume a reserved cluster. Inference is worse because you do not know how much you need in advance; it is volatile. The planning trap is familiar: over-provision and pay for idle cards, or under-provision and watch the service degrade when traffic spikes. Modal is fully usage-based. Customers pay only for the time containers are actually running. Bursty, unpredictable jobs are what a shared pool of thousands of GPUs across many tenants is good at. Gil notes the strangeness of training: you spin up a supercomputer for months and the output is a file, in some sense a compression of the internet. Shorter training runs and post-training need the same flexibility as inference.

Inference has been Modal’s main case. They are now interested in training, especially short, bursty, experimental runs — not the giant multi-month jobs, which he treats as a different market. The broader goal is end-to-end productivity for traditional ML engineers who build their own models: preprocessing, training, inference, feedback loops, recapture of data, online ranking, pipelines, nightly batch. They do not yet ship a training product, but customers already use Modal for batch preprocessing. A team training a video model might dump petabytes onto the platform, even using GPUs for feature extraction, train elsewhere, and come back for inference. Covering that whole lifecycle is the point of a platform.

Gil notes that the vision is a broad platform and asks how Modal differentiates. Bernhardsson’s first answer is that they are cloud maximalists: a multi-tenant platform that runs everyone’s compute, which is how they can hand out hundreds of GPUs instantly. The second is generality. They run custom code in their containers, which is a harder problem than serving one model family. Safe containerization, cold start, a custom scheduler, runtime, and filesystem are the cost of that generality. Unlike vendors focused only on inference or only on LLMs, the bet is a general-purpose compute layer that can grow more products as it matures.

Enterprises, Gil says, already have data on Azure, GCP, or AWS, applications there, marketplace credits, and finished security reviews. They worry about latency and extra hops to a third party. Bernhardsson remembers 2008 and 2009, when the cloud arrived and his first reaction was: why would anyone put their computer on someone else’s computer? That seemed insane. A few years later it made sense, and even large companies now often concede that compute may be safer in a hyperscaler’s hands than in their own rooms. He had the same allergy talking to Snowflake around 2012, when they said they would run databases in the cloud as infrastructure-as-a-service, maybe not even in the customer’s environment. He thought that was nuts. Snowflake is now a large public company.

There is still resistance to the multi-tenant model. He thinks the tailwinds are with it. Security is moving from the network layer into the application layer. Bandwidth costs are coming down, and there are tricks to shrink transfer bills — storing data in R2, for instance, which has zero egress fees. Pushing that adoption will take work. The capacity-management benefits are large enough that, to him, a big part of AI’s future is running a big pool of compute and slicing it very dynamically.

Gil asks which open-source models are taking off lately, given that Stable Diffusion pulled in the first users. Bernhardsson sees a shift toward proprietary models that still release open weights — Flux, most recently, is getting a lot of attention. Personally he is more interested in audio, which he calls underexplored, with real room for open-source models, though nothing truly cool has shown up yet. What is missing in infrastructure? He admits the biased answer: Modal, a way for engineers to take code and run it. Outside the LLM world, which he treats as a different planet, plenty of teams still want to train and run their own models, or at least run others with highly custom workflows, and that remains painful.

Modal has barely touched storage; they focused on compute. He is watching vector databases, more efficient training-data storage, and a networking problem: efficient large-model training currently means spending a lot of money on fabrics such as InfiniBand inside giant data centers. If training could be made less bandwidth-hungry, you could tie GPUs together across sites and the whole training map would change. On the standalone-vector-DB versus Postgres-plus-pgvector debate, he does not pick a winner. You can probably stick a lot of this in a relational database and be fine. The more interesting question is whether an AI-native store even has a database’s form factor.

People, he thinks, are shoehorning vectors into an old put-data-get-data-back model. A more native design might make the database the embedding engine: put text, video, or images in, and search by those, not by a vector you computed elsewhere. He knows that sounds fluffy. We are early; it may take five or ten years to shake out. Gil then asks for a heuristic on training versus off-the-shelf. Bernhardsson’s version: if model quality really matters, you eventually have to train your own, or it is hard to defend why your solution is better. You can put the moat somewhere else in the stack, and many companies will. If you have a model that is clearly better than anyone else’s, that is a moat by itself. Outside LLMs, for teams whose core work is audio, video, or image models, he thinks you basically have to train.

Gil notes that Bernhardsson is an IOI gold medalist and asks for contrarian predictions about coding and AI. Bernhardsson is not sure his view is contrarian. AI is one of many developer-productivity jumps, in a line with compilers, higher-level languages, databases, and the cloud. In hindsight it may not look more special than those. Every time it happened, latent demand for software was so large that the number of software people went up. Over about forty years he sketches a crude scale: every decade, engineers got something like ten times more productive from better frameworks and tools, and that unlocked more work. He is very bullish on software engineers. It would take a lot to destroy that demand. People treat AI as a fixed pie; he thinks it unlocks more pies.

Gil also notes a Swedish high-school physics championship and asks about physics-based models and simulation. Bernhardsson was a math-competition nerd as a teenager and would love a resurgence; the golden era of physics, in his telling, was the 1920s through 1940s, and the field then doubled down on a theoretical path he associates with Edward Witten and maybe got a bit lost. Gil clarifies he means HPC-style engineering: simulating an airplane wing, load-bearing structures, the ANSYS-class world. Bernhardsson agrees that HPC has always existed, especially in oil and gas, but as small, hand-tuned models for specific jobs rather than a rewrite. Meteorology is where he thinks deep learning should bite. Turbulence is hard for traditional physics solvers; NVIDIA and Google have published interesting weather-simulation papers. Adjacent, computational biology — protein folding, sequence alignment — has already been enormously successful, and Modal is seeing more of that usage.

He is not a biology person, so the customer list is a surface read, but medical imaging shows up a lot: modern methods can run huge numbers of experiments, automate electron-microscope imaging, then push the pictures through Modal for processing and computer vision. Asked what he is most excited about for human impact, he goes back to Spotify. Suno still sounds a little uncanny, not quite right, but each generation of their model is better. Music, he says, is usually among the first places a new technology shows real impact: gramophones, piracy, iTunes, Spotify. Suno is more specific than that. It is a product that could not have existed before generative AI. There is no way it could have existed five years ago. That, to him, is the frontier: not a faster version of an old tool, but a category that was previously impossible.