AlexNet Used Two GPUs. GPT-4 Took Twenty Thousand. The Shortage Will Be Back.

Nathan Labenz welcomes Dr. Ronen Dar, CTO and co-founder of Run:ai, an Israel-based company that helps enterprises train and deploy models by getting more out of their GPUs. The conversation is meant to cover the stack, the shortage, how companies actually lock in capacity, relative prices, who besides NVIDIA might matter, the view from Israel on the U.S.–China chip contest, and whether "compute governance" can really throttle AI. Labenz flags a second-half goal of talking to more builders outside the United States. Dar, who splits time between Tel Aviv and the U.S., is a first step.

The company started in early 2018 in Tel Aviv, with people and offices in New York and California. The work is AI infrastructure: orchestrating GPUs, sitting close to the hardware, and giving teams tools to train and deploy. The customer mix is already broad — Sony in games, financial institutions, research labs including MIT, air-gapped environments with no outside connection, and cloud accounts that run thousands of GPUs in a single cluster on AWS, with Azure and Google in the mix as well. One platform, one interface, wherever the cards live.

Five years is not long, Labenz notes, but the models of 2018 look small next to today. Dar's origin story starts even earlier. The industry's big break was the 2012–2013 ImageNet competition, when a University of Toronto team trained AlexNet and showed what deep models could do with images and video. That state-of-the-art model was trained from scratch on two GPUs. Then people trained bigger models, on more data, with more cards. By 2015, when ResNet landed, training already wanted hundreds of GPUs. A 100x jump in a few years is the demand curve Run:ai was founded to live on.

The 2012-to-2015 jump made a practical problem obvious: it is hard to train on a lot of GPUs. Distributed computing is a pile of footguns. Run:ai's original goal, Dar says, was to simplify that — let data scientists train huge models easily, even when the job is spread across a cluster. They started there, then noticed a second mess. GPUs were sitting idle, access was a political and technical fight, and a lot of complexity lived in just getting a card at all. So they built technology around access and utilization, not only around scaling a single job.

Then the same staircase appeared in language. The 2017–2018 NLP break repeated the computer-vision pattern. State-of-the-art language models around 2018, including GPT-1, trained on hundreds of GPUs. GPT-4, still closed, is rumored to have trained on more than 20,000. Four or five years, another 100x. Models got bigger, data got bigger, compute got meaner. Dar's line is that the trend is still running, and that companies watching generative AI are now racing to pull those capabilities inside the organization. Run:ai's job is to help them do it, whether the workload is an LLM or the computer-vision models people never stopped training.

Where they sit in the stack is deliberately hybrid. Customers run the platform wherever the GPUs already live: on-premises, any cloud, any Kubernetes distribution, one set of tools. Training and inference both. The integration stance is open on purpose. Experiment tracking, MLflow, logging, whatever else is having a moment on Kubernetes — they plug in, because the field moves so fast that locking the ecosystem would be a way to die. Labenz asks about MosaicML, which the show had just hosted and which had a very good outcome shortly after. Dar's contrast is positioning, not panic. Mosaic is younger and all-in on generative models. Run:ai is more general, and it grew from the GPU up, close enough to the metal to know how cards actually misbehave.

After the break he gets concrete. The first component sits on the CUDA layer: intercept CUDA calls, control who can touch which GPU, and virtualize the card so more than one workload can share it. They can swap memory between CPU and GPU so people can oversubscribe and run more models on the same device. A GPU that used to be locked to one job, often idle, becomes a pool. That is the hardware-adjacent trick, and it is why he keeps saying they built from the GPU up rather than from a dashboard down.

The second component is scheduling. Kubernetes is the default cloud control plane, but it was designed for microservices on commodity CPUs. AI jobs are heavy, long, and full of experimentation. A lot of the scheduling Kubernetes still lacks already existed in high-performance computing and in Hadoop's YARN, the scheduler built about a decade ago for Spark. Run:ai brought those ideas over: batch scheduling, the ability to preempt workers, queues, gang scheduling. The user-visible result is a guaranteed quota that is not a cap. A team gets reserved GPUs, then, if the cluster has idle cards, they can burst above quota. Idle capacity becomes someone else's experiment instead of a dark rack.

That combination, he claims, is unique: more efficiency and more availability at once, so data scientists can actually get a GPU when they want one. The other half of the product is simplicity. Without the right platform, practitioners spend something like 90 percent of their time on libraries, drivers, and setup. Scaling an experiment from one GPU to a hundred, or running many experiments in parallel, is nearly impossible in that world. With customers he cites utilization going up four times, sometimes more. In a shortage, Labenz notes, that is not a nice-to-have. It is the difference between waiting and working.

Does efficiency assume dedicated capacity? Mostly, for the big accounts. Getting more GPUs from a cloud account manager — asking to lift a limit from tens of instances to hundreds — is a big deal. You get a lot of questions. So large enterprises buy and reserve huge clusters. Startups do not want that upfront check; they live on on-demand, or a mix. Run:ai also schedules across elastic, multi-region clusters, shrinking and growing and placing a job where cards happen to be free. Reserved and spot both show up. The platform's value is "make the most of what you already paid for," which is why a shortage makes the pitch louder.

How bad is the shortage, and is it eternal? Dar's read of the last six months is that demand jumped in a way NVIDIA did not fully anticipate. OpenAI shipped ChatGPT, the world saw what generative models could do, and then Microsoft and the other cloud providers piled in. Microsoft, in his telling, went for on the order of 10,000 GPUs a few months after launch. Everyone went back to NVIDIA for more. Software demand can be answered with overtime. GPUs need more manufacturing lines and more machines, and standing those up takes time. The gap closes only on a hardware clock. H100s, the newest card people want, are still what a lot of companies are waiting in line for.

Training was always the compute story. For the first time he is also seeing a huge jump in GPUs for inference. State-of-the-art models with hundreds of billions of parameters typically do not fit in one GPU's memory, so serving one replica wants four, eight, or more cards. Microsoft, carrying both OpenAI and Copilot, has to hold a lot of inference GPUs. The deeper claim is tempo: AI applications are moving much faster than GPU hardware can. When demand outruns supply, you get a shortage. His prediction is that this one is temporary and will close, and that it will not be the last. The next one is a when, not an if, and it will keep rewriting the cost of AI.

If you actually want a hundred or a thousand H100s, Dar breaks the decision into a stack of trade-offs, not a single SKU. First: invest on-premises — buy and operate yourself, or in a colo that is still not a hyperscaler — versus go to the cloud. Cloud is no longer only AWS, Azure, and Google; specialist GPU clouds exist. On-prem, you still decide whether to buy from NVIDIA or through a distributor, and which generation you will live with: newest H100s, or A100s that are good enough at a saner cost. Storage matters. Networking matters even more. Small experiments can skimp on the fabric. Multi-GPU, multi-machine training lives or dies on the interconnect between cards. Most serious buyers, he says, end up hybrid: some on-prem, some cloud, often more than two providers.

Labenz has been Googling prices. An A100 listed around $7,000 on Amazon; H100 headlines at $40,000, which feels like a reseller number, not what most buyers pay. NVIDIA's own site was not helpful. Dar talks in cloud hours, which the providers do publish. An A100 machine — eight GPUs — on AWS costs about $30 to $40 an hour. H100s are more expensive and, at recording, still not available there the way they are elsewhere. The important arithmetic is not the sticker. H100s might cost on the order of twice as much and, on NVIDIA's benchmarks, run some workloads several times faster, so the expensive card can be the cheaper and faster one per experiment. Go buy a DGX-class box yourself and you are in the hundreds of thousands of dollars per machine.

Then Labenz hits the number that stalls both of them. Lambda Labs listed A100s at about $1.10 an hour, against an AWS figure around $4. Dar clarifies that his $30-to-$40 was for the whole eight-GPU machine, which lands near that $4 per card on AWS. The Lambda gap is still huge. Is a 4x on-demand difference sustainable, especially when multi-cloud schedulers exist to shop around? "That's an amazing question. I don't know. I really don't know." Lambda's H100 at $2 an hour is half the AWS A100 price. Labenz says he will keep trying to figure that one out. For a buyer, the lesson is narrower: do not let the three hyperscalers be the only quote in the spreadsheet.

Run:ai lives at the CUDA layer, which is exactly NVIDIA's lock-in, so Labenz asks whether they will partner off-NVIDIA and which other chips matter. Dar puts NVIDIA at about 87% of the market. The reason is not only silicon. It is the software ecosystem: libraries, frameworks, tools, so a new AI workload runs on NVIDIA cards on day one. Other chips often have decent hardware and painful software. NVIDIA also behaves like a company that stocks what the market will need in two years — very large-memory GPUs, new enterprise software — and the market did in fact need to run huge models. He will not name the second NVIDIA. Watching who becomes that player, he says, is one of the more interesting questions in the industry.

The rest of the board is crowded. Cloud providers are designing their own: Google has sold TPUs for more than five years, AWS has training and inference chips, and both do a lot of that design work in Israel. NVIDIA itself bought Mellanox, the Israeli networking company. Intel has its own fabs, is opening them to others, and has an AI-chip effort. AMD is the clearest near-term rival because the gaming base is real and the technology is good; Run:ai already has customers running AMD workloads, including on AWS. Qualcomm and Samsung should not be ignored on phones and edge inference, a market that will keep growing. Tesla, Meta, and a Microsoft custom chip sit on Labenz's list. Dar adds that the AI market is unlike prior hardware cycles: software on top of the chip is the product, and the pace of required innovation is faster.

Among specialists, Labenz raises Cerebras, whose CEO Andrew Feldman had been on the show, and the largest chip ever built. Dar has not tried it. He calls the offering unique and differentiated, and he thinks the AI-chip market will be large enough that even small players can earn huge revenue. Israel, throughout, is not a footnote. Israeli engineers, in his view, are strong on hardware and software both, which is why so much of the design layer — Mellanox, Intel's AI-chip work, cloud custom silicon — keeps showing up in the same small country.

Two or three years ago the fashionable story was Chinese papers and American decline. The last six to twelve months, Dar says, showed the strength of Silicon Valley: ChatGPT flipped the narrative. His current analysis is that China is behind the United States, but technology gaps are closing very fast — fast enough that he will not forecast six months out. Labenz's crystal ball fogs on the same horizon. On whether Europe or China can grow a top-tier champion, he treats chips as already strategic, not "the next oil" exactly, but a resource countries will fight over. There is a race in the Western world and with China, and it is just starting.

Labenz worries that cutting China off from leading-edge chips — H100s out, H800s in — raises the odds of other conflict, including over Taiwan, by making disruption cheaper to bear. From Israel the picture is not a seminar on two hegemons. Israel is small, has its own neighborhood problems, and lives with a sense of surrounding danger. Dar's number-one threat, speaking as an Israeli, is Iran, and from that vantage Iran sits on China's side, which is one reason Israeli and American views of the chip contest line up. He does not like watching things escalate. He is optimistic by nature. He also thinks AI will change the defense and digital landscape, that chips are strategic, and that political arrangements over the next decades will shift because of it.

The last question is compute governance. Can export controls, or a know-your-customer regime that licenses bulk GPU purchases, actually control who can do what with AI? Dar treats it as an interesting angle and an unsolved one. Training a frontier model still takes a mountain of compute, and the U.S.–China chip war is already an experiment in denying the newest GPUs. Then the open-source path cuts across it. You can take a strong released model, fine-tune on your own data, and you do not need tens of thousands of GPUs. Llama 2 had been open-sourced essentially as they were recording — free for research and commercial use, which he finds genuinely exciting. If capable weights are already out, controlling access to compute becomes less relevant. He will not close the question. A person whose job is filling every GPU is not convinced that locking the hardware locks the capability. That reservation is the note Labenz leaves on.