Ethereum Once Held Two Thousand GPT-3 Clusters of GPUs

Two of the most inspiring events in Jared Quincy Davis's career were the release of AlphaFold 2 and the release of ChatGPT. AlphaFold 2 began as a team of three and later about 18 people, and it solved a 50-year grand challenge in biology that universities and pharma companies had not. ChatGPT came from OpenAI at about 400 people and shook the global business landscape. The question he asked himself was what it would take for events like those to become a lot more common.

Neither story is quite David versus Goliath once you double-click, he told No Priors hosts Sarah Guo and Elad Gil. OpenAI had 400 people and $13 billion worth of compute. DeepMind's small team was standing on Google's infrastructure. The leverage that produced those breakthroughs is still the exclusive domain of a handful of labs. Foundry, the public cloud he founded after research at DeepMind and a Stanford PhD with Matei Zaharia, is an attempt to spread that leverage.

The product is a public cloud built specifically for AI workloads, with the components of the cloud reimagined from first principles rather than bolted onto existing GPU or hyperscaler offerings. By doing that, Davis says Foundry has improved economics by 12 to 20x in many cases over lower-tech GPU clouds and the major public clouds. The aim is elastic, affordable access to state-of-the-art systems, plus tools for reliability, security, and price-performance. If training that costs a billion dollars today can later cost $100 million, then $10 million, he thinks events like AlphaFold 2 could become 10x or 100x more frequent — maybe superlinearly.

Asked how underutilized GPU clouds are, Davis starts with the most optimistic case: an end-to-end pretraining job that a team has planned in advance and is trying to wring dry. Even then, utilization for a lot of companies is sub 80%, and sometimes under 50%, depending on how bad a batch of hardware they drew and how often the cluster fails. Newer GPUs fail a lot. The common compensatory move is to hold aside a minimum of 10 to 20 percent of the GPUs as healing buffer, so a failed device can be swapped without stopping training. Dedicated clusters also sit idle between jobs. Utilization, even in the conservative case, is much lower than people imagine.

The mental model is part of the problem. People talk about GPUs as chips. An H100 system, he says, is a system: 70 to 80 pounds, 35,000-plus individual components. One of the remarkable things Jensen Huang and Nvidia have done is compress a data center's worth of infrastructure into a single box, which is why 70 pounds is less alarming once you see it that way. Compose those DGX or HGX boxes into a supercomputer — thousands, tens of thousands, hundreds of thousands of them — and the failure probability multiplies. With millions of components, the chance the machine runs for weeks on end is, in a nearly verbatim line from Huang's keynote, basically zero.

AI infrastructure is still young and lacks mature tooling for those failures, which is why healing buffer became the blunt instrument. Newer, more advanced systems anecdotally fail more than older, weaker ones, and Davis attributes that to complexity rather than a simple quality-control miss. He offers a working definition of "large": you enter the large regime when even the model weights no longer fit on a single state-of-the-art GPU or node, and you need multiple top-end servers to contain the model and perform one synchronous calculation. That is a distributed-systems problem. Nvidia's acquisition of Mellanox, among the better deals in market-cap terms, was about welding many machines into one contiguous supercomputer. The payoff is huge. So is the number of places a single bad part can halt the whole job.

Davis thinks the industry has forgotten what cloud was supposed to be. Right now, he says, AI cloud is basically colocation. The category itself is enormous: Azure, AWS, and GCP sit inside the largest companies on earth, Nvidia sells into them, and AWS on its own is arguably a trillion dollars of market cap if you broke it out of Amazon — at one point recently it was all of Amazon's profit and more. The modern cloud started in 2003, when Amazon dedicated about 50 people to AWS, launched S3 in March 2006 and EC2 that September, and spent years convincing people the model was useful.

In 2009 and 2010, Matei Zaharia and Berkeley collaborators wrote *Above the Clouds* arguing why cloud would matter; the points were not widely appreciated. In 2015 Jeff Bezos said AWS was for all intents and purposes unconstrained in market size, and many people laughed. That same year the business was already about $3 billion of run rate — not small, and nowhere near what it became. Elad Gil, who was building startups around 2007, pushes back on the idea that nobody believed: for startups AWS felt like magic, Twitter-era companies that had to keep running their own infrastructure had a brutal time, and the real enterprise turn came when regulated industries stopped fighting it. Davis agrees builders and VCs saw value early. What people missed was the eventual scale, and episodes such as Dropbox leaving the cloud for on-prem kept the doubt alive.

The insight Davis borrows from a Snowflake founder is that the cloud made fast free. A job designed for 10 machines over 10 days could, in theory, run on 100 machines for one day, or 10,000 machines for 15 minutes, at the same cost — a thousand times faster, then give the compute back. That requires two things: reshaping a 10-machine workload onto 10,000 machines, and actually having 10,000 machines of elastic capacity sitting in the cloud. Elasticity is exactly what AI cloud lacks. Teams are forced into long-term reservations, often three years, for a fixed amount of capacity. Nobody really wants 64 GPUs for three years, or 1,000. They want 10,000 for a couple of months, then maybe none, and they may not know the number because the product has not launched.

Sarah Guo lines the history up as a chain: servers in a closet, colocation, hosting, virtualization, containers, serverless. Non-AI developers already live on the right-hand side of that chain. The AI hardware market, she says, is still somewhere between colo and hosting. Davis agrees, and the consequences are concrete. Companies have to raise the capital they might need up front, cannot grow elastically with demand, and take on long contracts that are catastrophic if the bet is wrong. Unlike wheat or oil, this market still has no real options, futures, or way to hedge. GPUs are heavy CAPEX, and hyperscalers' depreciation assumptions are not the same as Foundry's.

Davis compares the AI cloud business to a parking lot, which sounds like a joke until you look at the two prices. Pay-as-you-go is expensive and uncertain: you may hunt through San Francisco seeing "lot full" signs and still pay about $12 an hour if you find a space — the number he chooses because it is AWS on-demand. A reserved spot in the building might be $4 an hour, but $3,000 a month. A typical office worker using it 40 hours a week is actually paying about $16 an hour, worse than the sticker on-demand rate. Foundry's new spot product is the analog of letting pay-as-you-go drivers park in someone else's reserved space.

If you can do that, effective capacity in the lot might grow by 10x, and the split can be a win-win-win: the casual parker pays seven instead of twelve, the spot owner earns five instead of paying four, and the lot takes a couple of dollars. It fails immediately if you arrive at your reserved space and find a stranger in it, or if the interloper has to leave dinner two hours early because you called ahead. Version one of the analogy is a sensor that sends a valet sprinting to move the car from floor two to floor ten, after which the original driver gets a ticket and walks upstairs. That is spot in the inconvenient sense.

What Foundry has been iterating is spot usability: the sensor knows you have arrived, the occupying workload is moved automatically, empty spaces are managed so there is somewhere to put it, and the other party's job is brought back when needed. On GPUs that means packing healing buffer with preemptible nodes and swapping a failed card without the user noticing. Spot for GPUs is much harder than classic cloud spot, which is why there is almost no large-scale GPU spot with interconnect. Customers are already using the mechanism for training — classically a difficult spot workload — and especially for batch inference.

Davis likes a trivia question because it shows how far off most people's priors are: what share of the world's GPU petaflop or exaflop capacity do the major public clouds own? Guesses usually come in at the high tens of percent. The only lower guess he got was from Satya Nadella at Microsoft, who said basis points. That was the right order of magnitude. A dated but public data point makes the scale concrete. GPT-3 was trained on 10,000 V100 GPUs in an interconnected Azure cluster for about 14.6 days, a state-of-the-art system that by many estimates cost eight figures for a single run.

He then asks how many V100-equivalent GPUs, normalized on flops even if they were not fully interconnected, sat in the Ethereum network at its peak — running 24/7, unlike a two-week training job. Elad Gil guesses a few hundred thousand to a few million. Davis calls that an aggressive guess and then raises it: about 10 to 20 million. Peak Ethereum hash rate was around 900 terahashes per second, or roughly a petahash. A V100, if you knew what you were doing, delivered something like 45 to 120 megahashes per second. Mining firm Hive held tens of thousands of Nvidia GPUs and still had well under 1% of global hash power, around 0.1%. Bitcoin chewed through ASICs; Ethereum's mix was much more GPU-heavy.

The kicker is closer to home: an iPhone 15 Pro, he says, is already stronger than a V100, about 35 teraflops in FP16 against roughly 30 for the older Nvidia card. There is a lot of compute in the world. It is not all useful, interconnected, accessible, or secure. Even for high-end GPUs there is more than people think. High-quality data he has seen puts utilization of H100-class systems — the most precious ones — at 20 to 25 percent or lower in many cases.

The fraction of total AI compute that is accessible, useful, and actually used is a diminished slice of the whole. A companion product, Mars — monitoring, alerting, resiliency, and security — was built first for Foundry's own platform to raise GPU availability and uptime, and is being opened to third parties. Foundry reserves healing buffer aggressively so a failure can be swapped without a visible disruption; packing that buffer with preemptible jobs is how the economics close. Partners can now put their own healing buffer on Foundry and rent it out, offsetting the cost of the cluster.

Two things can be true at once, Davis says. State-of-the-art, extremely large interconnected clusters will remain valuable, and they are genuinely scarce: power, space, and interconnect are all bottlenecks. Thousand-GPU clusters are far more common than 10,000- or 20,000-GPU ones, and the supply vanishes as the cluster grows. Scaling-law curves are plotted on log scales; the next bump still wants another 2x or 10x in compute, and that next 10x gets intractable fast. Google is already training across facilities — PaLM 2 and work such as DiLoCo — connecting data centers in ways that recently would have seemed inconceivable.

A more radical shift is toward what Davis and collaborators have named compound AI systems. Microsoft's Phi-3 trained a high-quality small model on high-quality data and did not need a giant interconnected cluster; the hard part was data curation. Llama 3.1's 8B and 70B variants are small and, he says, smarter than much larger systems from the prior OpenAI generation. Those smaller models appear to have been distilled from synthetic data generated by the 405B. AlphaCode 2 reached very high code proficiency with small language models by calling the model a million times per question, then filtering to the top 10 and trying those one by one — a universally parallelizable workload. AlphaGeometry, at IMO silver-medal level, likewise has a major synthetic-data component.

People are interpolating between training and inference to use the infra they actually have. The Chinchilla results are usually read as the ideal mix of parameters, compute, and training iterations if you want the smartest possible single model. Mistral's move, in Davis's telling, is to inefficiently train a small model to be smarter than its size, "wasting" money in training because the small model is cheap to run. That life-cycle view of cost is shifting work from large pretraining toward batch inference, which parallelizes without the same interconnect and without the newest cards. One bootstrap looks like unrolling a current frontier model with chain-of-thought, turning a six-step trace into filtered training data for the next model — a different path from pouring more low-quality data into a massive supercomputer.

Davis's recent paper asks how to architect those compound systems. Practitioners who want to get past today's frontier — to get "GPT-5 or GPT-6 early" — are already calling the current best model many times, often without a theory for it. Developers might pay 10x, $200 a month instead of $20, for a coding model that is 10% better. The open question is how to compose those calls into a network of networks. We know how to build neural nets. We have not yet spelled out the principles for networks of networks that mix many model calls with external components.

The first principle they explore is verifiability: domains where checking an answer is easier than generating one. Software engineering has that property. So do many math tasks, including prime factorization. A proof can take years to write and hours to read. The move is to scale out, generate many candidates, and cheaply pick a winner with a best-of-K verifier — a judge model, a unit test, or a simulator. The verifier is often more accurate at selecting the best candidate than the generator is at emitting the right answer on the first try. Repeat the loop and you bootstrap performance without a new pretrain from scratch.

The numbers are sharp. On factoring a composite of two three-digit primes, a classic cryptography-flavored problem, accuracy went from 3.7% to 36.6%, a 10x lift. On MMLU, the approach helped in the subjects you would expect — math, physics, electrical engineering — by about 3%. To put that in perspective, the gap between previous best models such as Gemini 1.5 and Llama 3.1 is often less than 1%. Davis's picture of the near future is a large network in which each stage is a best-of-K, calling Claude, Gemini, and GPT-4 together, then mixing in classical heuristics and simulators, so that answering one question might trigger millions of calls. It sounds farcical now. He thinks it will seem like common sense soon, with applications in code generation, coding agents, chip design, and even recursive neural-net design. Those parallel workloads make batch inference and synthetic data the default, and they mean cutting-edge work will not require the one giant interconnected cluster that only OpenAI can afford.