A 50,000-GPU cap and a 1,700-chip loophole

Jacob Effron introduces Dylan Patel as one of the people the industry now reads on hardware and AI, writing at SemiAnalysis. The taping landed in the middle of “AI diffusion rule week,” which is why the conversation has the feel of a live reaction: what the rule does to hyperscalers, to China, to the UAE, and to the geopolitics that sit underneath all of it. They also promised NVIDIA’s moat, open-source models, and the stories from xAI’s 100,000-GPU cluster — “the most Elon awesome thing ever,” Effron says, after the lab could not find a data center on the timeline it needed.

Patel’s version of that week is simpler. The interview was supposed to be Monday. The rule landed Sunday night into Monday; he was up all night on it, cancelled, and they slid the date. By the time they sat down he had already shipped a long piece walking through clauses, loopholes, winners, and losers. Effron asks for a grade against the government’s stated goals.

The October 2022 semiconductor export controls, Patel says, did not mince words: Washington wanted to regulate AI, and it already believed scaling would move fast. Talk to people such as Ben Buchanan, then a White House special adviser, and they will tell you that was exactly why they did it — they knew it was coming. “The government is actually quite competent,” Patel says. The aim was blunt: the United States must stay ahead of China, because the next two to five years of progress will shape a century of hegemony. October 2022 was well-intentioned if you share that five-year worldview. If you think the transformation takes twenty years, the same rules look pretty bad, because they will limit U.S. competitiveness for a long time. Over a five-year window, they will keep America further ahead.

Later rounds in 2023 and December patched loopholes, but a large one remained: Chinese firms could still get GPUs through foreign clouds. Oracle has been loud about the new rule; one of its largest cloud customers is ByteDance. The other escape hatch was geography. Build in Malaysia and, until this week, it was suddenly fine. From 2024 to 2027, companies operating there — many of them Chinese, many now claiming to be Singaporean — are putting up about three gigawatts of data-center capacity. GDS, China’s largest data-center operator, moved to Singapore and says it is Singaporean. Three gigawatts, Patel notes, was roughly Meta’s entire global footprint at the start of 2024. The world’s social-media load is on the order of what Malaysia is standing up in three years.

This is the most far-reaching regulation he has seen in his own career — Cold War and World War II excepted — because it regulates overseas clouds and slashes what foreign firms can buy. Microsoft and Oracle have real assets going up in Malaysia and will probably keep building. The catch is a 7 percent cap: a U.S. cloud may put no more than 7 percent of its data-center capacity in any country that is not a designated ally. Oracle had planned something like 20 percent of its capacity in Malaysia. The only firms that can absorb those overseas halls without breaking the cap are the ones that already hold 70 percent or more of their AI data-center capacity in the United States: Microsoft, Meta, Amazon, and Google.

That is how a China policy becomes an antitrust event. Everyone else is “kind of like, so.” CoreWeave has moved close to hyperscale and can still expand in the U.S. and Europe for a while, but every extra monitoring requirement shrinks the set of clouds that can comply. Below CoreWeave and Oracle, the mini-cloud layer is hit hard. A lot of Bay Area startups have sovereign-AI buyers in Malaysia, Singapore, India, and the Middle East as their biggest customers; those buyers are now inside the regulatory perimeter. Korea, he notes, is a tier-one exception. Patel reaches for World War II: America told Henry Ford he would get rich making the tanks. The modern version is “here you go, Satya Nadella, you get to do this in AI.”

Before the rule, Patel calls the landscape the wild west. A U.S. executive order asked for notification if you trained above 10^26 FLOPs — about twice Llama 405B, a bar almost nobody had hit. Builds that could clear it were coming in 2025 and 2026. There was tacit pressure (OpenAI was told not to take a giant UAE cluster) but no published regime. A company in China, or anywhere else, could rent GPUs from whoever would sell them. The new rule smacks that down. You cannot export the biggest foundation-model weights outside the United States or trusted clouds — again, the hyperscalers — and you cannot casually ship a Llama 4 or GPT-5 class artifact abroad.

Synthetic-data generation is on the list for a reason. Chinese labs have been using a lot of GPT-4 output to post-train. That, Patel says, is part of the dirty secret of why Alibaba and DeepSeek look so good. Clouds must also be able to observe customer workloads. He does not treat that as a free security win: it forces every tenant to accept monitoring, and the monitoring itself is a new attack surface. ByteDance’s pattern was to rent from random GPU clouds in Europe and Asia, pay them a decent profit, and keep the workload opaque. Most of that, he is careful to say, is serving TikTok — but the firm is also on language and generative video, and the cluster sizes and notification rules now available to it are “heavily, heavily, heavily limited.” The same blow lands on the small clouds whose business case was “I will sell to ByteDance.”

The headline cap is 50,000 GPUs per country over the next four years. NVIDIA, he says, is making six million-plus this year. The hole is the exemption: a purchase of 1,700 GPUs or fewer does not count toward the 50,000. The obvious move is a pile of shell companies buying 1,600 at a time and routing them onward. It is much harder than before, not sealed. Near-term, leakage means the real cut may not be a full order of magnitude; a 75 percent reduction is the estimate he is willing to use. Longer term, DeepSeek’s engineering is “amazing” — it is crushing peers at similar compute — but it is not at OpenAI or Anthropic level, and its compute cannot scale the way theirs can. They will have to be not just better, but way better.

Is a huge compute deficit compatible with the new test-time-compute push? Patel’s first correction is that the method is far more compute-hungry than the “training is dead” slogan. Building a model that can use test-time compute takes a lot of post-training: generate data, throw most of it away, verify that the chain of thought is actually right, train reward models. You can already ship reasoning models — o1, DeepSeek’s R1, an Alibaba reasoning model whose name he cannot recall — but the scaling-law picture matters. Data and parameter count are both seeing diminishing returns. Frontier training runs are already billions of dollars and, next year, probably tens of billions, for a logarithmic step. Test-time compute is still on the bottom rung: hundreds of thousands, then millions, tens of millions, hundreds of millions, billions. The deficiency does not really bite until the billions-plus scale. There are rungs China can still climb with better engineering. The flip side is that Anthropic, OpenAI, DeepMind, xAI, and startups including Mira Murati’s are climbing the same ladder as fast as they can.

Inference is the tighter bind. He puts a GPT-4o-class query around 20 cents and an o1-class query around $6 — a dramatic gap, even though the more expensive call is still far cheaper than a person and it scales. Anthropic and OpenAI, he says, take something like 75 percent gross margins on inference; a lot of other firms sit at 10 to 20 percent. Even at those thinner margins, amortizing a $5 billion model implies about $10 billion of inference revenue and $8 billion of hardware cost. Because hardware depreciates over years, that can mean buying on the order of $30 billion of kit in the year you scale — well past the country caps. China can probably deliver impressive reasoning results. Whether it can scale and sell them is the open question. Software labor is much cheaper there, and platform SaaS never really worked because customers could just build it inside. Effron’s follow-on: AI coding is pushing the same logic into the United States. Data-migration pain, the old reason nobody upgraded, is something a pile of agents can chew through in a few days. Patel agrees it is a real threat to SaaS. Klarna has said the quiet part in public.

UAE’s G42 is the sample of how influence gets written into the building. The sovereign-AI flagship, funded with UAE money, had a Chinese CEO and faced allegations of CCP links and a cutoff. Media coverage went quiet for about six months; the next story was a G42–Microsoft partnership. G42 now talks constantly with the U.S. government and is deploying GPUs in Europe and the United States, not only at home. Through hyperscaler partnerships, Patel says, America can enforce a sphere of influence over who trains what. He gives the incoming Trump administration a 95 percent chance of keeping the rules, dumping the human-rights language, and swapping in economic sphere, energy, and weapons purchases. On the “Goldilocks” problem — starve China of frontier training without forcing a fully domestic stack — he is cautious. Ben Thompson’s line is that if the controls either kill China’s semiconductor industry or make it competitive, action on Taiwan becomes more likely. Patel says he is talking out of his depth on that geopolitics, but the zone he wants is “behind, with hope of catching up,” not a complete ban and not an open tap.

GPT-4, in 2022, trained on on the order of 20,000 A100s. The current step is 100,000-class clusters in one place: xAI has built 100,000, Meta has built 100,000, and both are going larger. Anthropic has 400,000 Trainium chips coming this year. The GPU is the H100 — about 5× the count and 3× the performance per chip, or roughly 15× the compute — and a cluster does not become a model the day the last rack lands. Experimentation, training, post-training, and safety each take months. An H100 is about $24,000 for the chip and $40,000 to $45,000 all-in once you add networking and the rest of the building. A hundred thousand GPUs is about $5 billion. Elon’s round was $6 billion, and most of it went into the cluster, plus data, people, and the work of turning a factory into a hall. Patel would be surprised if xAI had a leading model even that quarter. The next generation being built now is not a $5 billion object. It is more like $15 billion. Startups rent: Amazon may spend $5 to $10 billion of capex and charge $3 to $4 billion for a year. Oracle, he says, is spending $10 billion-plus this year on data centers and GPUs for OpenAI. Sam Altman’s actual strength, in this telling, is that he is a “god-tier fundraiser,” because the raise never stops.

Hyperscalers already know how hard electrical infrastructure, substations, networking, and silent chip failures are at this count. xAI could not find a data center on its timeline, so it did the Elon version: it bought a closed appliance factory in Memphis, next to a gigawatt-scale natural-gas plant, a gas main, a water-treatment facility, and a garbage dump. People from xAI who actually go there, Patel says, will tell you the place kind of sucks, though the wings and barbecue are good. They tapped the gas line and stood up their own generation, including mobile generators; they are upgrading the substation; they dropped in Tesla battery packs; they have filed permits to build a proper on-site combined-cycle plant, aimed at the million GPUs Musk talks about. Patel will not bet against him. Cooling is water, plus a field of rented chillers — some of them restaurant-grade containers — sitting outside.

GPU training is a brutal load for a grid. Idle the chips for a gradient update and power falls toward zero, then slams back toward 100 megawatts. That shape can blow up a plant. Elon’s answer was batteries and improvisation. Meta’s accidental contribution was a piece of open-sourced code whose flag is literally “Power Plant no blow up”: during a weight exchange, keep the GPUs busy with fake matrix multiplies so the draw stays flat. Training clusters present a square power curve, not a round one — the profile grids hate most. Conventionally built halls are still going up; what the industry wants is a straight line. GE Vernova’s gas turbines are sold out two to four years; substation gear is sold out four. In parts of the Midwest, moving power from a new plant to a customer can cost more than generating it. Google and Amazon still cling to green pledges; Microsoft sits in the middle; Meta and xAI are willing to go faster on gas. Meta is aiming at two gigawatts in Louisiana alone, almost all gas — against three gigawatts of global capacity at the start of 2024. Patel’s own line is that America should try to win harder, not to make the other side lose: upgrade the grid, relax environmental process, and pair solar, wind, and a little battery with backup gas.

Chip count is a bad ruler because power per chip is racing — A100s around 400 watts, H100s 700, Blackwell 1,200. Energy is the fairer metric. Today’s 100,000-GPU clusters are about 150 megawatts. What is being built now is in the 400 to 500 megawatt class; 2026 looks like a gigawatt; Meta is trying for two gigawatts in one Louisiana site by early to mid-2027. From 20,000 400-watt A100s, on the order of 20 megawatts in 2022, to two gigawatts in 2026–27 is two orders of magnitude in about five years. Asked how much OpenAI will have raised by the end of 2027, Patel says it is hard and then says north of $100 billion: o3, another reasoning model, maybe GPT-5, revenue up, sovereigns in the mix. Anthropic and xAI ride similar curves at a lag.

On what else test-time compute will eat, he likes OpenAI’s five-level ladder more than the startup scene’s “chatbots, then agents” slogan. The method is about verifying outputs. Computer use depends on synthetic-data pipelines and test-time compute; agents depend on reliability high enough to chain steps. Stack them and you get a system that works a computer without every API hand-coded. Coding is not software engineering. Customer service and doc search are obvious, but RAG is “really bad” for agents because you cannot verify that retrieval actually pulled the right thing. Any information task, he does not see a reason it fails — with the caveat that 2025–26 is a different world from 2030.

Open-source models have closed a lot of the GPT-4o-class gap, and the regulation treats today’s open weights as fair game while assuming the closed-open gap then widens. Patel agrees. Meta will ship Llama 4 and it will be better than GPT-4o, but open models still lack a voice mode and the inference cost Anthropic and OpenAI have already squeezed. GPT-4o is much smaller than Llama 405B in active parameters. Reasoning models are unlikely to be fully open. Meta has an internal code model trained on its own codebase, stronger than Llama at coding, and it is not being released. “If Meta has the best model in the world, they will not open-source it.” He would take that bet. Open source’s real yield is talent and a community that will tell you that your RL is lobotomizing the model — feedback Llama 2 and Llama 3 actually got. He is not sure there will be an open o3 equivalent in six to nine months; look at ARC. Google already has a reasoning model; Anthropic is rumored to have one internally that is better than o3. And most of Meta’s GPU buys are for Instagram recommendation, more than half of them, so you cannot read Zuck’s unit count as an LLM budget. Benchmarks in the 70s and 80s are noise. Ten percent versus 50 percent on a hard task is the gap that matters.

If he could jump two or three years ahead, Patel would not only ask how the 100,000-GPU models scored. He would ask how the new training recipes change hardware, networking, and whether halls must sit next to each other. A year ago he would not have known to ask about reasoning the way people now mean it. Synthetic data looked like a distillation trick, not the core of post-training. The live questions are ugly and specific: average sequence length at inference, what search looks like in test-time compute, whether synthetic-data generation can be split from the RL loop (he thinks not, because the generator must keep being updated), and whether that update is every 30 seconds. Llama, as he remembers it, updated gradients on the order of every 15 seconds. The ratio of network time to compute time is what tells you how to design the fabric — and whether good fiber between two sites is enough, or whether you still need a collocated, high-bandwidth hall. He suspects only OpenAI, Anthropic, and SSI really know.

Hardware startups, in his taxonomy, come in two waves. The old names are Cerebras, Groq, Mythic, Tenstorrent. The new ones include Etched, Positron, and MatX, plus a few still in stealth. Each has a gimmick, because you will not beat NVIDIA by doing the same job better on every axis, and NVIDIA is still moving the architecture every generation. The deeper trap is that the research landscape is NVIDIA-shaped: an idea that is 10× fewer operations but 10× slower on a GPU does not get pursued, because time is what counts. Train on NVIDIA, deploy on the GPUs you already bought, and a beautiful on-chip, high-bandwidth, tiny-parameter design can lose the chicken-and-egg fight. Hyperscaler chip teams still talk as if training chips and inference chips are the same thing, because the workload will not sit still. NVIDIA is pitching a 10–15× cost step on Blackwell (30× at GTC, with the usual massage); that alone can erase a startup’s projected 5–10× edge against Hopper.

Trainium, he has called the Amazon Basics of chips: a TPU-like topology with some original work, including stochastic rounding, cheaper and worse on most metrics, better on memory bandwidth and capacity per dollar. Anthropic has barely used Trainium so far — mostly Hopper at Amazon and Google, plus some TPU — and is now putting all its eggs in that basket. Some of that is the price of Amazon’s investment and distribution; some of it, he thinks, is a real performance-per-dollar bet they could have declined. OpenAI building its own chip makes sense only if revenue is already on a steep curve: if this year’s run rate is north of $10 billion and the next year north of $20, the cost stack justifies it. A few hundred million to a billion a year is also a call option and a 10 percent NVIDIA discount that can pay for the team. Jensen’s attitude toward every customer that tries this, Patel says, is “good luck, we’re going to crush you.” So far that has worked.

Enterprise AI, Patel thinks, has “really decent legs” once synthetic-data and reasoning pipelines can turn messy internal data into verified, company-specific supervision. Fireworks and Together sit at the top of a crowded inference-API field because they built their own engines, faster than vLLM or TensorRT-LLM, and they can keep riding Meta’s and China’s open weights. Mosaic’s “train your own” pitch went out of fashion when general models lapped BloombergGPT; Databricks lost its footing for a while. The same stack — take an open pretrained model, build reasoning and synthetic-data recipes on the customer’s data, then serve it — could make training-for-hire make sense again. Adaptive ML has been selling a white-glove version of that for about a year. Together and Fireworks, in his view, become the inference partner even when the GPUs live in a virtual private cloud on Amazon.

What made CoreWeave work is three things, not just Jensen’s blessing. First, allocation: NVIDIA’s biggest customers are all trying to build chips, so Jensen made small investments in four clouds to spark competition. In a tight market those GPUs rented at high prices and started the engine. CoreWeave now has 200,000-plus GPUs, billions in revenue, a last-round story around $20 billion, and a likely IPO above that. Early in a GPU’s life the rent is fattest; later the market floods. Second, speed. Hyperscalers customize servers with tens of thousands of people. CoreWeave takes Jensen’s reference design, tweaks memory, storage, and network a little, and can rack parts as soon as TSMC packages them. Amazon’s Blackwell rollout, he says, is later precisely because of that customization. Third, capacity. CoreWeave took double-digit-rate loans when that looked insane, then went to crypto operators, offered to buy one, and ended up retrofitting halls — including a 200-megawatt gas plant sitting in the middle of seven buildings. Power-usage effectiveness goes from a hyperscaler-like 1.1 to about 1.4, burning tens of extra megawatts on cooling and conversion. “Screw it, we need GPUs now.” The organization is small, and its GPU-rental software is, Patel says, literally better than Amazon’s and Google’s: a clean slate, no two-thousand-person storage org to morph.

Second-order bets he finds interesting: networking and optics as density and context length climb; solid-state transformers, because the conventional ones are sold out four years; carbon sequestration wrapped in a data-center story for the day Meta and xAI want to look green again; storage for video models; and unsexy software infra that sits on both the hardware and the model. Adaptive’s French founders trained Falcon in the UAE, then bet on synthetic data and reasoning before it was fashionable. Y Combinator suddenly funded a dozen AI-for-chip-design companies; Patel even invested in one. EDA already multiplied a profession that has not grown much in two decades; Cadence, Synopsys, Siemens (Mentor Graphics), NVIDIA, and Google are all in. AI will not design a chip by itself, but verification is half the cost of a design, and cheaper design is how you get from “the TAM must be billions before an ASIC is worth it” to more specialized silicon. NVIDIA, he notes, is already willing to be merely okay at CNNs because it chose Transformers.

Quickfire: RAG is overhyped; other forms of semantic search and retrieval are underhyped. Model progress in 2025 will be more than in 2024. His odd prediction is that inequality fears are overdone and that blue-collar work does well until robotics arrives, with the world’s poorest also gaining. A reasoning GPT-5 query, he expects, will cost dozens of dollars if not hundreds — and still be cheap for the work it does. His own eval is semiconductor manufacturing, a literature he knows; even now, Claude is the only model that can get a lithography patterning stack somewhat right, because almost nobody publishes the real stack. He is watching distributed training (Nous, Prime Intellect) and the multi-agent reasoning shops that never pivoted with the hype, Synth Labs and Adaptive among them. The serious product is the SemiAnalysis newsletter and the institutional data-and-consulting business; the unserious one is his Twitter, Dylan522P, where an “arsenal of democracy” joke can sit next to a ceasefire two days later.