DeepSeek's $5 million training claim and the five-year bet on compute

Lex Fridman sits down with Dylan Patel, founder of the semiconductor research firm SemiAnalysis, and Nathan Lambert, a research scientist at the Allen Institute for AI and author of the Interconnects blog, to unpack the DeepSeek moment that rattled labs and markets in early 2025. Nathan starts with the naming mess that confused almost everyone. DeepSeek-V3, released around December 26, is a mixture-of-experts instruction model in the ChatGPT mold. DeepSeek-R1, released on January 20, is a reasoning model in the spirit of OpenAI's o1. Both start from the same pretrained checkpoint, DeepSeek-V3 Base, a next-token predictor that will finish your sentences and is much harder to use than a chat product.

From that single pretraining run, DeepSeek ran two different post-training recipes. One path is the familiar stack of instruction tuning and reinforcement learning from human feedback, which produced the helpful V3 chat model, competitive with GPT-4 and Llama 405B. The other path is large-scale reinforcement learning on verifiable tasks, which produced R1. Pretraining is the expensive, one-shot compression of internet text; post-training is where formatting, preference, and now reasoning get layered on. That split is why R1 could look cheap: it did not need a new base model from scratch.

Open-weights, Nathan stresses, is not open source. Weights are large matrices you can download and run on a machine with no internet, which puts data in the user's hands rather than an API host's. True open source, as Ai2 practices it, also means training data and training code, because "data processing, data filtering, data quality is the number one determinant of the model quality." DeepSeek did not release those, but R1 carries an MIT license with no commercial or use-case limits and with permission to distill from the outputs. Llama's license is tighter, including a requirement that derivatives keep the Llama name. Nathan's punchline is that you cannot ship a cheap copy of Llama and pretend it is yours, but you can with the Chinese model.

Dylan attributes most of DeepSeek's efficiency to two architectural choices, with a long tail of implementation detail on top. Mixture-of-experts lets the model grow a huge embedding space while activating only a slice of it. DeepSeek-V3 has on the order of 600 billion parameters and activates about 37 billion per token, versus Llama 405B, which fires every parameter. Mixtral-style public MoEs often activate two of eight experts, a quarter of the model. DeepSeek activates eight of 256, a few percent. That sparsity is the efficiency, and also the engineering nightmare: experts live on different nodes, and a batch that all routes to one expert leaves the rest of the cluster idle.

Classic MoE training adds an auxiliary loss to stop the model from collapsing onto a few experts, which fights the actual next-token objective. DeepSeek instead updates a routing bias after each batch so later batches spread load more evenly. The other pillar is multi-head latent attention, MLA, a low-rank compression of the KV cache that Dylan and Nathan put at an 80 to 90 percent cut versus vanilla attention. Making MLA work with rotary positional embeddings required extra structure to carry position, which is the sort of trick Nathan associates with closed frontier labs. DeepSeek wrote it down.

Export-limited H800s gimp chip-to-chip interconnect, so DeepSeek did not just call NVIDIA's NCCL library. The team scheduled streaming multiprocessors by hand, flipping groups of cores between compute, allreduce, and allgather, in PTX that sits close to the instruction set. Meta has described custom NCCL for Llama 3 without publishing the details. Dylan's line is that necessity is the mother of invention: the low-level work is tightly bound to this architecture and size, and NVIDIA's generic libraries will always trade some performance for portability. For a compute-constrained lab, that leftover performance is the whole margin.

The dashboard that matters is still the loss curve, and MoE plus FP8 training make it spike for reasons nobody fully understands. Nathan's cautionary tale from Ai2 is the Microwave Gang subreddit, where posts are long strings of the letter M and comments are "beep beep." A model that expects English after an M sees a wall of M's and the loss explodes. Mature data filters catch that; later spikes are murkier. Dylan describes dinners with lab friends who check their phones every ten minutes, not for messages but to see whether the run has blown up. Some spikes recover. Slow spikes creep, then detonate, and you roll back hours.

Every frontier lab burns failed runs. A bad one can cost a week to a month, and that is the price of pushing infrastructure. Small ablations on a few GPUs become, eventually, a YOLO run: take every resource, pick the architecture you think will work, and go. OpenAI's original GPT-4 bet, Dylan argues, was more YOLO than today's supposedly failed runs, because the company put essentially all of its compute into a new MoE for months when Google still did not believe in the architecture. Post-training is, for now, the cheaper place to YOLO. That will not last.

DeepSeek sits under High-Flyer, a quantitative trading firm that already ran GPUs at scale. In 2021, before export controls, it claimed a 10,000-A100 cluster, then the largest in China. CEO Liang Wenfeng owns a bit more than half the company and talks, in translated interviews, like someone who wants China to lead an open AI ecosystem and will not switch to closed source. The V3 paper credits 2,000 H800s for pretraining only, not R1, not instruction RL, not research. Accepted practice, Dylan says, is two to four times the final-run compute in experiments. SemiAnalysis's estimate is closer to 50,000 GPUs shared with the fund. Meta trained Llama 3 on 16,000 H100s while buying on the order of 400,000 GPUs company-wide.

The H800 kept H100-class FLOPS and cut interconnect, which is exactly the axis DeepSeek learned to work around. After the U.S. dropped the interconnect cap and regulated only FLOPS, NVIDIA shipped the H20: about one-third the H100's paper FLOPS, closer to half or 60 percent in the real world, with more memory bandwidth and capacity. NVIDIA sent on the order of a million H20s to China last year, roughly a fifth of GPU volume, then canceled remaining H20 and B20 orders, which Dylan reads as anticipation of a tighter ban. H20, he argues, is a better chip than H100 for memory-heavy reasoning inference.

Dario Amodei's case for controls is that super-powerful AI around 2026 would confer military advantage, so a democracy should keep the densest compute. Nathan's narrower version is easier to hit: you will not stop China from training focused models on a few thousand GPUs, but you can cap how much AI gets served. Training a model, Dylan adds, does almost nothing by itself; implementation at inference scale is what turns a checkpoint into growth or weapons. Nathan treats language models as a form of AGI already and adds a few years to CEO timelines, landing near 2030. Dylan expects capability thresholds around 2027 or 2028, then a long gap before cost, power, and chips let that capability run through an economy.

If you do not believe AI transforms society in five to ten years, Dylan thinks export controls guarantee China wins. They starve U.S. chip firms of a growth market while China has more STEM graduates, more power plants, and steel and aluminum mills that dwarf U.S. industry. Stargate at full build is about two gigawatts, still smaller than China's largest industrial sites. If you do believe short timelines, compute is the only lever, because talent already flows both ways and China can pour concrete faster than America can permit substations. DeepSeek had only just met China's second-in-command, not Xi Jinping, when a trillion RMB, about $160 billion, in AI subsidies landed. Lex wonders whether this is the start of a cold war. Dylan's worry is that locking China out of leading chips raises the odds it decides nobody should have them, with Taiwan in the middle.

Most of the world's leading-edge chips now run through TSMC's foundry model: it designs nothing and manufactures for NVIDIA, Apple, Qualcomm, and, increasingly, everyone else. A profitable leading-edge fab costs north of $30 to $40 billion, and that is a starter block. Twenty or thirty years ago, dozens of companies built their own advanced chips; they fell away as fab cost compounded. AMD nearly went bankrupt and sold its fabs to Mubadala, which became GlobalFoundries. Even Intel's latest PC parts use TSMC process. NVIDIA, Dylan notes, is the only billion-dollar semiconductor company born in the foundry era.

Taiwan's version of the story is cultural as well as economic. Top National Taiwan University graduates go to TSMC for starting pay around $70,000 to $80,000, while their U.S. peers chase hundreds of thousands at Google or OpenAI. When an earthquake hits, TSMC does not call people in; the parking lot fills and the fab behaves like an ant colony. Morris Chang left Texas Instruments after he was passed over for CEO and built that machine. The United States did this once. Intel brought strained silicon, high-k metal gates, and FinFET to market first. Dylan's diagnosis is that the culture broke, not the PhDs still in Hillsboro.

High-volume manufacturing can move. Leading-edge R&D cannot. Dylan names three places on earth doing it: Hsinchu in Taiwan, Hillsboro, Oregon, and Pyeongtaek, South Korea. TSMC's Arizona site holds about 20 percent of its five-nanometer capacity and still depends on Hsinchu. "If I had a few missiles," he says, he would aim at those research centers, not the White House. Full U.S. insourcing would take a decade and a trillion dollars. The CHIPS Act is about $50 billion over several years; Dylan puts China's semiconductor subsidies near $200 billion a year. Export controls have slowed China's five-nanometer work and, he argues, accelerated its trailing-edge analog and power chips, the unglamorous parts inside cars and fridges.

Chips for AI, Dylan says, have three axes: FLOPS, memory bandwidth and capacity, and interconnect. Pretraining is a FLOPS game; reasoning is a memory game. Attention builds a KV cache, a compressed record of prior tokens that must stay in GPU memory for the whole generation. Prefill on a long prompt is parallel and relatively cheap. Decoding is autoregressive: each new token rereads the model and the growing cache. Reasoning models emit tens of thousands of output tokens, so the cache stays resident and the number of users you can batch on an eight-H100 box collapses. That is why H200s, with more memory, went nearly out of stock after R1, and why an H20 can beat an H100 on this job.

R1's API is about $2 per million output tokens against $60 for o1, a gap on the order of 27 times. Dylan splits it. OpenAI's inference gross margins are north of 75 percent, a four-to-five-times factor the company uses because it was first, even though training still leaves the firm unprofitable overall. Together, Fireworks, and similar hosts serve the same open R1 weights at five to seven times DeepSeek's price and still make money. The leftover gap is real systems work: MLA, the sparse MoE, and training kernels that were never open-sourced and likely transfer to serving. DeepSeek's own app still cannot meet demand. Signups freeze; throughput crawls. Even at SemiAnalysis's 50,000-GPU estimate, the lab is not Microsoft-plus-OpenAI.

Dylan does not think the CCP is subsidizing DeepSeek the way it backs some other Chinese labs. The hedge fund is the subsidy, and open-sourcing is a recruiting tool. He also doubts the release was timed to short NVIDIA: V3 landed the day after Christmas, a terrible date for a splash. What DeepSeek does have is speed. Anthropic, Nathan notes, spends months on safety review; Dario has said Claude 3.5 Sonnet sat nine to ten months after training, and San Francisco rumor is that Anthropic already has something stronger than o3 that it will not ship because raw chains of thought are "scary." DeepSeek ships. Nathan's uncomfortable conclusion is that if the U.S. stops open-weighting frontier models, some other country will, and a shocking $5 million training number is within reach of many entities at 100 times the cost.

Open weights do not phone home, but they can still carry a worldview. Dylan worries less about a classic backdoor than about cultural residue: models that refuse Tiananmen Square, treat Taiwan as an "eastern province," or, more subtly, make British English feel extinct because American internet text won. Nathan walks through three insertion points for censorship. Facts are almost impossible to scrub from pretraining unless you take them off the internet. Post-training can overshoot, as when Llama 2 refused to "kill" a Python process. System-level prompt rewriting produced Gemini's image-generation fiasco even if the weights were fine. RLHF still improves math and code; the question is who sets the preference.

R1-Zero is the result Nathan wants remembered. DeepSeek took V3 Base, ran reinforcement learning on verifiable math and code with no human preference tuning, and watched reasoning behaviors appear: wait, let me check, that might be a mistake. The full R1 model adds some RLHF afterward, but the deliberation was not copied from labeled traces and, Nathan thinks, was not stolen from o1's hidden chains. Lex reads Andrej Karpathy's distinction between imitation learning and trial-and-error. Almost every shocking deep-learning result, Karpathy wrote, is type two: AlphaGo imitates experts, AlphaZero dumps the human prior, and R1-Zero's backtracking is an "aha" the human labeler would not know how to annotate. Nathan ties it back to Rich Sutton's bitter lesson: scalable learning and search beat clever human structure in the long run.

The limit is verification. Math and ordinary coding benchmarks have been all but solved; frontier math is the exception. Dylan's candidate for a "Move 37" of chain of thought is not a journal paper but computer use and robotics, sandboxes with infinite checkable tasks, from putting a ball in a bucket to spinning up accounts until one gets a thousand likes. Pretraining on the web, he thinks, will eventually be dwarfed by this kind of play. Weak models already get grade-school math right sometimes; RL is built to climb sparse rewards. Setting up the domains is the hard part. Lex's 157-second R1 sample, asking for a truly novel insight about humans, is the non-verifiable cousin: the model argues with itself, then claims people convert selfish desire into cooperation by collectively pretending that money, laws, and rights are real.

After reasoning RL, DeepSeek still ran instruction tuning and math-heavy RLHF so the model would be usable. That extra stage is an open research question: how much eloquence and philosophy transfer from verifiable training? Nathan argues Gemini 2.0 Flash Thinking, released in December and cheaper than R1, is under-discussed, and that it feels more like reasoning bolted onto a normal stack than o1's large-scale RL. Early Qwen and DeepSeek-lite reasoners felt "on rails," good at math and code and little else. During the taping, OpenAI released o3-mini. In a later insert, Lex's vibe check ranks o1 Pro first on a novelty prompt, then R1, then Gemini Flash Thinking, with o3-mini-high smart and generic on philosophy but strong for brainstorming. He still prefers Claude 3.5 Sonnet for everyday coding.

o3's ARC-AGI result was not one chain of thought. OpenAI ran on the order of a thousand parallel samples and selected among them, lifting accuracy from about 30 percent to something like 70 to 90 percent, at $5 to $20 per puzzle. Nathan refuses to call that Monte Carlo tree search without evidence; it could be majority vote, a learned scorer, or something richer. The shift still matters. Models used to be trained to be right in one shot. They are now being run as search. ChatGPT Pro at $200 a month is, Sam Altman has said, losing money, which is a statement about how much test-time compute a serious user can burn.

Dylan's cost chart is a log-scale drop from GPT-3's $60 to $70 per million tokens to about five cents for similar intelligence, a 1,200-times collapse in a few years. GPT-4 launched near $60 and is already around $2. DeepSeek, he says, is not below the GPT-3 trend line; it is the first lab to land on it. NVIDIA's stock still fell, which both guests think mixed a weekend social panic with a false reading of the $5 million figure, which omitted research, salaries, inference, and post-training. Jevons paradox showed up immediately: AWS H100 prices rose after V3, H200s vanished, and Nathan could not easily rent 16 or 32 H100s for a demo. Cheaper tokens raised demand for the chips that produce them.

Dylan's smuggling map has three layers. Legal H20s: about a million last year. Gray routing through Singapore, Malaysia, and suitcase-scale Super Micro boxes on first-class flights: perhaps 200,000 to 300,000 GPUs. And the channel that was not even smuggling until the Biden administration's AI diffusion rules: Chinese firms renting foreign clouds. ByteDance, he says, has more than 500,000 GPUs, is Oracle's largest GPU customer and Google's second, and is the reason Singapore can be 20 to 30 percent of NVIDIA's revenue despite a long moratorium on local data centers. The new rules cap how many GPUs a country can buy and how large a cluster you may rent to a Chinese company. Clusters under about 2,000 GPUs, or purchases under about 1,500, still leak. You cannot hide $10 billion of GPUs. You can hide a DeepSeek-sized training run.

OpenAI told the Financial Times it has evidence DeepSeek trained on its API, and people at OpenAI have called R1 distilled. Distillation, Nathan says, is standard: you generate completions from a stronger model and train a weaker one to match them. Meta used 405B as a reward model for 70B. OpenAI's terms of service forbid building a competitor from outputs; a ToS is not a license, and "competitor" is undefined. Nathan's ethical retort is the loop the industry is stuck in: why may you train on the internet and I may not train on your outputs? ChatGPT transcripts already live on Reddit. Ask many models who built them and they say OpenAI, including Ai2's, unless the system prompt forces "I am Tulu." In the days around the taping, people were already distilling DeepSeek into Llama because dense Llama is easier to serve than a 600-billion-parameter MoE.

Lex asks how much actual code theft happens. Ideas travel, Nathan says, because California makes non-competes weak and labs buy each other's staff; a Gemini engineer associated with million-token context joining Meta is the mechanism. Nation-state espionage is older than this industry. Company-to-company source dumps are harder. Dylan's joke about honeypots aimed at "single dudes in their 20s and 30s" is not the load-bearing part of the argument. The load-bearing part is that a closed API is a porous training set as soon as its text hits the public web.

U.S. data centers were a few percent of national power through the whole cloud era. People who run traditional facilities think 10 percent by 2030 is nuts; people at Anthropic and OpenAI think it is too low. GPT-4's 20,000 A100s drew 15 to 20 megawatts. A 100,000-H100-class cluster is about 150 megawatts, nearly 10x in two years. xAI's Memphis site, an old appliance factory on a gas line, is the largest single tightly coupled cluster Dylan tracks: 100,000 H100s plus 100,000 H200s, with mobile generators, Tesla Megapacks, and on the order of 90 containerized water chillers. Meta is in the 128,000 range for Llama 4 training; OpenAI around 100,000. Stargate in Abilene is designed for 2.2 gigawatts in and 1.8 gigawatts to chips, more than many cities. Blackwell parts go to 1,200 watts, up from Hopper's 700, and they will require liquid cooling.

Power, not chips, is the U.S. build constraint. In places such as Virginia, transmission already costs more than generation. Labs will not idle tens of billions of dollars of GPUs waiting for a nuclear plant, because electricity is under 20 percent of cluster cost and depreciation is the rest, so they pour gas turbines. Training power is spiky: compute draws hard, then allreduce idles the GPUs. Meta upstreamed a PyTorch operator that Dylan renders as `powerplant_no_blow_up`, which makes GPUs compute dummy work during communication so the plant does not see a transient. Water cooling is not only thermal; packing chips closer makes high-speed interconnect possible. Google already water-cools TPUs and, Dylan says, has the largest cluster on earth if you count several sites about 30 miles apart in Iowa, Nebraska, and Ohio, tied with fiber. Elon still wins the "one building" contest.

Google does not sell TPUs because the silicon is a Search and YouTube cost center, not a product culture. Gemma 7B is really about eight billion parameters because the TPU matrix unit is wide, so Google grew the vocabulary to match, which is why Gemma is awkward on GPUs and Llama is awkward on TPUs. JAX and XLA serve DeepMind and Search, not external customers; NVIDIA's CUDA and NCCL teams have always served outsiders. AMD's hardware is, in Dylan's view, better in places, but SemiAnalysis became one of the top bug reporters because public PyTorch on AMD is a second-class citizen next to private images for a few hyperscalers. Intel still holds one of the three leading-edge R&D sites and is losing servers to custom ARM, PCs to Apple and others, and the AI accelerator race entirely. If Samsung and Intel stay weak, the world's process future sits in Hsinchu alone.

Who is making money? Dylan's answer is rude and specific. NVIDIA and other hardware vendors are. Microsoft shows accounting profit while burying GPU depreciation. Meta prints money on recommendation systems, not on Llama. OpenAI and Anthropic raise because they have to, even if GPT-4-class models already earn billions against a few hundred million of original training cost; the research to keep riding the 1,200-times cost curve never stops. ChatGPT is OpenAI's brand, and Nathan's worry is that most users do not need the company to spend billions on the next best model when they could license Llama 5. V3 already collapsed chat pricing. Meta serves 405B, probably at a loss, and Google can give the same thing away with ads. Dylan's line is that chat is not the endgame. Reasoning, code, agents, and computer use are where OpenAI has to go, or it is finished.

The missing product is an ad that sits inside a generated answer without feeling like a scam. Meta knows feed targeting; Google knows keyword search; nobody knows how to place a measurable, non-repulsive suggestion in a paragraph or a voice reply. Perplexity, Google, and Meta will try. OpenAI and Anthropic, the guests think, are not even looking, because they have bet the company on short AGI timelines. Nathan wants the word agent reserved for systems that solve open-ended tasks under uncertainty, not Apple Intelligence shuffling photos and messages. Dylan's manufacturing analogy is the one that sticks. A fab with tens of thousands of steps at 99.9999 percent yield can still land at 60 percent, or zero. Agents chain dozens of fallible model calls the same way. That is why Waymo still uses teleoperators, and why a company whose pitch is "human backup when the agent fails" is not a joke.

Software engineering is the agent domain that already pays. SWE-bench, hard enough to need a few years of programming, went from about 4 percent to 60 percent in a year. It will not hit 100 percent, because six nines are hard, but the loop is verifiable: compile, unit test, inspect a whole repository at once. Dylan expects software costs to crash, and with them the U.S. habit of buying Salesforce-class SaaS. Chinese firms already build internal stacks because engineers are cheaper; if coding becomes cheap everywhere, custom logic wins. Old industrial software still runs on Windows XP, including ASML tools. The people who bring modern models into aerospace, semiconductors, and chemicals, he says, will be at the frontier. Nathan does not expect a job-market cliff. When Meta shipped Stories, Snapchat's exponential flattened. Programmer demand may do the same, while CS enrollment keeps rising for years.

Nathan's Tulu 3 is a fully open post-training recipe on Llama 3.1 405B: data, code, instruction tuning, preference tuning, and reinforcement learning with verifiable rewards. The name continues the post-ChatGPT fashion of mammal names; a tulu is a dromedary-Bactrian hybrid. On Ai2's ten-evaluation suite, covering chat, picky instruction following, math, code, and safety, Tulu 3 averages about 80. DeepSeek-V3 is about 79 without safety and drops when safety is included. Nathan refuses to win a leaderboard on a safety definition other people may not share. The next open-source job, he says, is to scale verifiers and copy more of what DeepSeek put in the paper. The structural problem is that open-source software has a feedback loop and open-source AI does not. Releasing weights, data, and code still leaves the next person needing a cluster, a team, and scars. Until that loop exists, he calls the work "mostly an ideological mission," one Zuckerberg is right to frame as an American standard, and one the field should cash in while the motivation is high.

Stargate's $500 billion headline, Dylan says, is not in the bank, and neither is $100 billion as capex. Phase one in Abilene is the 2.2-gigawatt site Oracle had already been building, once offered to Elon, who found it too slow and went to Memphis. Filling 1.8 gigawatts with the next two NVIDIA generations is about $50 billion of servers, plus data center, power, and rent, which is how OpenAI gets a $100 billion "total cost of ownership" for phase one. OpenAI is on the hook for $19 billion into the joint venture and had, in Dylan's telling, about $6 billion from the last round and $4 billion of debt, with talk of SoftBank putting in $25 billion after selling ARM. Oracle is paying the first $5 to $6 billion of servers itself. MGX in the UAE talks about $1.5 trillion for AI; Dylan does not know how real that is. "Elon is right," he says. "The money does not exist." It arrives if the models keep getting better. Trump's role is permits, federal land, Texas's independent ERCOT grid, and a televised number that forces every other board to keep up. Dylan does not think federal dollars are in the deal.

Asked what gives them hope, Nathan says humans will still be here in a thousand years: we are bad at distant risk and decent once it is in the room, and he is not losing sleep over an AI takeover. Dylan's darker picture is not a singleton god but a few hundred, thousand, or million people fused to AGI through brain-computer interfaces, a "techno fascism" of amplified elites. He still thinks most of that power will go to profit-seeking, which raises supply and cuts suffering. The joke they close on is a Brave New World of food tubes and infinite feeds. Dylan is willing to call that a kind of happiness. Nathan's correction is the one that lingers: scrolling holds the status quo of the world. Lex signs off with Feynman: for a successful technology, reality must take precedence over public relations, for nature cannot be fooled.