The Architecture Barely Changed. Data, Cost, and Open Weights Did.

Lex Fridman opens on the DeepSeek moment of January 2025: an open-weight Chinese lab released R1 with near-state-of-the-art performance, allegedly on far less compute, and the industry has been accelerating ever since. Sebastian Raschka refuses a simple winner. Researchers rotate between labs, he says, so no company in 2026 will hold technology no one else can reach. The scarce resources are budget and hardware, not secret ideas, and he does not see a winner-take-all.

Nathan Lambert dates the recording to a moment of organic hype around Anthropic's Claude Opus 4.5, which followed Google's Gemini 3 by weeks and then overtook it in the conversation even though Gemini 3 had looked like Google's chance to cash structural advantages. Ideas flow freely, he argues, but effort and culture do not. Anthropic is known for betting hard on code, and Claude Code is working. That still leaves what he calls ominous technology from China: DeepSeek kicked off a movement the way ChatGPT kicked off chatbots, and a pile of Chinese tech firms now ship strong frontier open-weight models.

Lambert's ranking is the one that stuck. DeepSeek is losing its crown as China's preeminent open model maker to Z.ai's GLM line, MiniMax, and Moonshot's Kimi, especially in recent months. Raschka adds that DeepSeek did not get worse: others copied its ideas, including Kimi's similar architecture, and leapfrogging means the newest model is often the best. The business logic is colder. American tech firms will not pay Chinese APIs for security reasons, so open weights become a way into the U.S. AI spend. MiniMax and Z.ai have filed IPO paperwork and chase Western mindshare; DeepSeek, incubated by the hedge fund Highflyer, stays quiet about what the models are even for.

Asked who won 2025 and who wins 2026, Lambert will not bet Gemini over ChatGPT even though last year's momentum sat with Google. Gemini was climbing from Bard, and he gives Google credit for powering through organizational chaos. OpenAI still looks chaotic and still lands things. His mixed review of GPT-5 includes an underrated design: a router that keeps most users off the expensive thinking path and, he suspects, saved a fortune in GPU cost. Personal taste and mass-market differentiation, he warns, are easy to confuse.

For 2026 he is willing to be spicy. Gemini should keep closing on ChatGPT because Google can separate research from product at extreme scale, while OpenAI chases the high-impact thing in startup style. Anthropic should keep winning software and enterprise. On infrastructure, TPUs matter because NVIDIA's margins are "insane" and Google can design the stack from chip to data center without paying them. If a new paradigm arrives, though, he still expects it from OpenAI — Deep Research, Sora, and o1 all came from that research org. A lot of this year, he thinks, is scale and picking low-hanging fruit.

None of the three uses Chinese models day to day, which Fridman flags. Lambert's answer is blunt: U.S. models are better, and he will pay for the marginal intelligence. OpenRouter and Perplexity make DeepSeek easy to try; after a look he does not go back. Analysis he has seen suggests Chinese replicas often run on fewer GPUs, so they are slower and fail differently — a plausible export-control artifact. Raschka programs with a Codeium plugin that stays in the repo without taking over; Fridman splits Cursor and Claude Code to practice programming in English. Muscle memory, memory features, and one dumb failure are what actually move people between apps.

The open-weight map they recite off the cuff is crowded: DeepSeek, Kimi, MiniMax, Z.ai, Mistral, Gemma, gpt-oss, NVIDIA Nemotron 3, Qwen, AI2's fully open OLMo, LM360, Swiss Apertus, Hugging Face SmolLM, Stanford's Martini project. Chinese open models tend to be large Mixture-of-Experts systems with higher peak performance; U.S. and European favorites have been smaller, though Mistral Large 3 and teased 400-billion-parameter MoEs are set to change that in early 2026. Llama is not on the list. "RIP Llama," Fridman says, and nobody claims it was an accident. Raschka's standouts are DeepSeek's architecture tweaks bookending the year and gpt-oss-120b as the first open-weight model really trained with tool use in mind — search, a Python interpreter — which the ecosystem is not fully ready to trust on a local machine.

Licenses explain some of the Chinese uptake. Llama and Gemma attach strings, including user-count caps that can force a report of financials to Meta or Google. The Chinese weights, Raschka says, often ship under unrestricted open-source licenses: no catch. American startups then host those weights and sell tokens. Sam Altman, in Lambert's telling of a gpt-oss session, put the closed-lab motive equally plainly: release the weights, use other people's GPUs, keep the distribution. For enterprises the unlock is customization — law, medicine, internal data that never leaves the building.

Raschka then walks the transformer from GPT-2. The "Attention Is All You Need" paper had an encoder and a decoder; GPT kept the decoder, predicting one token at a time through embeddings, attention, a fully connected block, and normalization. Mixture of Experts splits that expensive feed-forward layer into many experts — he uses 256 as the picture — and a router activates only a few per token, so parameter count can explode without a matching jump in FLOPs per forward pass. Dense models such as some OLMo variants keep one fully connected path always on because MoE is harder to train and can collapse.

How many new ideas sit between GPT-2 and gpt-oss-120b? Fewer than the hype implies. Multi-head attention became grouped-query attention; LayerNorm became RMSNorm; activations swapped the way sigmoid once became ReLU. "It's not really fundamentally that different," Raschka says. His bonus materials start from a 124-million-parameter GPT-2 and add components until the net is OLMo or Gemini 3. The turbulence, both guests argue, is elsewhere: pretraining, mid-training, and post-training as separate stages; ChatGPT as GPT-3 plus supervised fine-tuning and RLHF; NVIDIA FP8 and FP4 letting labs push more tokens per GPU. Text diffusion and Mamba exist, with trade-offs. State-of-the-art is still an autoregressive transformer in the GPT-2 family.

Lambert starts with the technical definition. A scaling law is a power-law between compute-plus-data and held-out next-token accuracy, and that relationship has held across many orders of magnitude of pretraining. o1 then made inference-time scaling famous: generate more hidden tokens before the first word of the answer. Less famous, he says, is that reinforcement-learning training also scales — log compute, linear eval gains. The three axes are pretraining size, how long you can run trial-and-error RL, and how much inference you spend per problem. He is bullish on all of them. The low-hanging fruit of the last year was RL with verifiable rewards plus that inference-time step change, which is why models now go off for seconds, minutes, or hours and why tool use and software engineering jumped.

Pretraining has not died. It has gotten extremely expensive, and a bigger pretrained model implies a bigger model you must serve. GPT-4-class systems were rumored around a trillion parameters; later training got more efficient and models reportedly shrank because serving hundreds of millions of users is the real bill. DeepSeek's famous pretraining figure was about five million dollars at cloud rates. OLMo 3's paper, section 2.4, put cluster rental including engineering failures and extra seeds at about two million. One to ten million trains a model; recurring serving is billions. A thousand GPUs can cost a hundred thousand dollars a day, and these companies may have millions of GPUs sitting around.

People *vibe* that pretraining is dead, Lambert says, but that is not the practice. AI2's November 20 deadline bought five days of RL on a 30-billion-parameter model; a December release let RL run another three and a half weeks and the model got notably better. You cannot leave a run forever: researchers keep inventing, you redo pretraining, you post-train, you ship, you safety-test. Raschka's accounting is the one product people need. Pretraining is a fixed cost that sits in the weights forever; inference scaling is per query. If you will replace the model in six months, another fifty or a hundred million on pretraining may lose to cheaper test-time compute. ChatGPT's user base is why GPT-5 went smaller. In a world of infinite compute you twist every knob. In this one, Claude Opus 4.5 is evidence that other ways to unlock performance currently pay better.

Synthetic data, Lambert says, is broader than people think. DeepSeek, AI2, and Meta all built OCR stacks to pull trillions of candidate tokens out of PDFs that were never clean text. Pretraining sets are measured in trillions: five to ten for smaller research models, fifty documented for Qwen, rumors of a hundred at closed labs. The funnel is huge; the trained slice is small. ChatGPT's best answers are also synthetic data now, which is a different object from early hallucination dumps. Raschka's picture of pretraining is next-token prediction on a vast mix, then mid-training that specializes — long-context documents last, because there are not many of them and catastrophic forgetting is real — then post-training as the skill unlock.

Dataset mixing has become a small science. Sample GitHub, Stack Exchange, Reddit, Wikipedia; train tiny models; run linear regression against your evals; get a mix. Change the evals and the mix moves. A lot of OLMo 3 was new reasoning sources for math and code, then that mixing loop. If you join a frontier lab and want impact, Lambert's advice is not to invent the next o1. "The best way to do it is just find new data that's better," or make infrastructure 5% faster for everyone. The glamorous algorithmic breakthroughs happen; most of the contribution is data and systems. Training data is also among the closest-guarded secrets, for legal reasons, including teaching models not to recite sources.

Then the case almost everyone forgot. In 2025 Anthropic lost in court and owed authors $1.5 billion. Buying thousands of books and scanning them was cleared, Lambert says, because the company paid for the books. Torrented copies were not, and that is the path that produced the bill — a mind-boggling number that came and went in the Valley conversation. Even a purchased Kindle or Manning title sits in a gray zone. Big firms now approach private companies for proprietary data, and those companies are getting more protective because the data *is* the moat. Pharma, law, and finance will eventually hire frontier-lab people to train in-house on clinical trials and other corpora nobody else can buy. Pretraining scaling, on that view, is very much alive — just moving off ChatGPT and onto vertical models.

The biggest post-training idea of 2025, Lambert says, is reinforcement learning with verifiable rewards. Fun academic fact: his Tulu 3 team coined RLVR before DeepSeek; DeepSeek did the training breakthrough that scaled it. The loop is simple. The model generates answers, you grade them right or wrong on math, code, or other checkable tasks, and accuracy is the reward. Classic RLHF optimized a learned preference model; swapping in verifiable scores let optimization run at much larger scale. Rubrics and LLM-as-a-judge try to push the same idea into vaguer domains. Anthropic's older name for a cousin of that loop was reinforcement learning from AI feedback, from the Constitutional AI paper.

You give the question and the answer and stay mostly hands-off. In practice the model writes step-by-step derivations, uses more tokens the longer you train — the DeepSeek R1 paper showed responses growing — and sometimes says it got something wrong and tries again. Raschka finds that "aha moment" beautiful. Lambert is dryer: the moments are a bit fake, because pretraining already saw the whole internet, including math-lecture transcripts where someone says "you try this, oh I messed this up." RLVR is very good at amplifying those behaviors because they help the model think longer and check its work. The beauty, he still grants, is that the amplification makes final answers better.

Raschka's kitchen example is Qwen 3's base model on MATH-500: about 15% accuracy, then 50% after fifty RLVR steps in minutes. You cannot have learned new math in fifty steps, he says; you unlocked what was already there. Lambert complicates it. Two papers this year, one of them his, document Qwen contamination: change the numbers in a word problem, keep the words, and Qwen will emit a high-precision decimal without tools — something a tool-free language model should not do unless it saw near-identical items and a calculator. How much of the RLVR literature on Qwen is real learning versus formatting a memorized solution is still not a controlled result. Even so, RLVR and inference-time scaling reinforced each other and changed how the models feel to use.

Lambert's recipe for a reasoning model starts before the RL loop. Mid-training has to supply carefully curated reasoning traces — language that breaks a problem into intermediate steps — or later verifiable RL has nothing to amplify. Then you choose harder problems as the model saturates easy ones. GRPO, the algorithm everyone names, assigns reward by how a completion compares to other answers to the *same* problem; if every sample is identical, the signal is zero. That is why labs talk about scientific domains and nastier software tasks where the model is not already at 100%. RLHF remains the finishing pass: organization, tone, the quirky personality some people want and the markdown lists some people hate. Style, format, and method of solution are tightly linked in training, which is why RLHF can still help at math, but verifiable domains are the more direct path.

Compute for RLVR has only gone up. Ilya Sutskever is famous, Lambert says, for using similar compute on pretraining and post-training; the hardware story is different. Pretraining is compute-bound, dense matrix multiplies, all GPUs talking. RL is memory-bound: long generations, attention memory growing quadratically, sequences that can take an hour if the deployed model thinks that long. In wall-clock days the RL runs are approaching pretraining; they probably use fewer GPUs at once. Labs have a rule of thumb not to let pretraining last more than a month, because large runs fail catastrophically. Plan a cluster for two months, die on day 50, and the opportunity cost is brutal. GPT-4 was the ultimate YOLO run — three months, everyone shocked it worked. People are more incremental now.

Raschka's split is that preference tuning saturates. Relatives asking which laptop to buy get different correct answers depending on battery versus RAM; RLHF averages those preferences and then there is no reason to keep spending. RLVR can keep climbing into harder problems, so the long-term budget belongs there. He expects an RLVR 2.0 that scores the explanation, not only the final answer — process reward models, plus value functions that score every token. Lambert notes both are largely unproven in the reasoning era, and the seminal RLHF scaling paper is about *over*-optimization. o1 and DeepSeek published plots where log RL compute gave linear eval gains. There is no such plot for RLHF. That line, he thinks, will define how labs allocate the extra 10x.

Raschka's curriculum is concrete. Implement a small model you can run on one GPU, not a personal assistant. Learn attention, pretraining, and supervised fine-tuning until the code either runs or it does not — figures can lie, running code cannot. Then open a Hugging Face config, about a hundred human-readable lines of layers, attention flavor, position encoding. Start from GPT-2, add those components, load pretrained weights, and match the Transformers library's outputs. Alignment is a verifiable reward. Reproducing OLMo 3, he spent a day stuck on RoPE with a YaRN extension and custom scaling; the struggle was the lesson. Transformers implements hundreds of models and is the wrong place to *start*, but it is a working reference to work backwards from.

That struggle is the same Goldilocks problem they had already hit in a survey of 791 developers with ten-plus years of experience. Both juniors and seniors ship AI-generated code, yet seniors are *more* likely to have over half of shipped code come from a model, and about 80% find AI somewhat or significantly more enjoyable. Raschka's fear is the other side of the chart: how do you become an expert if you never do the thing? Math textbooks teach more if you attempt the problem first. An LLM always sitting there makes struggle optional, so his rule is dedicated offline study — two hours a day — and models for the rest. Lambert's research advice is the same shape: go narrow after the fundamentals, read the three papers that define a niche, and write a careful email to an author. An Oxford student did that on character training and the paper now exists.

Lambert's career map is a pair of gradients: more closed means more money and less credit. OpenAI's average stock compensation, he says, is over a million dollars a year per employee, life-changing for a normal American, and you become a cog who publishes less. Academia builds a clear portfolio; PhD students are paid almost nothing, which rewards people with a safety net. If a student has an OpenAI offer, he understands leaving the doctorate; he would not leave for a random startup that might go to zero. His professor friends, on average, seem happier than friends at frontier labs. The work is still three jobs — grants, teaching, research — but students, mentorship, and a people-oriented mission land differently when the industry is chaotic.

They spell out 9-9-6: 9 a.m. to 9 p.m., six days, seventy-two hours, a Chinese import that Silicon Valley is trending toward even if the clock is not literal. Raschka says the intensity has almost flipped his picture of academia; professors now look less overloaded next to a lab. Lambert cites Patrick McGee's *Apple in China* on supply-chain teams with "saving marriage" programs and, on a podcast, people who died from that pace. Nobody had to force them. Passion plus a reality-distortion field does it. His advice if you go to San Francisco anyway: get out sometimes, read history, visit other places. Twitter and Substack are not the world. He would hand a new arrival *Season of the Witch*, a history of the city from 1960 to 1985 that almost nobody in the current scene has read.

Text diffusion borrows the image-generation trick: start from noise and iteratively denoise. Text is discrete, so the intuition is closer to BERT's masked fill-in than to GPT's one-token-at-a-time decoder. The promise is many tokens in parallel and, in theory, speed. Raschka has seen papers where matching autoregressive quality means cranking denoising steps until you spend the same compute, and tool use or stepwise reasoning is a poor fit because you cannot pause for a code interpreter mid-denoise. Google has announced Gemini Diffusion, claiming much faster generation at similar quality on most benchmarks versus Gemini Nano 2. He does not expect it to replace autoregressive LLMs; he expects it on cheap, high-volume tasks, maybe a future free tier.

Lambert is already hearing a production case. Code startups face a user who says "make this change" and needs a huge diff without much external context. Autoregressive generation can take minutes; every second churns users. Diffusion can dump the long diff in a batch. He had expected different architectures to specialize faster than they have. The blocker is tool use: Claude Code and ChatGPT search interrupt an autoregressive chain with an external call, and he does not know how to do that cleanly in a diffusion setup. gpt-oss was trained with tools in mind; open models still need to work with whoever's search API the downloader prefers, while closed labs wire one tool into the product. That gap, he thinks, will force more interesting orchestrators on the open side once they catch up.

Continual learning is the AGI-shaped hole. A remote-worker milestone, Lambert says, fails because a good editor told about a mistake does not repeat it, and language models do not modify themselves that fast. His counter is in-context learning: dump every blog post and a description of your voice into a long window and the model can *look* like it learned. Raschka notes that GPT-5 to 5.1 to 5.2 is already a coarse global weight update from community failures, and RLVR is a finer one; per-user updates are too expensive unless the cost sits on-device, Apple-style. Context length is mostly compute and data. They expect two to five million tokens this year, not a hundred million. Agents, Lambert adds, will learn to compact their own 100,000-token histories as an RL action — keep eval scores high, tokens low — instead of dreading Claude Code's bulleted-list compaction.

World models, Raschka says, are a way to simulate a toy version of the world so the system is not stuck on next-token prediction plus a final-answer check. He points to a Meta paper that applies the idea to LLMs by also checking intermediate variables — the model learning a code environment. It is expensive and more sophisticated. Graduate-school CASP, the protein-structure contest whose answers are unknown until experiment, is the eval design he wants for LLMs. AlphaFold 1 modeled physical interactions and impossible angles; later versions dropped that and brute-forced scale. LLMs are in the brute-force phase. He thinks constraints may need to come back.

Lambert is split on robots. He is very bearish on learned robots for the home, very bullish on self-driving, and bullish on warehouses Amazon designed for robots from day one rather than retrofitting human buildings. Repetitive work humans can do but do not want, in an environment you can rebuild, is the reasonable path, and it will take longer than the hype. Fridman adds the point robotics talks skip: LLM failure modes are conversation. In homes, across millions of minutes, you are almost never allowed to fail. Continual learning bites harder when every house is different. The leap from an AI singularity to scaled U.S. manufacturing, Lambert says, is also a political problem, not only a model problem.

Nobody agrees on AGI, though a common anchor is reproducing most digital economic work — a remote worker. Lambert does not love it. Today's models are not a drop-in employee, and some tasks sit above that bar: an unexpected scientific discovery, or mining medical records for a common drug that treats a rare cancer. AI 2027 aimed at a superhuman coder, then a superhuman AI researcher, then a superintelligent one, then full ASI, originally in 2027–28, later slipped to a 2031 mean. Lambert's guess is later still. His camp is that AI is jagged: excellent at some code, bad at large-scale distributed training because that data is scarce. Humans will keep filling the gaps. "Big tech is going to spend $100 billion much faster than we get an automated AI researcher that enables an AI research singularity." Computer-use demos in 2025, he notes, still sucked. APIs may beat taking over the screen.

LLMs feel free because they are subsidized. Lambert thinks ads are coming. Google is the obvious first mover: it already has supply, and if Gemini demand can be turned into useful ads it can open the gate. Nobody moves first because the headline would be a reputation hit; once someone does, a ten-year flywheel can fund better R&D the way YouTube's ads scare Netflix. He pictures the first version as a labeled promoted post, X-style, not a secret Nike shoe in a recommendation. On mergers, Dario will never sell Anthropic. Consolidation is already ugly: Groq around $20 billion, Scale AI almost $30 billion, licensing deals that take star talent and leave rank-and-file options unvested. That structure, he says, is detrimental to the startup ecosystem that used to pay people even when the company only sold cheap. MiniMax and Z.ai filing IPO papers in China is the contrast; giant U.S. labs can raise forever and have little reason to go public.

Cursor's Composer, a fine-tune of a large Chinese MoE — it sometimes answers in Chinese — is the closest thing they cite to real-world RL: a blog sentence about updating weights every ninety minutes from user feedback. Llama is the cautionary tale. Lambert does not expect an open-weight Llama 5. Raschka's read is that Llama 1 through 3 earned love as usable, trustworthy, modifiable models, including small ones; leadership then tried to splash the brand with giant Llama 4 boxes tuned to look good on benchmarks, skipped the small models people could run, and the project felt forced. Lambert is blunter: it imploded under internal fighting and misaligned incentives, researchers versus managers proving they had done the thing. Community backlash may have pushed Meta into "reevaluating" open source after Mark Zuckerberg's 2024 case for it. X and Reddit can hate Grok Code Fast in public while plenty of people still use it.

That U.S. gap is what Lambert spent five months on. In July 2025 he counted four or five DeepSeek-caliber Chinese open-weight models and zero American ones, and launched ATOM — American Truly Open Models — originally the "American DeepSeek Project," a name that does not work in D.C. Open models are the engine of AI research, he argues, so the U.S. should build the best of them or the charts stay Qwen, Qwen, Qwen. A generation behind closed labs costs about $100 million, a lot of money and not a lot to these companies. NSF gave AI2 $100 million over four years, the largest CS grant in its history. NVIDIA's Nemotron 3 released models and some data. Reflection AI's $2 billion raise is explicitly for U.S. open models. The White House AI Action Plan has a section titled "Encourage Open-Source and Open-Weight AI." He could have made the pitch viral with authoritarian ASI; he talks about innovation and science instead, because that is the world he wants.

A Manhattan Project that classifies the models and builds an American firewall, both guests say, is not possible in 2025–27. Training costs of one to a hundred million dollars are attainable to huge numbers of people, so the models will be trained all over the world. Containing them would mean restructuring the internet. Chinese open releases, Raschka suggests, may even push U.S. labs to open better weights once DeepSeek-V3.2 is used without a security panic. Lambert agrees those companies set things in motion that might not have happened otherwise. If progress saturates while money is still cheap, optimized common open architectures plus dedicated chips could undercut custom closed serving. He just does not want the U.S. to arrive at that future as a customer of someone else's weights.

Raschka's NVIDIA story starts in graduate school, doing molecular dynamics on a Tesla GPU about fifteen years ago, before anyone thought of it as an AI plan. The moat is CUDA's two-decade compatibility, not only the silicon. With LLMs it may be easier to clone a CUDA than it was to build one. Training and inference compute are splitting: the Groq acquisition is aimed at that, and Vera Rubin includes a pre-fill chip with little high-bandwidth memory, cheaper per FLOP when you are doing the matrix-multiply prompt phase rather than KV-cache generation. So long as AI progress is fast, Lambert says, NVIDIA's flexibility wins; if things stagnate, Amazon Trainium, Google TPUs, and other bespoke chips get time. Great individuals, in his view, make things earlier and more focused. Without Jensen the GPU revolution still happens, maybe a decade later, maybe after another winter. Raschka's analogy is ETFs versus single stocks: passion changes the path, not the destination.

A hundred years out, Raschka thinks historians keep the umbrella word "computing," not the details of CUDA or this year's software fights. Lambert wonders whether internet connectivity and compute merge. The dying dream he named earlier — one central model that runs your digital life — is already becoming many agents on many clouds, "many AGIs in the data center" that talk to each other, which only works if information and GPU networks flow. Devices may not be phones; Fridman bets on brain-computer interfaces; Lambert still expects people to carry a physical brick of private compute. Humans will not biologically change in a century. Agency and local community, he says, are what people actually want, and UBI does not supply them.

Fridman insists that every job lost in the transition is a person suffering, whatever the GDP story. Lambert hopes society drowns in AI slop until it snaps and puts a premium back on physical rooms, events, and handmade things — the museum painting versus the perfect reprint. Raschka already bounces off obviously generated prose. On takeover risk he is calm for a specific reason: consciousness and agency. You tell today's systems what to do; they are more automatic than a hammer and still a tool. In a Terminator setup Lambert jokes that humans win, probably with local open-weight models. Einstein gets the last word from Fridman: it is not that he was so smart, but that he stayed with the questions much longer.