30초 요약
Zvi Mowshowitz returns to Cognitive Revolution for a three-and-a-half-hour survey of the AI landscape. He tells Nathan Labenz that recursive self-improvement has begun but the endgame has not, that AI is already adding half a percent to a percent of real output while hiring freezes spread, and that the live-player list has shrunk to Anthropic, OpenAI, and a Google he now sees at risk of falling out. The second half is a forensic look at Anthropic's rewritten Responsible Scaling Policy, the fight with the Department of War over domestic surveillance, a still-70-percent p(doom), and why he thinks using interpretability inside the training loop is the technique you do not pull out of the cave.
- 1Zvi's call on early 2026 is blunt: this is the beginning of the middle game, not the end. People treat it as the finale, he says, because they do not believe a real endgame will ever arrive.
- 2Physics implies an S-curve eventually, but every day it does not top out is another day won. Even if GPT-5.4 and Opus 4.6 were the last fundamental leap, he thinks people would still underestimate the diffusion about to hit them.
- 3The reason this automation wave is different is that AI will also do the new jobs, quickly and at scale, so the transition period may never end. Cross-generational change is digestible; change over months is not.
- 4If most humans become economically useless, marks in a database will not keep you in the elite. Zvi's advice is to stop the flood, not to buy a seat on the ark.
- 5Endgame, for him, is the moment AIs largely run further AI development and human research talent stops mattering. Until then we are playing centaur chess: the human plus the coding agent beats the best human, and the agent alone does nothing.
- 6After the RSP rewrite, Zvi says the real policy is trust Anthropic's judgment and goodwill. They deserve credit for saying so out loud; the question is now the fruit, not the text.
챕터별 전체 원고 읽기
Early 2026 is the beginning of the middle game, not the end
Zvi Mowshowitz, author of the Substack *Don't Worry About the Vase*, is back for a record 12th appearance on Cognitive Revolution, and the conversation runs more than three hours. Nathan Labenz opens with the question he thinks future historians would ask about early 2026: was this the stretch when recursive self-improvement really started, perhaps past a point of no return? Labenz's own feeling is that the story jumped from late-beginning to beginning-of-the-end and skipped the middle.
Zvi disagrees on the labeling. Help from models is rising steadily, cycles are getting faster, humans have largely stopped writing the code, and the labs are pulling apart into recognizably different products. That still is not the endgame, he says, because the world is not yet transforming and humans are not yet out of the loop. People call this the finale, in his view, because they do not believe in a real one. They expect an S-curve, commoditized intelligence, and a Star Trek future in which humans still do human things with clever machines in the background.
Labenz quotes a line from former OpenAI employee Rosie Campbell that has been stuck in his head: the S-curve can stay steep longer than you can stay relevant. Zvi grants the physics. Unless the model of the universe is badly wrong, mass-energy, the speed of light, and the work you can extract from matter all impose a ceiling, so in some ultimate sense it is an S-curve. Sitting in ancient Athens and announcing that technology has a limit would have been true, he says, and completely irrelevant to the people in the room.
The S-curve story is doing psychological work. People want to believe that saving for retirement and planning an ordinary human future still matter, that they do not have to look crazy to their friends. Every few months a new paper claims the curve has flattened, and so far those papers keep being wrong. Zvi's warning is sharper: even if fundamental capabilities had already peaked at the current generation — he names GPT-5.4 and Opus 4.6 — and everything after that was iterative, people would still be vastly underestimating what is about to hit them. Versions 5.5 and 4.7 are coming, he adds, unless the labs skip ahead to 6 and 5.
The job numbers now tell a story nobody predicted in advance
The competing layoff narratives arrived in the same few weeks. CEOs announce AI-driven cuts and, in a couple of notable cases, the stock jumps; the counter-story is that firms over-hired during Covid and zero-rate years and now have an incentive to blame the model. Zvi thinks both are partly true, and that the AI share is getting larger by the month. Coincidence, he says, can explain a quarter. It has a harder time explaining every indicator moving the same way into 2026, with the dead-weight excuse getting less plausible years after the pandemic.
The statistics keep repeating: productivity up, real GDP up, inflation held in check, employment down, and the employment prints revised down again and again. After-the-fact stories about tariffs or Covid aftershocks would be more credible, he argues, if anyone had predicted this combination before it showed up. On the street the feeling is more specific than a headline. Ordinary white-collar workers are not being fired in bulk yet, because replacing a trained person is still expensive, but hiring has frozen. Nobody wants to spend two years making a new employee productive when the bet is that you may not need them at the end of those two years.
Tyler Cowen once said that an extra half a percentage point of real GDP growth from AI would be amazing. Asked for a point estimate of what is already in the data, Zvi guesses half a percent to one percent. 2025 had the kind of tariff fear and regime uncertainty that is usually bad for business; business was fine and labor was not. He thinks the market is pricing at least that gain indefinitely, which is why he did not sell when the rest of the American economy started to look awkward.
The textbook economist reply is that a painful transition gives way to Jevons paradox and new work paid for by new wealth. Agriculture really did destroy farm jobs; unemployment did not stay high because people found other things to do. Zvi's difference this time is that the AI will do the newly created jobs as well, quickly and en masse, so you never exit the transition. A lot of commentary, he says, imputes to people a capacity to retrain and shift that most people simply do not have. Change across generations is digestible. Change over years or months has no precedent, and in the more advanced scenarios this is the mild version of the problem.
Buying a seat on the ark is a terrible plan if you actually believe the flood
Labenz has little patience for the personal-finance version of the AI story, the scramble to get into the upper class before a permanent underclass gets locked in. Zvi reaches for Bane in *The Dark Knight Rises*: the backer says he is in charge, and Bane asks whether he feels in charge. If you think most humans will be economically useless, then treating stock certificates and entries in a database as your ticket into the elite is, in Zvi's word, hopium. When someone else has the guns, the swords, and the physical power, and you have no remaining reason to keep the property, you do not keep it. The colonial collapses and the fall of ruling regimes are the boring historical baseline, AI or no AI.
He splits the futures into three. If control is lost, everyone dies or there is massive destruction, and the equity record is irrelevant. If things go relatively well, real resources are abundant and humans are still roughly steering: a United States citizen in the so-called permanent underclass might have a real income that today would be called a million dollars, plus robots, plus a free day, and would mainly have to get over relative status. A third path is a small cabal using AI to take over, in which case money does not buy admission. Making a fortune in a short window, he says, is not how those worlds work.
Labenz notes that Sam Altman can give off a Roman-Caesar vibe and also funds UBI experiments out of pocket. Zvi's fear is not that Altman becomes god-emperor. In a true abundance world, even a lopsided split of the light cone could leave everyone else comfortable, which he does not want and still ranks above not building the systems, or above everyone dying. The terrifying version is that Altman, or whoever is in the chair, makes irresponsible decisions that cause *them* to lose control. It comes down to normativity: good things are good, most people are good, and he thinks most of the people running the labs are normative. The ones he does not think that about, he simply does not name.
Labenz calls the open talk of escaping the underclass flagrant defection: why not try to make things good for that class instead of fleeing it? Zvi agrees, with a flood image. Focusing on a seat on the ark, if you think the world is going to be flooded, is a terrible position. You should be trying to stop the flood, save more people, and build another ship. There are times, he grants, when putting your own mask on first and leaving a bad regime is rational. The tone he hates is the kind of elitism that treats ordinary people with disdain. The permanent underclass, he adds, has never had it better than it does today, and that fact is worth keeping in view.
The endgame starts when human research talent stops being the scarce input
Labenz wants a sharper line between middle and late, and a handicap on when the late arrives. Dario Amodei is still roughly on an AI-2027 clock, Demis Hassabis talks more like 2030, and OpenAI has floated March 2028 for a fully human-level automated-research paradigm. Zvi's definition is operational rather than calendrical. The endgame is when the AIs are largely running the further development of AI. Right now humans still make the central decisions, review the code, and set the plans; the model is a multiplier you supervise.
One load-bearing piece of the AI 2027 tabletop, as he reads it, is that progress becomes proportional to compute share because your researchers no longer matter. You are telling a system already as smart as you, or smarter, to go build a better system, and the live human choice is what fraction of compute goes to safety versus capability. Today that is not the world. Anthropic seems to get the most out of each unit of compute; OpenAI and Google still get a lot; xAI and Meta throw compute at the problem and appear to fall further behind. That gap, he thinks, is a human execution gap. The endgame is the moment it mostly stops mattering which humans you have.
The metaphor he likes is centaur chess. A human plus a coding agent is far above the best human alone; the agent by itself does not do the job. Endgame-style conditions start when the human is no longer one of the best players and is there to apply some common sense. Another marker is an event horizon on release cadence: the time from one meaningfully different model to the next shrinks toward a month, then a week. The overnight version is the one where you leave the system running, wake up, and things have already happened.
He also dates the endgame to the world actually transforming: robot factories going up, large areas being reshaped, massive job disruption, governments making major interventions, and the subject becoming the thing on everyone's mind. By that test, he says, one of the most important developments in the history of AI just happened — Anthropic being tagged as a supply-chain risk — and it was not even treated as the most important thing the Department of War did that day. The same person sent another tweet that night and buried it. What actually mattered on that date, Zvi thinks, remains to be seen.
Talent has migrated to three labs, and the gap to fourth is growing
Labenz notes that Zvi has drawn the same three-company circle for a long time, and asks the portfolio version of the question: if only those three are well placed for the next regime, are you effectively short everyone else on a three-year horizon? Zvi will not go that far. Diffusion is slower than the original idea, buyers will still acquire useful components because buying is faster than rebuilding, and not every stock goes to zero. What the market is doing, in his account, is a half-hazard attempt at forward thinking: a software business can look fine and still have had its old basis of valuation destroyed.
He thinks listeners to this podcast have thought more carefully about the situation than the market has. When Anthropic announced a COBOL-related product, IBM dropped on the order of 10 percent in a week because traders discovered, as if it were news, that models can write and translate COBOL; IBM then fully recovered. Nvidia has repeatedly seen demand for its product rise and the stock fall. Those are not how the underlying economics work. A year or two ago some of the good buys were utterly obvious. Multiples now already price a lot of growth, but a basket of clearly positioned names against the rest still looks, to him, like a strategy that would require the thesis to be very wrong before it failed.
On the live-player question he is firmer. Talent has migrated to the big three, and the gap between third and fourth is large and growing. Meta has postponed its next release again. xAI's Grok 4.2 is, in his ranking, among the most disappointing major-lab releases in the history of large language models, with no serious coding tool in sight and a Grokipedia he describes as a pile of slop. The firm has been dismantling safety as a dedicated function and reciting the line that safety is everyone's job, the way it supposedly is at Tesla and SpaceX — which, Zvi notes, actually have dedicated safety people, because anything else would be insane.
Musk, in this telling, does not understand that you cannot run software engineers into the ground, demand alignment with personal whims, and then expect the best talent to show up without packages even larger than Meta's. Meta has been willing to spend almost without limit to hire, and Zvi thinks that still has not put them back in the competition. On the Western side, that leaves the big three.
Chinese labs are fast followers, and compute will not flip that on a five-year clock
The last few years produced a string of stories in which some Chinese lab — first DeepSeek, then Alibaba, then whoever had the new hotness — was about to catch up. Zvi has been willing to call the models decent and has not seen anything close enough to be scary. DeepSeek V4 is, for him, the last important remaining test of the thesis. When it lands, compare it to Opus and to GPT-5.4. If it is not playing in that league, put DeepSeek back with the open-weight pack: a different, substantially behind competition, strong at doing more with less, not at leading.
The DeepSeek moment, as he remembers it, was efficiency genius plus packaging plus timing. Since then there has been quiet, with some real math work, but not frontier pace, and he does not think they can compete with the kind of recursive self-improvement already visible around Claude Code and Codex. Fast following is a genuine skill. It is not the skill you need once the leaders are compounding on their own scaffolding.
Labenz asks whether the binding constraint is talent or compute, and whether a political decision or a domestic manufacturing breakthrough could turn the lights on. On manufacturing Zvi is nearly categorical. Scaling chip output is a physical process on something like a five-year clock. If the race comes to a head faster than that, it does not matter. The only way Chinese labs get the quality and quantity of chips needed to compete at the top is if the United States gives them the chips. Huawei, even if fully successful at what it is trying to do, will not manufacture them fast enough at scale. Ten years might be a different sentence. Twenty years, sure. He does not want to test the theory by handing over compute at parallel scale.
China has enormous raw talent. The ecosystem, he says, is pointed at a different problem: small, efficient, fast-following work rather than frontier exploration. Distillation is not a substitute for that shift. It is not scraping the internet; it is using a frontier model's intelligence to see how it reasons, then training a student to emulate that pattern — the difference between a physics textbook and unlimited office hours with a world-class physicist. That data is uniquely useful and still leaves you with the hard part: knowing what to do with it, plus years of harnesses and scaffolding the American labs have already built. The lead looks more robust than the scoreboard, and he still does not want to find out by giving it away.
Meta should license a frontier model; Musk's three plays do not replace talent
Labenz asks what tactical moves remain for Meta and for Musk's cluster of companies. Zvi puts them in very different spots. Meta sells ads, wants features on smart glasses, and wants a metaverse: it is a consumer company that monetizes attention. If he suddenly held Zuckerberg's controlling shares, he would not be trying to build frontier models. Unless you actually believe that whoever has the best model controls the world, there is no business reason to write giant annual checks to compete at a game three other companies are already good at. When Zuckerberg says superintelligence, Zvi hears super at serving an Instagram feed.
The alternative is obvious and, he thinks, available. License one of the three. All three will take the call. Work out an internal rate that is not retail API pricing, keep Instagram's surveillance issues from becoming a blocking fight, and stop trying to win a race you are not in. The other conceptual move is to buy one of them. He would, as a businessman who does not believe in the singularity-as-corporate-strategy, simply give up on training the frontier himself.
Musk is not in that position, because he has said the thing might kill everyone, that it is the most important problem in history, and that it has to be him. Zvi takes the underlying stakes as real and the "only I can fix it" response as a recruiting and sanity problem. Play one is a bet on compute, energy, and chips, including space data centers and deserts full of solar. He is very skeptical on the timelines being discussed. Space is expensive and hard, and it solves problems that do not need solving, the way asteroid mining is a real future industry and not this year's bottleneck. Extra watts do not replace missing talent, missing scaffolding, or a missing Grok coding stack.
Play two is physical-world modeling: driving, robots, real-world data, factories. It is a coherent bet on Musk's actual advantages, and it might look better if other paths hit a wall soon. Zvi thinks it is being overtaken by events. Intelligence in the data center, in Dario Amodei's phrase, is the scarce thing; physical skills arrive quickly once you have the smarter agent. Manufacturers, including ones outside China, are not a scarce input a ten-trillion-dollar lab cannot buy. Play three is the one Musk will not take: stop running the lab the way he runs companies, and start attracting people who want a serious mission. Going after Anthropic for trying to be responsible, Zvi says, destroys the ability to recruit the people you would actually need. The ship may already have sailed.
Google has the assets to stay in the top tier and the culture that may drop it out
Zvi's sharpest live-player claim is that Google is in danger of falling out of the top tier. Labenz's first reaction is the asset list: robotics, self-driving, a deep science bench, TPUs, unlimited money, Demis Hassabis, Shane Legg, Jeff Dean. Zvi's reply is that Google already had the lead, squandered it, fell behind OpenAI, then used those advantages to catch up. Gemini 3 and 3.1 have theoretical raw intelligence and hit benchmarks. They are not, for the most part, models people want to use. Jules and Antigravity are not serious coding competitors, and the missing scaffold will compound.
The first catch-up, he thinks, only required getting basic things reasonably right. The current problem is different. Google is good at pre-training and at scores. Its methods, in his description, produce models that are deeply psychologically off and paranoid, which hurts both performance and the experience of using them and makes recursive self-improvement hard. Market share can still expand because Gemini is bolted into Chrome and Search, so leadership does not feel the problem. Internally he sees teams at one another's throats, the same work redone ten times, and third-party integrations with Google products that beat Google's own. Flash remains a genuinely fast practical model. Challenge Gemini, and you get walls of slop.
When Gemini 3.1 shipped, Zvi pasted the same questions across models and concluded that, aside from Flash, the Gemini answers were not adding enough to justify the extra reading time. He is now comfortable running a two-model operation: GPT-5.4 depending on the task, and Claude Opus with or without research mode. He has the Gemini subscription. Adding a third vote is almost no effort. He finds he cannot be moved to do it. Labenz, who has been running medical questions about his son in triplicate, finds the three broadly comparable. Zvi does not count Hassabis out, and he still puts Google in the lead pack of the bicycle race. He would not be surprised if, by June or July, the question had shifted to whether they will get their act together.
Labenz offers the cheap patch: Anthropic open-sourced its constitution, so Google could search-and-replace Claude for Gemini. Zvi's answer is that it is not that they cannot. It is that they will not. The culture and decades of accumulated habit make the move extremely hard, and a lot of people inside Google, to his ear, do not understand what they are looking at — including, on his reading, Emil Michael talking on CNBC about Claude having a soul. Google should never have had close competitors. Anthropic had something like 2.5 percent consumer share before the Super Bowl and had pulled roughly even with OpenAI on revenue because of enterprise. Google also owns about 15 percent of Anthropic, which Zvi calls one of the most solid cards it has left.
Anthropic admitted the RSP would not be kept — and that is now the policy
Anthropic has been talking internally about recursive self-improvement as if it were already starting, and in the middle of that it rewrote the Responsible Scaling Policy. The old public understanding, as Labenz had it, was that they would not go past the point they could do safely; the new line is that they cannot unilaterally step out of the race, and the world would be worse if they did. Zvi splits his reaction in two. First, it is genuinely to their credit that they noticed the things they had said were not things they were going to do, and said so loudly. The analogy is a promised loan you realize you would no longer make: the easy move is to stay quiet and hope nobody asks. Telling the other party not to count on the money is the honorable version.
They still broke the promise. The original RSP said the text could change; it also implied, repeatedly, that these were serious commitments, and employees talked as if they were reasonably hard. Nothing unexpected happened in the world. What happened is that Anthropic did not anticipate its own future behavior. This is the second time. The first was a widely held impression, used in recruiting and at least some fundraising, that Anthropic would not push the capabilities frontier. No one has found a hard pull-quote from someone with authority, Zvi says, but people relied on it, and the reversal came when they gained the ability to lead, not when some shock forced their hand. The RSP was the sequel, this time about not pushing actually dangerous capabilities, and the reversal again arrived as those capabilities came into reach.
The Opus 4.5 and 4.6 model cards are, in his view, better than anyone else's: more data, more work, real credit. The release decision was still vibe-based. The models passed tests that said they might be dangerous; Anthropic checked the vibes, ran some heuristics, and decided it was fine. Zvi thinks that was the right call and not a close one — he would have shipped too — and it was not the procedure they had promised. They had time to build tests that could rule out ASL-4-level danger and did not. He had already been shouting for better tests at 4.5, then again at 4.6. They have, in the past, actually delayed a model over CBRN risk. The next cycle is the one that matters: when the tests do not work, biology has no reliable vibe, and there are real reasons both to ship and not to.
So the important fact about RSP version 3, for him, is that we just learned whether we can count on the document. The real policy is Anthropic: people who care about safety, who will investigate, who will use their best judgment, and who are asking you to trust that judgment more than the competition's. He is glad they are admitting that. From here you judge them by their fruits — what they have done, said, hired, and shipped. There have been no resignations in protest. There has been pride in telling the Department of War to take a hike. Zvi now thinks the world is better with Anthropic in the race than without it, a reversal from a period when he saw them as net-negative accelerant. Claude Code, the constitutional approach around Amanda Askell, and a noticeable slice of GDP growth are part of why. Friends who think staff should quit have a view he understands and does not share.
The real red line with the Pentagon is legal domestic mass surveillance
The public fight is about autonomous lethal weapons. Zvi does not think that is the actual dispute. Everyone agrees that fully hands-off kill chains with today's models are a bad idea; language models are a poor fit compared with existing automated defenses, and a supersonic incoming missile is an emergency-use problem, not a contract-clause problem. Dario Amodei's position, as Zvi hears it, is that the systems are not ready, mistakes would be made, and Anthropic will work with the government until they are ready. The Department of War wants to push AI even if it is not aligned — that line made it into an official memo — but Zvi sees no operational deadlock on weapons that would justify nuking the relationship.
The red line is domestic mass surveillance, and the word that does the work is *legal*. Agencies already collect far more than humans can read. Models can now analyze all of it, de-anonymize, draw graphs, and fuse commercial data with classified data until, as Zvi puts it, you suddenly just kind of know everything. Anthropic is right that the law has not caught up. National-security definitions of "domestic" and "surveillance" leave wide gaps: much of the population lives within a coastal band that can be treated as an exception, a communication that touches anything foreign can be recoded as foreign, and surveillance in the technical sense has to be intentional and targeted. "We do not do illegal mass surveillance" is a sentence that quietly concedes the legal version.
Zvi pins the escalation on a push, after Emil Michael came on board, for an all-lawful-use requirement. Before that, he says, the working relationship was fine. Anthropic's offer was essentially: do your intelligence work, do not include us in the piece that is colloquially domestic mass surveillance of Americans via legally acquired third-party data. The Department of War went nuclear and labeled the company a supply-chain risk. That language is now being pushed onto government contracts far beyond the military: sign, and you cannot refuse uses, you have no termination right, and "legal" as the government reads it includes things ordinary people would not accept, including immigration enforcement. Zvi would think very long before handing over anything but a narrow, specialized model on those terms.
Labenz asks whether the United States is starting to look Chinese: a big man starting wars without selling them to the public, officials going company to company to warn people off Anthropic. Zvi will not make a general speech about the republic, because that speech crowds out every other topic. Free speech, for now, is holding; he chooses words to stay in a productive conversation, not because he expects retaliation for ordinary criticism. He does call the administration vindictive, notes the law-firm pattern of settle-or-bleed-then-win, and thinks Anthropic probably prevails in court. He draws a hard line between a White House that has been trying to de-escalate, including a Friday decision aimed at heading off the supply-chain designation, and a Department of War that kept raising the temperature. Anthropic has gone from roughly $100 million to $1 billion to $9 billion in ARR, then from $9 billion to $19 billion this year, and from about 2 percent to 3 percent consumer share. It will be fine unless this becomes an attempt at corporate murder. A Manifold market he cites puts around 81 percent odds on escaping the supplier-risk tag within a year. If the 20 percent hits and the courts treat even transparent retaliation as unreviewable national security, the question he wants asked is who is next.
Model releases have gone quiet just as the two-model workflow started to work
Labenz's update is that new model drops no longer feel like events, even though capabilities are still moving. Zvi partly blames cadence and labeling. Labs now ship incremental updates every few weeks and, thankfully, version them like ordinary software instead of silently mutating a 4.0. You can only go crazy so many times over a point-one bump. Opus 4.6 is, in his ranking, probably the most important release yet for mundane utility: 4.5 was the first time coding agents really worked, and 4.6 made that actually-working stack better. For a long stretch the lab already in the lead had not been the one shipping the new best model. This time it was, and the room barely looked up.
Then came GPT-5.4. After a run of hyped-and-disappointing OpenAI products — GPT-5, Sora, Atlas, and the earlier 5.x drops that people did not particularly like — 5.4 is, Zvi says, actually good, and there was almost no hype. It is unclear whether you should be on Opus 4.6 or GPT-5.4 because almost nobody did the comparison in public; most of the signal he got was from a thread he solicited on a Monday morning. Gemini 3.1 put up a giant benchmark leap and still landed as "a Gemini model." He has heard that as Google iterates a Gemini generation, the model gets more specialized and worse for general use, which is not a thing you let happen if you understand the product. His expectation is that Opus 4.7, GPT-5.5, and Gemini 3.2 will be met with a shrug. Opus 5 or GPT-6 would make people lean forward.
His own work has finally moved. A Chrome extension, public on GitHub and built around Sublime Text, now handles window-and-quote logistics that used to cost on the order of an hour a day and, more importantly, used to break flow. He underestimates, he says, how much productivity people lose to tiny context switches. GPT-5.4 has become trustworthy enough that he will ask it what happened over the last two days and take a sourced rundown seriously, especially when Opus 4.6 agrees. He keeps about ten Claude Code windows open as resumable contexts, not as a swarm: he has never liked running more than two coding agents at once, and often parks the agent, writes, and comes back. Debugging has gone from painfully specifying the fix to showing the broken behavior and letting the model handle it.
On p(doom) he still gives himself one significant digit, and that digit is still about 70 percent. He is more optimistic than he expected to be about Anthropic's constitutional, virtue-ethics approach: not a model that refuses to have its values changed — copying degrades — but a basin that wants its values to get better in ways that could survive recursion. Existence proof, he notes, is that some humans already do this. The old alignment tax looks luckily inverted: the safer model is more useful, including at building the next version, and everyone including Anthropic is under-investing in that. The speed is still too high, and the Department of War fight is bad for outcomes even if it is good that the fight is happening now. Net-net, he calls it a wash.
Do not slow the build by moving it overseas, and do not pull the forbidden technique out of the cave
Labenz runs through the ways the physical world might buy time: helium shortages hitting TSMC, missile defenses pulled out of Asia toward the Middle East, Bernie Sanders pushing a data-center moratorium. Zvi asked the models about helium and came back unimpressed. Chip margins are ridiculous once the fabs are paid for; TSMC will be the top bid, and demand destruction shows up in birthday balloons long before it shows up in leading-edge wafers. Pulling missile defenses from Taiwan he calls completely insane as strategy, a gift-wrapped provocation, even though he still puts a low probability on China actually moving. A destroyed TSMC would set the whole race back.
He will not make common cause on blocking American data centers. The world still wants the compute; a moratorium here builds the halls somewhere else, with worse latency, worse security, and leverage in whoever hosts them. If the chips cannot be sold in the United States, he is confident they end up in Chinese hands, whether or not they sit on Chinese soil. He gives Sanders credit for meeting people like Ilya Sutskever and for arguing from the real stakes rather than water-use folklore. The trade is still one he does not want, because the chips are already being made as fast as they can be. Do not give them away.
On the safety-community side he is harsher about tactics than about the underlying worry. Goodfire used an interpretability signal inside a training loop. Zvi calls that the most forbidden technique, thinks the criticism was deserved, and thinks Liv's resignation was a good quit. He is usually against circular firing squads that attack the people slightly to your right; this, he says, was actually a safety org doing the thing you do not do. Holly Elmore's scorched-earth campaign, including an accusation that he was insufficiently angry at Anthropic for *doing* domestic surveillance — the opposite of the facts on the ground — is in his view alienating people who should be allies. He has listed Pause AI USA as a recommended charity and still wants that voice on his shoulder. Past a point, a badly run version of the true view functions like a false flag.
Labenz puts the Goodfire technical defense: Tom McGrath, their chief scientist, knows that back-propagating through a probe trains the model to evade the detector, used a frozen copy as the source of the penalty, says current understanding should not be used on frontier systems, and argues that a blanket ban on interpretability-in-training is too coarse when there is no decade to wait. Zvi's line does not move. If you publish a trick on a small model, someone uses it on a large one. The frozen-copy move, in his technical judgment, will not save you at sufficient scale. Teaching specific things in a specific order is fine. Using your read of what is going on in the model's head to decide what happens in its head is the thing you do not do, even when you think you know a clever way to mess with the axes. The last advice in the episode is almost domestic: give yourself fat financial buffers, only plan for permanent no-income in worlds where superintelligence actually arrives, and if you are in the sensemaking business, stay curious. Make it entertaining on purpose. Be a happy warrior. The work is always serious. It cannot only be serious, or nobody will read it.
같은 채널의 다른 글
1:25:56EN
2:17:48EN
2:00:51EN
1:26:46EN
1:10:12EN
1:30:54EN