One Company Scares Him More Than China: Zvi's Map of the AI Frontier

The same GPT-4, three different verdicts. Jan Leike, who co-leads OpenAI's Superalignment effort with Ilya Sutskever, has said the model is roughly at the level of a well-read college undergraduate. Host Nathan Labenz has often called it human-level but not human-like. Guest Zvi Mowshowitz, the prolific writer behind Don't Worry About the Vase, puts it well below human level. None of them disputes that GPT-4 is the strongest system available in the summer of 2023, and likely to remain so for some months. The fight is over the ruler.

Labenz frames the episode around *live players*: the handful of organizations actually pushing the frontier and in a position to change the course of events. The tour is OpenAI, Anthropic, Google DeepMind, and Meta. Before they get there, Mowshowitz wants the measurement dispute on the table, because Leike's four-year plan to align a human-level alignment researcher only makes sense if something much more powerful than GPT-4 is coming. Point GPT-4 at alignment and you get a modest speedup, he says — maybe 50%, maybe 100% if the tooling is excellent — not a substitute for a researcher.

Leike, in Mowshowitz's reading, is asking how the model does against an average well-read undergrad on a grab-bag of ordinary questions. That is a fair practical question. Mowshowitz is asking about raw G: every system, human or artificial, has a general-intelligence floor, and once a problem needs more G than you have, difficulty rises exponentially and then drops to zero. GPT-4's knowledge swamp — every book, Reddit thread, and tweet — lets it crush day-to-day Q&A against an IQ-100 helper. Give a college graduate time on a hard, novel problem such as aligning a model and they might make progress. Give GPT-4 the same problem and you get nothing.

The undergrad on day one of the job is, in his unkind phrase, a useless piece of junk — but a junk that can *learn* to be something. GPT-4 lacks that channel. Labenz asks whether memory, retrieval, and agent wrapping would close the gap; Mowshowitz expects those pieces to come online over six to twelve months and to lift the model toward average undergraduate *performance*. They will not, he insists, fix the orthogonal failure: the moment you ask for something truly original, off the training distribution, the system falls over. The Cognitive Revolution's episode with Hollywood writers is his exhibit. GPT-4 was much faster than the writers at generating generic schlock for episode 47 of a show. Asked for work they would recognize as distinctly creative, it collapsed every time.

Mowshowitz has a name for the thing Labenz is more excited about: mundane utility. Even an average person spends almost all of the day on work that never approaches peak intelligence. Someone has to file the paperwork, stand at the retail counter, cash the check, be civil. Physical labor is the same: if a worker had to operate at peak strength for more than a few minutes, the job would break them, and those jobs mostly do not exist. You keep a strong person around for the moments that need strength, and a smart person around for the moments that need thought.

The arithmetic is the point. Take the bottom 80% of a job and do an 80% job of it, and two-thirds of the day comes free: output can triple. That, Mowshowitz agrees, is what GPT-4 can actually do if people learn to use it — free hours from drudgery, not solve problems humanity has never solved. Labenz is more enthused about that dividend than about the alignment-researcher hypothetical, and Mowshowitz does not dismiss it. He does list the traps. Automation may hit exactly the tasks nobody wanted automated. The moment paperwork gets faster, institutions add more paperwork, and humans stay busy until quitting time on even less useful work. GPT-4 becomes a nicer treadmill.

If you want Leike's sea change — systems that solve problems we could not solve before — the model has to do every step a human can do. Otherwise the leftover bottlenecks force a costly translation: dump context back into the human world, let a person take the hard step the model still fumbles, then translate back. That yields factor-of-two or factor-of-five gains, worth having and less dangerous, and it will not solve alignment by itself. Superalignment is aiming at something else. Mowshowitz's line is the one Labenz queued in the cold open: the project is trying to keep humans out of the loop entirely, "and that should be about as scary as it sounds."

Labenz has a public scorecard he calls the tale of the cognitive tape, used in his AI scouting report to compare people and the best models along breadth, depth, breakthrough insight, speed, cost, availability, memory, diffusion speed, and bedside manner. Mowshowitz marks the card as they go. Breadth is the model's biggest win: it can cover every topic at once. Depth is worse than Labenz graded — maybe one out of three, not two. Breakthrough insight is still a human blowout. Real-time output is perhaps 10x faster; internal processing is thousands to hundreds of thousands of times faster. Cost already favors the models when they are useful, even before companies stop eating losses to buy share.

Availability is about to get strange. Compute, Mowshowitz says, is ceasing to be essentially free and becoming unpriced: you cannot simply pay more and get more. A single GPU is still on eBay. Industrial quantities are a different market. Nvidia is not pricing at the clearing price, there is not enough to go around, and access tilts toward who has the right arrangements. Tyler Cowen's line that context is that which is scarce applies twice over: models have vast long-term memory for facts and are much worse than humans at holding a particular working context. Bedside manner is a split screen. When the model is optimized for what the user actually wants, it beats any human. When Claude or Llama refuses, the manner is "negative one stars" — a lecture on social justice instead of an answer, something no person would do unless they were trying to punish you. That is an RLHF choice, not a law of nature.

He then adds columns Labenz did not have. The first is handling something genuinely new. Image models make the failure obvious: one face, one style, one action is fine; overlapping elements that have to interact lose the thread immediately, which is why so much AI art repeats and why inpainting a region is often wiser than regenerating the whole frame. Language models have the same disease with *vibes*. They remember facts but can hold only one vibe at a time, and multiple tasks in one prompt bleed into each other. Mowshowitz says he has not seen a serious attempt to solve that. The second column is robustness. A coder friend named Allan tried the tools and reported that once the work left the training distribution — which, in his job, is the point — the error rate made the model slower than writing it himself.

The third is adversarial robustness, and here he defends the models. It is unfair to judge them by jailbreaks, because the attack is one humans never face: infinite clones, reset the memory after every miss, keep going until the secrets come out. "I guarantee you I'm getting your bomb secrets," he tells Labenz. A new paper on universal adversarial attacks found suffix strings that transfer across Llama, Bard, and GPT-4, and GPT-4 could even write some of the attack code. Offense is much easier than defense. Labenz pockets two new rows for the scorecard and they climb back to OpenAI.

Labenz notes that Mowshowitz and Leike would probably agree on what GPT-4 can do *today*; Mowshowitz says he would almost flatly believe Leike's empirical predictions, because Leike is closer to the metal. The update that mattered was not a capability forecast. It was a long comment Leike left on Mowshowitz's blog, which Mowshowitz calls the best comment he has ever seen there, and possibly on any blog of that type. On the short list of people who might affect the fate of humanity, Leike ranks "remarkably high." He engaged in detail, did not dodge, treated an independent writer as worth answering, and in several places made points Mowshowitz had not thought of.

The clarification everyone — Mowshowitz included — had gotten wrong: Superalignment is not trying to train a human-level alignment researcher in four years. It is trying to align the human-level alignment researcher that will, on their timelines, emerge from someone's lab in that window. When that system appears, OpenAI wants to know what to do with it, keep it under control, and share the method with whoever built it, whether that is a nonprofit, Google, or someone else. The project flips, in Mowshowitz's words, from quite plausibly a capabilities effort at heart to a clearly net-positive idea if the comment is accurate. The worst case drops from actively building the thing that might kill us to trying something that fails, giving people false hope, and still getting everyone killed — which is "so much better." The advertised 20% of compute is then spent on a useful job; the thing that actually builds the researcher still has to come from the other 80%.

He did not change Leike's mind on the big issues, as far as he can tell. What he got was evidence that Leike had thought about the objections at a deeper level than the public announcement showed, and was reading to see if he was wrong rather than mining for dismissible quotes. Leike later recorded an episode of the AXRP podcast so dense that Mowshowitz stopped after ten minutes to restart with a notebook. Alignment people at DeepMind have been similarly willing to talk. The lesson Mowshowitz wants: if you have a technical argument, these people will engage. That puts independent critics in the game, even though he remains "deeply skeptical of everybody involved's plans."

Anthropic was founded by seven people who left OpenAI. The public relationship between the firms is warmer than a split-then-compete story would predict: they still talk, they still co-sign statements, they still show respect. Labenz wants a sign on whether two labs are better than one. Mowshowitz does not have one. All else equal he would rather have Anthropic than OpenAI, because Anthropic has built a culture that actually takes the danger seriously — if anything the Vox profile worried him the other way, an unhealthy level of fear that can cross into paralysis. The cost of that culture existing as a third company is a three-horse race. Capital is being raised to build the best model ever, which by default makes Google and Microsoft-OpenAI run harder.

"The proliferation of shepherds is inherently bad in itself," he says, even when the new shepherd is relatively good. Coordination among Anthropic, OpenAI, and, to a lesser extent, Google DeepMind is the hopeful offset: it keeps a future bargain thinkable. It would have been much better if the Anthropic founders had converted OpenAI from the inside. He would not be eager to throw them extra billions or extra capabilities. He would love to have a company he trusted enough to bring technical ideas; he does not. Individuals in nonprofit and rationalist circles, maybe. Anthropic is the closest company, and still not close enough.

Constitutional AI is, in his view, Anthropic's biggest contribution — and he has a strong prior that it will not scale. For GPT-4-level systems, done well, it is a good idea. For human-level and superhuman systems, nothing like the current technique will deliver what people hope. He has a head full of ways to improve Anthropic's implementation and does not plan to tell them, because he does not want to encourage a better version of something he thinks still fails. He would rather see more alignment-only research organizations that are not pushing capabilities, and he has "irons in the fire," too early to announce. Separately, Anthropic published a security-practices post that is barely about AI: two-person control to unlock sensitive production systems, the nuclear-submarine pattern without the nuclear image. Labenz files it as a live player taking physical security seriously.

A paper from the week of the taping found nonsense token suffixes that, appended to a refused query, jailbreak RLHF models into answering. The attack transferred: it worked a majority of the time on Llama, Bard, and GPT-4. On Anthropic's Claude models the success rate was about 2%, more than an order of magnitude lower. Labenz reads that as a real win for constitutional AI, on an attack the lab probably did not design against. Mowshowitz is cooler. He was unsurprised that an attack trained on a system works on that system. Transfer across Llama, Bard, and GPT-4 is the interesting bit, and it is less mysterious once you notice those models were trained with similar RLHF, similar goals, and similar evals. Similar muscles, similar bruises.

Claude was trained largely with constitutional AI, which is cheaper per cycle, so most of its post-training cycles were almost certainly constitutional ones. A different training method did not flex the same weird muscle, so the same character salad mostly failed. That is promising, not conclusive. The test that would move him: take the paper's search procedure, run it *on Claude*, and fail to find a new exploit of the same type. Until then, a transferred attack that "has no right to transfer" simply did not transfer. He also doubts OpenAI is already running a quiet constitutional-AI stack. The paper is recent; GPT-4's post-training may have finished before it existed; stacking incompatible alignment methods on one model does strange things; and OpenAI is rich enough to keep buying human feedback — including, according to reports, raters in Africa — while Anthropic, hiring in the US and Canada, has a much stronger incentive to automate.

Claude 2's actual product problem, in his daily use, is that it is scared of its own shadow. It apologizes, bows out, and will not speculate. He now uses it as a first filter: if it refuses, he pastes the same prompt into GPT-4 and is usually done in ten seconds, even though Claude's training cutoff is later. The constitution in the paper, and the examples Anthropic chose to publish, make the failure mode obvious. Dozens of rules all say *choose the least X*; there is never a point at which the model is non-racist enough; refusing is always available. His coffee analogy: if you score one for delivering coffee and zero for failing, the rational policy is four cups and four routes to the office. A system that only punishes screwing up, and never rewards doing the job, learns to do nothing. Nothing about constitutional AI *forces* the super-harmless assistant; Anthropic chose that constitution. If they raise billions for the next generation, they will have to decide whether to keep making the product less useful.

Anthropic's other differentiator is a 100,000-token context window, free. Labenz built the outline for this episode by pasting entire Mowshowitz posts into consumer Claude and asking one sentence — what are the most important points? — because those posts do not fit in GPT-4. He still read them first, on his phone, without notes; the model was a memory aid, not a replacement. Mowshowitz has been running a different experiment: one long Claude thread, no new chats, to see what happens as context accumulates. Anthropic's bet, as he hears it, is that once the model is "harmless," extra context can buy back usefulness.

A separate Anthropic result, on chain-of-thought and task decomposition, fits the vibe theory from earlier. Highest accuracy and reliability came not from a practitioner-style mega-prompt that sequences many jobs in one generation, but from discrete, isolated calls — do only this, then only that, then assemble. Labenz wonders whether subcircuits get drowned by a crowded context. Mowshowitz says he would have predicted the finding: a model given two tasks can only vibe off their aggregate, the way an image model fails when two scenes overlap. Rate limits and API pricing have trained users to ask for everything at once and walk away for coffee. The better procedure is to break the job into the smallest possible pieces, perhaps after asking the model how to split it. Humans under fine-grained direction do better the same way, logistics aside. Elicit, the literature-review product from an earlier episode with Andreas Stuhlmüller and Jungwon Park, is built on exactly this principle: many small, auditable steps, some on fine-tuned models, some on frontier APIs.

Will a proverbial GPT-5 flip the result, because more context finally helps? Models will not get alienated from their labor, Mowshowitz says; the question is whether parts of the task actually need each other's context, or whether extra context is just distraction that makes vibes clash. He expects, within a year or two even without a new scientific leap, a routing layer: classify the query, decide whether to split it, and send each piece to a specialized model so you do not spend GPT-4 prices on a narrow job. On the personal question hanging over the Anthropic segment — should someone like Labenz take a red-team engineering role there? — Mowshowitz is conditionally positive. Work only on alignment, not capabilities; stand up when something is wrong; be willing to quit on a moment's notice; and be willing to tell the world why. Red teaming is one of the jobs where the sign is most obviously positive. The interview is two-way. He still would not work on capabilities for any company, and he still would not hand any company a capabilities idea.

Google DeepMind's week looks like bombshells from a year ago and like the calendar catching up today. RT-2 puts a language model in the loop so a robot arm outputs commands directly: move this object onto the Denver Nuggets logo, pick up the extinct animal from a table of toys and take the dinosaur. Two years ago it would have felt like The Jetsons. Mowshowitz's grade is mild good news. You already knew language models could parse those instructions and that arms could execute those motions; combining them is the inevitable next paper, not a new dragon. He plays the game of good-news-or-bad-news: if you thought robotics was mysterious forever, every increment is a scare; if you knew it was another technology that would eventually work, this result is fine, even slightly calming for someone who does not want capabilities to sprint. "AI will not have a body" was never going to be the barrier that saves anyone. At best it is a temporary inconvenience that forces the system to be slightly more clever.

Multimodal Med-PaLM is the paper Labenz is happy to accelerate. Med-PaLM 2 already beat physician answers on eight of nine evaluation dimensions in text. The new system takes pathology slides and radiology scans as well, and its radiology reports were preferred to a human radiologist's more than 40% of the time — on par, after a decade of people saying radiology would go first and a recent talking point that it still had not. Mowshowitz does not doubt the model. He doubts the bottleneck was ever model quality. Radiologists, he says, were trained to be computers because we did not have computers. The obstacles are cultural, regulatory, legal, contractual, and incumbent. Eliezer Yudkowsky's old skepticism about pre-singularity growth was the same point: we already know how to build cheaper houses and more efficient healthcare, and we are not allowed to. If the radiologist stubbornly double-checks everything and substitutes his judgment, the system does not help, and the lab does not recoup the training bill.

Leapfrogging into countries with no radiologists is not the easy out Labenz hopes for. Those markets are poor and small, have often stripped market signals out of healthcare, and nobody wants to be the person who sticks their neck out for a tool rich Americans will not use themselves. Mowshowitz analogizes to COVID: the world's elites converged on a consensus, and no one ran challenge trials. As a Google shareholder he is blunter about the consumer stack. Bard still "sucks." Gmail and Docs generative features are ornaments. He pasted an article into Docs, asked for a summary, and was refused: if you cannot read the document I just gave you, why are you here? Email is worse — by the time he has specified the request, he could have written the message. The diagnosis is one letter: these products lack the G. Customization around a weak core does not help. The exam is Gemini. Demis Hassabis said after the Gato paper in April 2022 that they were already scaling it up; more than a year later the successor is still missing. GPT-4 itself sat about six months between the end of training and the first release, and Google is more risk-averse than OpenAI. Even if Gemini finished tomorrow, months of caution would remain. Publishing a pile of narrow papers can look like shipping. It can also look like not shipping the one thing that matters.

Labenz quotes Emad Mostaque of Stability AI: the leaders are non-economic actors. OpenAI is on the record trying to drive the price of intelligence as low as possible. Google is defending a franchise. Anthropic is a safety-first play and does not look like a revenue maximizer. Meta pushes the pattern to an extreme: tens of millions of dollars to train Llama 2, then give it away. They did not dump a naked pretrained checkpoint; they ran red teaming and RLHF, the responsible-lab checklist. Mowshowitz still reaches for "idiot disaster monkeys," then tries to answer the business question. The strategy is cannibalizing the complement: foster an open-source stack that replaces rivals' specialties so Facebook's social products compete on a level field. Android is the precedent. Alternatively they need free labor from the community, or they are recruiting the tribe that believes software wants to be free. From a let's-not-all-die perspective, he says, it is crazy. "Creating frontier models and open sourcing them is the worst thing you can do."

If the United States is serious enough about China to run export controls, it should not be releasing Llama 2 as an open-source product. Senators already asked what-the-hell about Llama 1; they should ask a louder one now. Yann LeCun and others at Meta either sincerely believe there is no danger from AI, "despite this making absolutely no physical sense," or they do not care. Mowshowitz will not psychologize further, except to note they are smarter than the arguments they are making. Zuckerberg told Lex Fridman that future models will need great care and that for now open-sourcing does not. Mowshowitz's reply: you are raising an open-source community and a set of precedents that will pressure you later. Labenz cannot make the P&L work. OpenAI is GPU-limited; token volume will not collapse. Quantization tricks and on-device hacks accrue to Google, OpenAI, Anthropic, and Baidu within a month. Mowshowitz agrees. "I'm glad they suck." If Meta were as good as OpenAI, we would be in a lot of trouble.

Llama 2 is, Labenz protests, the first open model on par with GPT-3.5. Mowshowitz does not disagree, and he does not think that means Meta maxed it out. The loss curves in the paper have not flattened; even the 70B run looks like more training would help. What they published reads as the standard-issue recipe at every step, a marginal gain that is mostly more money than Vicuna or Hugging Face could light on fire. Alignment is ambitious, crude, and optional. Reports he has heard, and not sought out, say an unaligned Llama 2 was on the internet within several days. Fine-tuning a model to never refuse is the easiest fine-tune there is: negatively reinforce every "I can't" until it stops. Refuse Arabic in the original checkpoint if you want. It will not last. Meta is a live player because it is willing to spend, not because it invented anything. Zuckerberg has already shown he will set money on fire.

Labenz has another half-dozen names. Character.AI and Inflection have large budgets and, in Mowshowitz's view, no intention of being live; they are "content to be dead" and make money that way, which is fine with him. xAI is Elon Musk's latest attempt to string words together as if they meant something about alignment. The good news from the OpenAI years is that engineers learned to ignore Musk's suggestions; if xAI's engineers do not, they will not ship anything useful. The live question is whether they get enough compute to be a rival. Salesforce and Replit do not worry him as model labs. The real wild cards are states: the UK wants a global summit and a safety-and-prestige play; Congress is starting to get up to speed; the EU's rules will decide whether this is a game for everyone or only for the largest firms. China is the biggest unknown. He hears both "fanatics bent on world domination" and a regime that has issued guidance that basically bends all large-model deployment. Nobody in the US has picked up the phone to ask whether they would cooperate on safety. As in *Oppenheimer*, the only people actually racing are "us."

Rank the published stack and you go several names deep — Meta, Microsoft, probably Inflection soon — before you hit Baidu and Ernie Bot. Palantir does not sit on his threat model as a driver of the underlying technology, though a military-ish lab stripping safeguards is a classic way to die. Hedge funds such as Bridgewater could be invisible live players; he does not know. Then the line Labenz has been waiting for, the one from the cold open: "We talk about worrying about China, but I'm more afraid of Meta. One individual American company scares me more than all of China right now." Labenz calls it a useful corrective to detail-free China talk. Mowshowitz would still take countermeasures against both. Revealed preference, he adds, is that America does not act like a rival it cares about beating: Chinese STEM graduates do not get to stay.

Labenz pushes Character.AI, Inflection's Pi, and Replit as different vectors of transformation — companions, coaches, a substrate where generated code actually runs. Mowshowitz does not see a path to recursive self-improvement or AGI from companion apps, and he notes they are mostly wrapping GPT-3.5 or GPT-4, with a classifier that can strip jailbreak scaffolding. Replit's CEO has called the product the perfect substrate for AGI and once mused that some AutoGPT variant had a non-zero chance of taking over Replit's servers; Mowshowitz's reaction was to put up a Manifold market and to file the attitude under "idiot disaster monkey." Providing a place to run other people's code does not make you a live player. A LessWrong post about a man who knew how language models work and still talked himself into an AI girlfriend does not move him much either. People fool themselves; the tech, to him, is not even good enough yet to explain a two-hour-a-day habit on GPT-3.5. That market is buying an experience, not intelligence, so those companies will fine-tune small models rather than train GPT-5. The live-player list stays short: OpenAI, Anthropic, Google DeepMind, probably Meta, regulators, and China as a wildcard. Anthropic is barely a year old and still under 200 people. A team that says "foundation model" can still raise hundreds of millions by asking nicely. Default path for the companion startups is product, not frontier.

The White House convening, the commitments, and the Frontier Model Forum look to Labenz like a live-player invitation list plus a venue for sharing classifiers and other public goods. Mowshowitz's reaction: great, do not get ahead of yourselves. Cheap talk is underrated. The right cheap talk is much better than no talk. Labs will pay a price with audiences that do not want safety, which is useful; they can justify the work to boards and shareholders; they get a coordination channel; and an announcement on the White House lawn is, he notes, something he had explicitly wanted, because it punches an antitrust hole. You can now coordinate to not kill everyone as fast as possible without being accused of declining to compete. The commitments still lack enforcement and concrete content. Two strong Senate hearings, especially with Senator Blumenthal, are the same story: a step, not mission accomplished. "The real work begins now."

Mowshowitz is a recommender this year for Jaan Tallinn's Survival and Flourishing Fund, reading on the order of 150 grant applications. Asked what the bottlenecks are, he refuses a single limiting factor. Funding still binds once you are buying serious compute and paying capabilities-competitive salaries — hundreds of thousands a year, maybe a million, plus millions in compute per senior person — but that money need not come only from the old EA pots; foundations, billionaires, governments, and even the labs themselves are in play. Talent binds for a narrow kind of person: leadership, self-direction, fundraising, and people who understand the Yudkowsky difficulties and will work on the hard problems rather than alignment-flavored puzzles that publish every six months. Generic undergrads who want to spend six months in Berkeley are not scarce. If you have that scarce mix, "it would be a major mistake to join an existing organization and try to make a difference as an individual" rather than spearhead a new org or a new branch. Labenz's phrase for the same gap is research agendas; Mowshowitz agrees you cannot hand a plan to someone who does not already see why alignment is hard.

On policy shops, he looks first at the goal. The ball to keep eyes on is compute regulation: permissions, tests, and eventually limits or a halt on the largest runs, plus GPU tracking that makes those limits enforceable. Anything that does not lead there may help mundane utility and will not save us. Liability with mandatory insurance makes sense for harms up to a point — force labs to internalize fat left tails — but you cannot price existential risk, because when it hits the insurer is dead too. He wants actual-damages liability and is wary of asymmetric justice: full punishment for harms, almost none of the surplus from benefits, which would tax mundane utility while missing the big risks. Independent red-team organizations are worth standing up, three to five of them with different tests, not thirty. ARC's first GPT-4 run was a trial of the gear, not a real stop. The danger is labs patching the narrow demo, treating a passed checklist as safety, or ignoring the finding; Nathan's own ransom-call recording of an un-jailbroken phone agent was mostly a yawn on Twitter even though the vendor did fix it. Mechanistic interpretability is, in his line, "like Western civilization: it's a good idea." Do it, in parallel, with a culture that can sit on a dangerous finding instead of rushing it out. He has been positively updating on all three major labs, on government, and on the public; the negative updates have been elsewhere. The doom tour, so far, is going better than he expected when he started.