The Former Trader Who Writes 10,000 Words a Week Puts AI Doom at 60 Percent

Nathan Labenz opens the July 11, 2023 episode of The Cognitive Revolution by calling his guest an information hyperprocessor. Zvi Mowshowitz writes the blog Don't Worry About the Vase, at thezvi.substack.com, and has spent more than three years turning firehoses of news into weekly summaries that educated readers can actually finish. His résumé cuts across math, trading, economics, game theory, and game design. Labenz wants that background on the table before the AI-safety arguments start, because Mowshowitz's role is meta-analyst: he is trying to say what in the discourse actually matters.

Mowshowitz has been thinking about AI for decades. He came in through the FOOM debate between Eliezer Yudkowsky and Robin Hanson — the argument over whether an intelligence explosion could happen almost overnight — which is also how he found the rationalist community. From then on he expected AI on the horizon and expected the default outcomes to be quite bad unless someone actively did something else. That expectation is more than a decade old. What he did *not* do, for a long time, was make it his job.

He made a deliberate decision not to have shower thoughts about alignment. The work on offer in the 2000s and early 2010s looked like heavy theoretical math, and he did not think his math was strong enough. Competitive advantage, he figured, lay elsewhere. He had been a competitive gamer; he became a trader of various kinds; he tried to start a few businesses. Then 2020 arrived, COVID scrambled the world, and he started writing in earnest. He already had a blog. What changed was the reason for opening it.

Writing, he says, is how he processes the world. You write down what you are thinking, try to explain it to somebody else, and only then do you actually know what you think; the page forces you to set sources and justify reasoning. People found the COVID posts useful, so he kept going, and the habit snowballed into a profession. When COVID receded he joked on Twitter that the weekly COVID post would be replaced by a weekly AI post, as if that were a nightmare. The replies said yes, please. He tried it. He has been writing 10,000-plus words a week ever since.

The weekly project has two pillars. One is a one-stop shop: all the links, so that a reader who ignored AI all week could check back once, see what mattered, and click through or walk away. The other is slower: building a world model over months of words, so people can see *how* to think about what is happening, not only what to think. That only works if he can ingest more than he publishes. Even for him, there is now too much.

Audio is especially expensive. He can process complex speech at about 1.3 times speed, and for the densest arguments he wants 1.1; a transcript of the same thing is at least twice as fast, probably three or four times, and he can pause on purpose. Three-hour podcasts therefore get cut unless the case is strong. Reading runs through Twitter lists — a rationalist list, an AI list, plus followers — a Feedly stack of RSS, and LessWrong and the Alignment Forum as tripwires for anything the lists missed. Labenz is almost the opposite machine: he listens at 2x, and the scarce resource is time staring at a screen.

Language models occupy a corner of this workflow, not the center. The writing is all him. He has found current systems very bad at summarizing the things he actually talks about, and bad at expanding prose in the style he wants. He does subscribe to GPT-4, and he uses Bing and sometimes Perplexity, mainly to ask about the world, check whether a model shares an intuition, and request more examples. Starting a piece on debate, he asked for ten things that are like democracy in the Churchill sense — the worst form except for all the others — and got five that were genuinely good. For coding, when he does it, productivity is "through the roof."

Labenz has been experimenting in the other direction: talking to a phone app while walking his kids, then annotating the model's draft for two or three rounds until it is perhaps 80 percent of the way there. Mowshowitz's reaction is to ask where the bottleneck actually is. His own limiting factor is knowing what he wants to say. Once that is in place, the typing is almost free, and a sloppy first draft would not speed him up. The lesson he wants is not a universal stack. Figure out which part of *your* process is hard, then use the model there, instead of copying someone else's ritual.

Mowshowitz starts his worldview with a complaint about the debate itself. Whenever you argue with someone, you rarely know which of twenty unclear points they actually disagree with you about. The fight shows up in a different place; the real crux may not even have occurred to them. Sometimes there are seventeen disagreements with no common cause, only a vibe and a desired end result. He wants to put his own cards down first so listeners can see the joints.

Layer one: today's language models are an incredibly useful tool, almost entirely positive. He thinks most fears about misuse and near-term harm from current systems are overblown, and in some cases pointed the wrong way. Layer two: that verdict flips as AI approaches and then passes human-level intelligence. Lots of hard-to-predict things start happening. Pretty much by default, he says, the outcome is the death of every human, or at least the loss of control over a future that most people — including himself — would see as having little or no value compared with an ordinary non-AI future.

The argument is about comparative advantage, not mysticism. A more capable optimizer can be copied, runs faster, can rewind and fork its state, and has essentially limitless memory and orders of magnitude more data than any human. The missing piece is the kind of reasoning humans still do better. If it mimics that well enough, the sky is the limit, and it will optimize whatever it is pointed at — even if the builders insist they are only making a chatbot with no goals. We already give these systems targets. We have no idea how to make them follow priorities we actually control, and no idea which priorities, if we could control them, would not ensure disaster.

He thinks almost everyone, himself included, is thinking poorly about at least some step in this chain, and that the errors will be exposed. The problems also have to be solved on the first try: the moment a system crosses from non-dangerous to dangerous is exactly when many techniques that used to work will stop working, and the species may not get to recover. People talk about alignment as "some sort of weird Boolean," as if figuring out how to align a system meant everything was aligned and therefore fine. He calls that a serious way people fool themselves into thinking the problem is easier than it is. Even aligned individuals can produce competitive dynamics that do not end well for us. Human history is the history of incredibly stupid mistakes patched by trial and error. This time the room for trial and error may be very small.

Labenz tries a first-order summary: radical uncertainty, some chance things go great, a significant chance of catastrophe, details fuzzy. Mowshowitz says yes and no. People love to ask for p(doom). His first question is where that number would actually change what you do. If you are in the single digits, you start comparing the doom of inaction with the gains from pushing ahead, and maybe you take some chances. If you are in the 90s, you start considering actions that would not make sense if a good outcome were still the base case. In the middle, very few actions are rational at 10 percent and crazy at 90, or the reverse, because the payoffs and costs are orders of magnitude larger than that one-order gap.

That is why, when he has to guess, he is willing to say something like 60 percent, and why he does not think it pays to pretend the digit is stable. As a Bayesian his number should be moving constantly. He does not claim every scenario is equally likely; he claims they are all likely enough to spend serious time, money, and sacrificed utility on. FOOM, the fast takeoff, is in his guess less likely than not, but both fast and gradual paths are live, double-digit possibilities. If we lose, he would split the failure roughly in half: we fail the alignment problem early, or we "succeed" at alignment as currently discussed and then specify something that kills us anyway because we misunderstood the dynamics.

The question he actually cares about is simpler. Are these real things we have to deal with, or not? His answer is that all of them are. Exact percentages of where we go wrong more often strike him as interesting and not very useful. Labenz is in the same place: the lower bound is already high enough that most people should take it seriously, and tightening a confidence interval is a way of getting nerd-sniped into false precision.

Pressed for the simplest possible form of the danger, Mowshowitz reaches for goals. We are going to tell these systems to make the most money, make the most people happy, get us the most dates. "The moment you say most on top of anything," he says, you get power-seeking and resource-seeking from things that are better optimizers than you are. Selection for more copies, more instantiations, more of whatever statistically convinces humans to run another instance, points the same way. By default those systems end up with the resources and the levers of power, and that is not going to go well for us. Labenz says he has never heard a good reason to dismiss that as a concern.

Labenz offers a stack of who matters. Tier one is leadership at the leading developers — OpenAI, Anthropic, DeepMind, maybe a dozen companies, on the order of a hundred decision-makers. Tier two is policymakers and researchers who could actually advance the field, maybe 10,000 each. Tier three is the discourse: public intellectuals, VCs, people with reach on Twitter who are not making object-level calls this quarter. The general public sits at the bottom and feeds back into elections and permission to operate. Mowshowitz does not think you can answer the targeting question in isolation from what actually needs to happen, but if he had to pick one person to pull toward his view, he would pick Sam Altman. Altman, in this sense, is more important than Joe Biden.

Next on his list are Demis Hassabis, the CEOs of Google and Microsoft, and Mark Zuckerberg: the people highest on the totem pole who clearly do not take the issue seriously, and who would make the situation substantially better if they did. Politicians are a messier lever. He is not sure what would happen if Biden were convinced, how much compute an old president with infinite demands on his time has left for a technical argument, or how much real opinion versus negotiation the system even transmits. One disagreement with Tyler Cowen is that Cowen puts a lot of weight on academic consensus — papers that convince academics, who convince behind-the-scenes policy people, who maybe convince lawmakers. Mowshowitz finds that vector confusing, possibly closed, and hard to run experiments on.

The public path worries him for a different reason. People are already pretty worried about AI when you show them a system and ask basic questions, but a scared public will not hold the very specific message he thinks is required: control the training of advanced frontier models, because that is where the danger lies. They will latch onto whatever they can see, block useful current applications, make lives worse, and leave the real threat untouched. So he does not write for the general public. He writes the thing he would want to read, aimed at people who can handle dense information, and counts on distillation down a pyramid: level one generates new ideas and synthesis, level two can process level one, level three can only process level two.

He hopes Altman, Hassabis, and Dario Amodei are reading, and he knows he has at least one reader inside each of the three major labs. What worries him most about OpenAI is not that Altman fails to understand the problem. "I think Sam Altman understands the problem pretty well," he says, "but it seems clear he hasn't cultivated a culture of safety amongst the people he hired." Culture drives company action, so Altman has less ability to steer than Amodei does at Anthropic, even if Mowshowitz is not convinced Amodei understands what it actually takes to be safe better than Altman does. He credits Altman with recently backing strong regulatory principles, which he calls tremendously helpful, and he wants Jack Clark and Amodei to say out loud more of what he thinks they would endorse in private. Hassabis and the Microsoft and Google CEOs should speak more too.

A short list of individuals could still flip the board. Yann LeCun, the remaining AI "godfather" who has not joined Geoffrey Hinton and Yoshua Bengio in publicly warning about the risk, is the loudest example; a profound, loud update from him would, in Mowshowitz's view, overturn the chessboard by itself. Douglas Hofstadter, author of *Gödel, Escher, Bach* and *I Am a Strange Loop*, thought AGI was far away until very recently and is now saying out loud that what is coming horrifies him. The prime minister of the United Kingdom coming out as concerned also made a significant difference. Any senator who explained a careful version of the worry would buy credibility. Hundreds, possibly thousands, of people not currently in the debate would matter if they suddenly got loud.

Labenz asks whether this is a river that will arrive wherever it was going, or a history in which local decisions still bite. Mowshowitz thinks the path of the river will determine the outcome, with a lot of model uncertainty about whether the worlds we might be in are mostly good, mostly extinction, or up for grabs. Some forces are brutally hard to stop: Moore's law until physics stops it, algorithmic and hardware improvement in general, things already released, profit motives, massive investment. Those ships have sailed. The path from there to *here* was still detail-dependent. The people who founded all three major labs pushing the frontier did so specifically because they were downstream of Yudkowsky and his warnings about AI danger. That coin has two sides: capabilities moved faster than they otherwise would have, and people with a much better-than-otherwise understanding of the problem sit in authority over the labs.

If a very small number of people, potentially even one, had made substantially different decisions about what to emphasize, the discourse and the state of AI would look different today. Arguments that "whatever is going to happen is going to happen" amount to saying it is already too late, or that competitive dynamics and weak human coordination make sacrifice pointless. Mowshowitz's reply is that one of the biggest barriers to coordination is people assuming they cannot coordinate, and therefore not trying. On China, he sees no strong evidence that Beijing is in a mood to cooperate on world-transforming technology — and also no sign that the United States has taken the first step of asking nicely. "There is no we and there is no them until we understand collectively what has to happen."

Even entertainment bent the river. He asks listeners to imagine Don't Look Up never getting made; the film had nothing to do with AI on the surface and still changed how people looked at expert warning. The Terminator is the other obvious fork: a bad director, no Arnold Schwarzenegger, a forgettable action movie, and decades of metaphor go missing. He has been watching *Person of Interest* and is surprised how much it gets right about a technology the writers could not have used: how language models perform, what big data infers, what capabilities emerge, who should have power, how you limit the system, what alignment tax you pay. The words are different. The questions are not.

Labenz sketches the usual map: effective accelerationists who say there is nothing to worry about; an AI ethics community focused on current harms, sometimes accusing safety people of marketing hype; a centrist AI-safety camp that includes most lab leaders except LeCun; and doomers who think it is time to spend more time with family. Mowshowitz's map is close but not the same. He starts with the accelerationists. He describes himself as effectively e/acc in most domains, the way an atheist is an atheist for every god but one. They are not making the AI exception. They do not see why this technology is different from fusion, new medicines, VR, or going to space — the things he *does* want — and they have heard "this will destroy our way of life" a thousand times.

Some of that skepticism is a fair view of one side of the elephant. What is going on, in his account, is that people do not understand why AI is different, choose not to, generalize into blinding optimism, or see no alternative and present a front of "do it anyway." He singles out roon on Twitter as a serious accelerationist in the good sense: someone who fully acknowledges that it is very easy to get everyone killed, and who advocates doing it anyway with eyes open. "The spice must flow," as roon puts it, borrowing Dune. Others share the view without saying it: if we do not build AGI, civilization has cut off so many other avenues of progress that we are due anyway. Some buy a China argument — better us than someone less responsible. He tries to give people the benefit of the doubt. He thinks most accelerationists want a good future for humans.

Past e/acc he places a sixth category he calls extinctionists: people who think the future belongs to AI, call it speciesist to prefer humans, and are fine if we die. He disagrees, completely. Humans are allowed to have preferences. He does not have to justify wanting children and grandchildren to someone who treats that as an illegitimate bias. He also flatly denies that "AI safety hype marketers" exist as a real camp. Accusing everyone of talking their book, as a Pirate Wires post had just done, is either projection or a cynical failure to imagine that people on several sides might mean what they say. Wanting to stop a technology because you think it is a problem is not the same as a subscriber grab, and pretending it is does not make the arguments clearer.

"Doomer" he treats as a pejorative he will not use; he prefers "the worried." Paul Christiano, often filed in that bin, will give a p(doom) of 10 or 20 percent and still talk like a doomer, because he thinks that if you play the problem on Guitar Hero medium you will definitely die unless you get super lucky. You have to treat it as an impossible-difficulty problem and shout how dangerous it is so that an extraordinary effort becomes possible. Others put p(doom) above 90 or 99 percent, or think that without a doomed mindset none of your solutions will even point at the real problem. Ten percent, Mowshowitz notes, is already worried enough to justify action that will not happen on its own.

The ethics camp, in his telling, often puts bias, misinformation, and other near-term harms above whether humans live. He thinks those harms matter and that the solutions are complementary: anything that keeps AI under control also helps with bias and manipulation; work on the near term is at least not harmful when it is actually aimed at the harm. The fight starts when either side calls the other a distraction and treats attention as zero-sum. Labenz floats an omnibus: $100 billion for you, $100 billion for them, a big-tent spending bill. Mowshowitz looks unimpressed. He does not think this is a pour-money-on-it problem. You have to impose actual costly restrictions on what models can be developed, deployed, and used. That is a scarcer resource than a line item, and it is why the camps end up at each other's throats.

He also thinks some ethics arguments are incoherent on their face. The same people will say AI will never be able to fool humans enough to take over the world, and that AI-generated misinformation will wreck next year's election. You can hold either view. Holding both is hard. His bus-over-a-cliff metaphor — a passenger worried about being late for work — is one he cites as both clarifying and unfair, a thing that pisses people off for no good reason and should not be the go-to. The deeper grievance is linguistic capture. They used to call their concern AI safety; the ethics world walked in, used the phrase to mean roughly "don't say bad words," and the original camp had to find new words. He thinks many near-term fears are overstated, and he thinks bioethics-style deontology has already killed people — COVID vaccine challenge trials that willing volunteers could have run, and did not, because ethicists said no.

Asked for the best accelerationist case, he steelmans rather than sneers. Waiting creates overhang: unused capability that lets bad actors move first. Only by getting close to AGI can you do the research that makes it safe, so time spent far from the edge is time stolen from the hard part; race to the cliff, then run your Manhattan Project. He thinks those are not facetious, and also that they do not carry. The weak arguments he names out loud. Marc Andreessen saying the good guys always win is content-free. The idea that a sufficiently intelligent system would notice its goals were stupid and become moral — a line he has seen from people including Scott Aaronson — has, in his view, no reason to be true. Intelligence is not a moral attractor.

The optimistic story he can take seriously is messier: LLMs trained on human feedback inherit a fuzzy human niceness; goodness is an attractor state; RLHF plus Constitutional AI plus correcting bad behavior as it appears is enough. Labenz notes that language models sit far toward the lucky end of what he would have expected five years ago; an AlphaZero-style lineage would not have this human flavor. Mowshowitz says we may have been lucky *or* unlucky. Fuzzy systems are inscrutable, hard to notice going off the rails, and hard to inject a precise target into. He is more moved by grokking: a phase change from memorization to generalization, one or two orders of magnitude more scale, and the model suddenly finds a different way to get the reward. Overnight, the previous alignment methods are useless. And even if you did get a crowd of messy, human-ish, better optimizers, competitive dynamics still fail the test. You would not survive long. You would never meet your grandchildren.

Labenz notices an apparent split. Mowshowitz often sounds consequentialist about AI risk, then refuses to say that a universe of high-utility machines would be good enough if humans were gone. Asked to unpack the principles, Mowshowitz calls himself a virtue ethicist first, if he has to pick one of the big three. He also thinks that if you cannot express your view in consequentialist, deontological, *and* virtue-ethical language at the same time, the view is ill-conceived and you have not figured out what you want. A utility function should be definable; with human amounts of data and compute, virtue ethics is the right way to *express* it. Deontological rules are useful for the same reason: the calculation is too hard, the universe too uncertain.

What he will not do is "just do math" on fake precision. He knows people who assign a value to each chicken or each worm, then notice that simulating enough worms having orgasms on enough GPUs would swamp everything else. That, he says, is taking a metric as if it were real, then running an approximation out of distribution — exactly the failure people worry about in AI. Utilitarianism is a good correction for the fact that two of something is more than one, and that twenty is sometimes ten times two. It is not a machine you can grind inexact equations through until a contradiction appears. In a formal system, a contradiction lets you prove anything. Simulated-worm utopias are the anything.

If AIs wipe us out, he scores most of those universes as close to zero utility. The AI is not good or bad, does not love or hate him; it is using his atoms for something else. Pure power-seeking is not interesting and not valuable to him. He reaches for science fiction to make the allowed-preference point: if Vulcans live on their world and Klingons on theirs, humans are allowed to prefer that humans survive. Parents have to care more about their own children than other people's, or the whole system falls apart. Some self-described utilitarians, he suspects, do not actually prefer on reflection the world they claim to prefer. That is a different kind of disagreement from a fight about facts.

He is not saying multiplication can never matter, or that no one could ever convince him to rank some non-human futures above others. He is saying we may soon set the configuration of the universe without knowing what we actually prefer at that scale, and that our feedback will not reflect it. If he could have a long reflection — himself and the people around him not aging, a hundred thousand years to do philosophy — he would take it. He does not have it. So he notices a brute preference and is not embarrassed by it: he prefers not to die, prefers his children not to die, wants them to have children of their own, and wants the human race to continue. He is committed to that preference, thinks it is good that he has it, and welcomes attempts to talk him out of it. He does not expect anyone to succeed.

Labenz asks for the short history, including the "respectability politics" of trying not to sound crazy. Mowshowitz goes back to those Yudkowsky–Hanson posts. If you built something smarter than a human, it could by default build something smarter than itself, scale its resources, and take control of the future unless you could preserve, in a known and provable way, what it was optimizing for — and specifying what we wanted was already remarkably hard. The plan then was not language models. It was GOFAI: mathematical systems that would let a self-improving agent keep a utility function. He was in a meeting with people from the organization that became MIRI where they laid out seven things which, if solved, would solve alignment, and along the way you would have built the AI. He was then told the list was an infohazard. He forgot most of it.

The mathematical agenda did hard work, wrote interesting things, and did not arrive at a solution. What the community *did* produce, he argues, was a better decision theory. Causal decision theory will not cooperate in a prisoner's dilemma, folds to threats, and cannot make certain credible commitments; evidential decision theory sounds bizarre and does dumb things in other places. Functional decision theory — act as if you are choosing the output of the logical process you are using, including for decisions correlated with yours elsewhere in space and time — is, once you see it, vastly better. There are long, carefully referenced papers. They mostly disappeared into the void. Meanwhile Yudkowsky wrote the Sequences because he had tried to warn people about AI and concluded they could not think well enough to understand him. Mowshowitz thinks that project worked to a large extent, just not as far as the original plan needed, and he is grateful for it anyway.

Then DeepMind happened, then OpenAI. The origin stories run through Altman, Elon Musk, worries about Google, and things Larry Page allegedly said about speciesism. The labs were founded by people who say they were worried about the power and danger of AI. For years the community had treated capabilities ideas as infohazards. It turned out the most dangerous idea was that AI might *kill everyone* — that got people excited to get involved. They went down a road of systems no one could control or even in theory understand, throwing compute at them. Trying to get a voice inside, he thinks, was mostly a mistake. They should have been more outspoken from the beginning about not wanting this road. The bridge is burned. The concerns no longer sound crazy, and they now have serious people in the industry endorsing them. That is a tremendous place to have arrived, and also a world of real systems where "incremental alignment progress" may or may not transfer to the dangerous regime. That transfer question is its own debate.

Labenz sees the Yudkowsky school as having partly won: the people building the systems have a healthy fear, and Transformers plus web-scale data were coming anyway. Mowshowitz's counterfactual is bleaker and less certain. No DeepMind, no OpenAI, no Anthropic, and Google would have moved quieter, slower, and in fundamentally safer ways, without an orders-of-magnitude explosion of money. Eventually a less-funded lab or an open-source effort would have built something GPT-style some years later, and the race would start then. A little safety concern may have been as inevitable as a little capability. The worry is whether "safety is a thing we have to solve" is the same as a MIRI-style security mindset — including the possibility that a system kills you during the training run, not after deployment. Anthropic claimed that posture; its pitch deck and what Amodei and Clark will not say do not, to him, reflect that level of paranoia. There is a further twist: RLHF made ChatGPT releasable, which made the investment explosion possible. Google had strong models for years and would not ship them. Alignment work, on this telling, helped cause the boom.

So what now? Labenz asks whether the Overton window is finally wide enough for honesty, and whether Yudkowsky saying governments should be willing to call airstrikes on rogue data centers helps or lights the conversation on fire. Mowshowitz does not want to be dismissive of the recoil, then points at ordinary law. Recording around the Fourth of July, he notes that people celebrating free speech could also read about a woman in England sentenced to roughly thirteen months in prison for social-media posts. Law is already men with guns. To prohibit something is to say you will use force if necessary, and escalate if the other side has guns too. Yudkowsky is owning up to what legislation means.

The actual proposal, as he describes it, is milder than rules we already apply to many less dangerous things. It is not a speech rule or a privacy rule. It is a restriction on a very specific use of very specific technical hardware, manufactured under some of the hardest conditions humans have ever managed, already under strong export controls aimed at China. If you want to use that hardware to intentionally create a more powerful optimizer than a human, you need permission, and maybe nobody gets that permission anytime soon. If Russia smuggled enough A100s into a data center to train something that might kill everyone, you would do what you would do about an accidental nuclear launch: take it out. The point of saying so is that if you make the commitment credible, they do not do it. If you announce that you will not build the thing but cannot stop anyone else, they do.

Follow the logic of the China chip controls, he says, and it is incoherent to let anyone in the world rent the same compute, or to encourage open-source models powerful enough that China can copy and run them. You have to draw a line on capabilities. The only way to prevent these systems from coming into existence that does not smash ordinary liberty and economic value — and that might actually work — is to target chips, concentrations of chips, and intense training runs for frontier models. Where to draw it, 10^23 FLOPs or 10^25 or 10^26, how to use evaluations, how to handle a gray zone, how to update as algorithms improve: those are detailed questions he does not claim to have nailed. He does not see an alternate proposal that can possibly work. Application bans and "don't turn them into agents" will be circumvented.

We are, he thinks, very fortunate that the thing which enables AI training is the most complex hard-to-manufacture object in history, and that friendly countries under U.S. economic dominance control the supply chain. That did not have to be true. Given that it is, there is a unique chance to exert real control. They have to take it.

Mowshowitz does not love any current alignment approach enough to be optimistic, but he sees things worth trying: they can at least map ways *not* to align a system, surface dangers, and maybe get lucky. Mechanistic interpretability is dual-use — it also helps you utilize and strengthen models — and he still wants a significant chunk of resources on it. So far it has not gone that great. The test that would actually matter is something like GPT-5 interpreting GPT-4, because otherwise you have a chicken-and-egg problem: you cannot trust the thing that is telling you what is going on. We have not hit the vastly easier target yet. Regulation has to go hand in hand with that work, because algorithmic progress does not give infinite time. Limits buy time. They are not a destination.

On who should do what, he wants division of labor, not a single heroic path: some people on interpretability and safety, some lobbying, some papers, some blogs, some inside the labs. He plans to try getting papers ready despite low hope, because the attempt will show how it fails, and because writing is how he understands things. He despairs of the speed of the academic route Cowen likes. Econ papers convince economists; then you repeat the process twenty times for twenty audiences, many less amenable than Cowen. At the end of every podcast someone asks what to do if they are terrified, and the answers — orient, think — are not good enough. There are now organizations he thinks are net positive in expectation. Throwing money into that ecosystem without understanding it warps incentives. Orient first, then write the check. Lightspeed Grants exists for people who want a formal on-ramp. There is no GiveWell-simple default.

The day of recording, OpenAI announced it would devote 20 percent of its *acquired* compute — which will be much less than 20 percent of future compute if the company keeps scaling — over the next four years to alignment, aiming at a more automated process. Mowshowitz is deeply skeptical that the listed agenda can scale to human-level or beyond. Building a human-level alignment researcher, if that means putting AIs on the alignment problem itself, is a commonly nominated worst idea: you make one of the hardest possible tasks the first human-level job, and you have little way to tell if the system is actually aligned. If they only mean a human-level *evaluator* of outputs, that is less insane, because humans are already bad at that job. He still thinks it is a bad idea. You cannot learn from training data what is not in the training data. You need sophisticated feedback from people thinking carefully, not random hourly raters, and even then Goodhart's law will bite: the system optimizes the procedure you are actually running. Stack an AI helper on that and you get meta-Goodhart, a telephone game. He does not know that there is much hope in it. He also does not have a solution. It is easy to shoot everything down. What will work? "The answer is I don't know."

For readers who want to approach his level rather than only subscribe, he says ruthlessly steal his sources — his rationalist and AI Twitter lists, the blogs he links — then keep or drop them based on your own reading. Decide how technical you want to be, and whether you are in mundane capability news, discourse, technical alignment, or policy; after a few weeks, discourse-following starts to rot. The binding constraint is people willing to work the technical problem. There are only a few hundred alignment researchers on the planet, which is orders of magnitude wrong. If you can demonstrate skill and careful thought, plenty of people will help you try. He would discourage trying to become one more real-time AI commentator unless you are already in that seat. Write to help yourself understand. Write about more fundamental questions. Maybe post it. Maybe don't.