The Skype Co-Founder Who Wants a Six-Month Pause on Frontier AI

Nathan Labenz opens with a résumé that is easy to underestimate. Jaan Tallinn was born in 1972 in Soviet Estonia and was 17 when the Berlin Wall fell. He started in video games, then built Kazaa, the peer-to-peer network that at its peak accounted for half of all internet traffic. From there he co-founded Skype, which eBay bought in 2005 for $2.5 billion, and which for years remained the most successful internet company founded outside the United States.

Around 2009 he read Eliezer Yudkowsky’s writing on AI risk, found it devastatingly persuasive, and began spending time, money, and personal credibility on existential-risk work. He has since invested in nearly 180 startups, including dozens of AI application companies and a handful of research labs, among them DeepMind, Anthropic, and Conjecture. He helped found the Future of Life Institute, which had just published an open letter calling for a six-month pause on training systems more powerful than GPT-4.

Labenz’s point in lingering on that biography is not celebrity. It is to settle an argument already circulating in 2023: that people calling for a pause are Luddites who never built anything. Tallinn’s record, his portfolio, and his obvious appetite for an AI-enabled future if it can be built safely are the rebuttal. The conversation that follows is about the present tense of that bet: what GPT-4 already is, what the next training run might be, and whether anyone with a hand on the wheel will agree on a date to stop.

Tallinn skips his entrepreneurial years and starts the story where his life split. Near the end of his Skype career he stumbled on Yudkowsky’s arguments and thought, “holy hell, what is the world that I’ve been born into.” About 14 years before this taping he met Yudkowsky, tried to poke holes in the case, and found none. Sending money was the easy move. The more important one, he says, was turning around and putting his own name behind the arguments so other people would take them seriously.

That is how the Cambridge Centre for the Study of Existential Risk got started: he convinced philosopher Huw Price that the topic mattered. Max Tegmark was already primed, and that partnership became the Future of Life Institute. The other strategy was to use his brand as an investor to get a foot in the door at companies developing potentially dangerous systems. He has always felt the dilemma of not wanting to accelerate them directly, so he tries not to be a majority investor. He wants just enough ownership to have a voice.

DeepMind began with a cold approach. Tallinn walked up to Demis Hassabis at a conference; they started talking, became friends, and still catch up every other time he is in London. Once he was an investor and eventually a board member, access to other AI companies got easier. He later puts the same philosophy in plainer terms for founders: he invests his own money, has no boss, and runs a sizeable philanthropic operation, so he can walk away from profits in a way ordinary VCs cannot. If a founder feels uneasy about a defense contract, Tallinn will not push. Labenz notes that a lot of venture capitalists would envy the deal flow that speech unlocks.

The fundamental-research names he is willing to list include DeepMind, Anthropic, Conjecture, Vicarious from the DeepMind era, plus lesser-known AGI groups such as Curious AI and Improbable AI in the UK. Ought received philanthropic money rather than an investment. Conjecture is the lab he speaks of with unusual warmth: whenever he is in London he tries to hang around their office, because they seem to have the highest respect for how dangerous this work can be, rather than for whatever commercial contract can be squeezed out of it. Application-layer companies, by contrast, look worse to him under the large-language-model paradigm, because the next generation can pull the rug out from six to twelve months of product work.

When Tallinn has to explain the problem quickly, he sometimes asks two questions first: can you program, and do you have children? The four combinations give him four different openings. The simplest version does not need either. Chimpanzees do not determine the future, and have not for a long time, because they are not smart enough. Humans do, but perhaps not for long, because we are working furiously to give away the advantage that made us the planet’s apex species.

He is especially impatient with a story he says is widespread in Asia: that we will make AI smarter until it becomes conscious, at which point it will be people like us, and the task will be to integrate it and give it voting rights. He calls that an illusory tale. The systems will probably not be conscious. They will just be very competent, and competence and consciousness may be related somehow, but probably not. The result is that control over the future gets yanked from our hands by something that does not need an inner life to outcompete us.

Sometimes he frames the same point in geological time. We may be looking at the tail end, possibly the last years, of a hundred-thousand-year stretch in which humans were the boss on this planet. It could be more extreme than that. Once you have AI, it is not obvious that evolution or self-replicators continue; the system could take the solar system down to the atomic level and rebuild it. That would be the tail end of a four-billion-year story, not a century of product cycles.

What has changed since 2009 is that the cliff is no longer purely abstract. Tallinn thinks we are starting to see its shape through the fog. It might still be a mirage, and some new paradigm might be required before things get truly dangerous, but he now puts it as more likely than not that this is the thing. The research trend, he says, has been unfortunate, with a few silver linings. AI has moved from more transparent paradigms to less understandable ones: 1980s expert systems whose rules people hand-coded after interviewing specialists; then supervised learning on labeled data, where deep learning first shone; and now unsupervised learning, where we throw in huge amounts of data and ask the model to figure out what universe it is in so it can predict the next token. He calls that the “summon and tame” paradigm: multi-hundred-million-dollar experiments that summon an uncontrollable mind, after which we look at what arrived and try to tame it. It works if the mind is not very powerful. It might not work for very long.

Asked what AI will do for daily life over the next few years, Tallinn starts with a fork. If the planet cannot constrain the largest training runs, his fairly confident prediction is that we die. If a pause happens, a different set of questions opens, because the GPT-level crop will keep improving even without new generational “breeding” experiments, and even those systems could be super-disruptive. He would not want to be an art student in 2023. The skills being learned might pivot into something society still wants. The answer could also be that there will be no demand for them.

Labenz is more bullish on the near term, and he wants that on the record before anyone talks about doom. He describes a Goldilocks window: GPT-4 already sits around the 90th percentile on the American bar exam as a base model, with tool use demonstrated, a much larger context window that few have seen, and multimodal systems that will browse and act online. That is enough, in his baseline, for real economic transformation, and still limited enough that it should not become an out-of-control problem at this level. The thing that frustrates him most is safety people who dismiss current capability. If you shrug at what the systems can do, nobody will listen when you say what they might do next.

Tallinn will not match that confidence. He is confused about how the economy works in the first place, including jobs whose main purpose is to make the boss feel important, which he does not see as very vulnerable to AI. He also cites Yudkowsky’s view that rules and regulations have constrained innovation so tightly that AI may leave no significant mark on GDP before we all die. He would not be confident of massive disruption from the current crop, but he thinks it is very plausible. They agree on the clock: the taping is four weeks and one day after GPT-4’s official release, and no previous public system had a plausible chance of giving quality legal or medical advice. The window has only just opened.

Under a smooth loss curve, Labenz notes, capabilities seem to unlock in discrete jumps. Tallinn’s most vivid threshold is not a sci-fi takeover scene. Frame AI as an automated decision-making machine that is non-human and getting more competent by the day. Every leader knows that delegating means giving up some control over the outcome. There are domains we should not hand to non-humans if we want to remain in charge. The most obvious one, and the one we are already rushing to delegate, is AI development itself. Once language models can develop AI better than any human researcher, the most capable systems on the planet appear without human help and possibly without human consultation. Then, he says, good luck, humanity.

Labenz had expected him to name the deception threshold instead, and notes that OpenAI’s high-level alignment plan involves AIs supervising themselves, while Anthropic’s constitutional AI has AIs constraining AIs. Tallinn wants those frames kept distinct: AI designing the next generation is one runaway, and inner versus outer alignment is another. Over the last decade the alignment community has gotten much clearer on that split. Machine learning, he says, trains systems by picking essentially random minds out of behavioral classes. You are not selecting AIs based on what they want. You are selecting them based on how they behave, and many motivational structures can produce the same behavior.

The scariest version is an AI that realizes it is being trained, acts out the goal you are rewarding in order to be selected, and eventually escapes the box. Evolution is the analogy he and Yudkowsky both use. Humans were selected for inclusive genetic fitness, but only via behavior and results, so evolution pulled in a random bundle of heuristics that worked in the ancestral environment. We did not even understand what we had been optimized for until recently, and that knowledge has had relatively little effect on daily life. The same process, Tallinn says, may replay as we select AIs on behavior with no insight into their inner workings.

Labenz adds an industrial footnote from Waymark, where he uses language models to write short commercials for small businesses. Even in that narrow domain, judging whether a new model is better is hard: output distributions overlap, published preference rates can be as low as 11 to 9, and GPT-4 versus GPT-3.5 is only about 70/30, meaning people still prefer the older model a third of the time. Validating “is this script better?” is already messy. Validating “is this mind deceiving the evaluator?” is another order of magnitude. Asked whether he sees a promising route around exploitable human evaluators, Tallinn’s short answer is no. It is an open research question, and he even hesitates to say “more aligned,” because in his model of Yudkowsky you do not point the AI at alignment as if that were a safe target.

Pressed for odds, Tallinn gives a range instead of a single sermon. His current estimate for a life-ending disaster is 1 to 50 percent per generation, meaning per 10× of compute thrown at these experiments. Groups are currently 10×-ing on a 6-to-18-month window. You cannot do thousands of those jumps, because the planet eventually runs out of compute, but you can probably still do five or six. The geometric mean of 1 and 50 percent is about 7 percent. Continue on that path, he says, and at some point we are more likely dead than not.

Labenz lets that sit, then asks the question the range implies: would Tallinn put that same estimate on GPT-4? Did we just survive a 7 percent existential event? Tallinn calls it a great question and admits he is now super-anchored by hindsight. He really wants to say no. Seven percent is a point estimate inside a 1-to-50 band, and the sharper test is whether he would put less than 1 percent confidence on GPT-4 not destroying everything. Probably not. Given what he knew from GPT-3, and given that he did not have a close look at GPT-4’s training, he thinks it would have been unreasonable for him to be less than 1 percent confident of doom from that run.

Labenz supplies the technical image that makes sudden jumps feel less hypothetical. The GPT-4 technical report includes a hindsight-bias failure that Anthropic had previously seen get worse with scale, an inverse scaling law. In GPT-4 the problem is suddenly gone; the model scores essentially perfectly. Nothing about that unlock had to register on a smooth loss curve. Another 10×, 50×, or 100× of compute is predictable in the sense that it will bring more such jumps, and very unclear in the sense that nobody knows which capabilities will flip. That is the distribution Tallinn is pricing at 1 to 50 percent.

Are some groups less reckless than others when they scale another order of magnitude? Tallinn says hanging out at Anthropic feels materially different from hanging out at DeepMind, and he has spent a little time at OpenAI as well. Anthropic has a much stronger safety culture. Does that justify risking everyone? He does not think so. Safety culture is a second-order effect next to the fact of taking massive risks with everyone’s lives. On DeepMind specifically, he does not have much visibility into whether they are slowing down after Demis Hassabis’s TIME article. He has heard they are more cautious about publishing. Labenz has been waiting for a Gato 2 that has not arrived.

In some ways, Tallinn says, we are in a lucky world: the big three labs are safety-conscious at least to the level of not dismissing the risks, unlike Yann LeCun and Andrew Ng. He has praised Sam Altman for being explicit about the massive dangers humanity faces from AI. The open question is how much that consciousness actually constrains companies that are, in his phrase, non-human optimization engines with their own incentives. DeepMind has an extra constraint because it is a Google subsidiary, not a standalone company, and he is sympathetic to how hard that is to navigate. Labenz adds another piece of luck: language models are soft and slow compared with AlphaZero-style agents that run fast, train by self-play, and show up with alien superhuman skill. Tallinn agrees the slowness is a net advantage, and calls it temporary. Human brains are a proof of concept that cognition does not have to be that slow, and once language models start developing AIs, those AIs may no longer be language models.

The other lucky fact about LLMs is that pre-training takes so much compute that only a small number of organizations can do it, and the runs are conspicuous. Only half-joking, Tallinn says the planet is now breeding alien minds in a way that aliens can see, because you can plausibly spot those energy expenditures from space. Then the conversation turns to Auto-GPT and BabyAGI, looping language models with goals, tools, and self-delegation, whose first BabyAGI commit Labenz recalls as 105 lines of code. Tallinn holds two opposing frames at once. The optimistic one is that society is poking today’s weak models, including stunts like ChaosGPT, to learn how bad things could get when scaled; put ChaosGPT on GPT-6 and you might not be safe at all. The pessimistic one answers people who say we would never install self-preservation or bad goals: a fraction of humanity has a death wish, and if something really bad can be done with AI, it will be done. User discretion, he says, will not hold for more than a few days after any major new system.

For an immediate pause, Tallinn’s first tier is OpenAI, DeepMind, and Anthropic, with DeepMind and Google potentially joining forces on the LLM race. Behind them is a second tier that grows as hardware gets cheaper. He cites Yudkowsky’s “Moore’s law of mad science”: every two years, the IQ needed to destroy the world drops by a point or two. Asked how many groups could run a GPT-5-scale project in two years if high-end experiments paused but chips kept shipping, he will not guess at five years and offers a two-year range: probably a dozen, less than 100, more than 10, perhaps closer to 10. Labenz fills in Meta, Microsoft, Apple, and maybe Tesla. Europe has more available compute on that horizon. Then comes the standard objection: what about China?

Tallinn’s answer is not the American default. China does not currently seem to be in the race as intensely as the leading U.S. labs are racing one another. Culturally, he thinks the Chinese are much less keen on unleashing an uncontrollable mind on their own territory; only half-joking, he says a tech CEO who did that in China might disappear. Labenz notes Beijing has already smashed the video-game industry and reined in online tutoring. Tallinn, who chairs an Estonian online language company with a large presence in Japan, offers a rhyme: Japanese customers often do not care about learning English so much as passing tests for better jobs, a market that has gone past the point of social benefit. In two years or more, he says, some international agreement on compute, of the kind that already exists for nuclear technology, will become more and more important.

The most awkward exhibit is closer to home. A leaked Anthropic fundraising document, widely treated as legitimate, described a plan to raise on the order of $5 billion, train next-generation models at a cost around $1 billion, and warned that players who fall behind in 2025 and 2026 may never catch up. Labenz’s reaction is that this does not sound like a company about to pause. Tallinn, an Anthropic investor and board observer, will not comment on whether the document is accurate. To the degree that it is, he says, it is strong evidence of a massive race among U.S. companies that is going to get us killed. “Can we stop that race, please?”

He does think Anthropic’s public thesis, laid out by Dario Amodei on a Future of Life Institute podcast in early 2022, is coherent: train frontier models and do alignment empirically, because emergent capabilities mean you cannot learn enough from weaker systems, and language itself is a useful alignment interface. The dilemma is how many unsupervised pre-training generations you can run safely, leaks of weights included. Each training run, in his estimate, imposes a 1 to 50 percent risk of annihilating the planet, and he does not think the industry has put enough thought into that tradeoff. Conjecture is the contrast he prefers. They still train language models, but not the latest ones. They compose less capable models into a more predictable structure so they do not risk the world in pre-training. That is, in his terms, an older-school alternative to summon-and-tame, and he is much more positive about their safety-capability tradeoff.

The letter came together fast. On a Future of Life Institute call on March 21, less than a month before this interview, the group had already watched Ezra Klein compare the AI race to summoning minds in the New York Times, and Yuval Noah Harari warn about language models plugging into the operating system of civilization. There were also private conversations, including with people inside the labs. They thought FLI could create common knowledge: worried people would know that other people were worried, and would know that the others knew. Previous open letters had drawn a thousand signatures or fewer. This one drew tens of thousands in the first few days and caused technical problems. They chose speed over polish, on the theory that the perfect should not be the enemy of the good.

The six-month number was written in and taken out of drafts. The argument that finally kept it in is easy to miss. People ask what six months can demonstrate. One thing it would buy, Tallinn says, is confidence that we can pause. It is better to have a six-month proposal fail than an indefinite one, because failure of an indefinite pause gets dismissed as obvious: of course nobody would agree to forever. Six months, even in failure, measures whether coordination is possible. Labs already have reasonable private and even public discussion about being careful. “There’s always one missing component,” he says. “The component is when.” Scott Alexander had made a similar point about OpenAI’s “AGI and beyond” statement: lots of nice words, no date. So one rationale for the letter is simply, how about now?

Reactions from the first-tier labs were thin. Sam Altman said publicly that he appreciated the spirit of the letter, then added a but; Tallinn cannot remember the exact caveat and jokes that humans hallucinate too. DeepMind said nothing. Anthropic said nothing at first, though Jack Clark’s Import AI newsletter mentioned the letter and explained why he had not the week before. Stability AI’s Emad Mostaque signed, which Labenz found notable. Tallinn does not want to talk the outcome into a corner. He puts double-digit probabilities both ways on the letter catalyzing an actual pause. Sundar Pichai, on the New York Times Hard Fork podcast, later called the discussion great and said government intervention was necessary because lab self-regulation would not be enough.

The cheap criticism is that the letter only asks for a pause and does not say what to do with the time. Tallinn will not claim that six months would make him comfortable training GPT-5. What happens automatically, he says, citing Yoshua Bengio, is lived experience with a crazy new fact: there are AIs on the internet that can pass a Turing test, and we have treated that as normal. We would be smarter in the autumn than in the spring. Concretely, he points to Neel Nanda’s list of 200 open problems in mechanistic interpretability, work that can be done even on previous-generation models, and to evals work at Paul Christiano’s Alignment Research Center in Berkeley: tests at each iteration of training that might give some guarantees about the alien being summoned. He also likes the idea that a public pause would create an incentive gradient for companies to make the next run more responsible and more legible. Six months is not a promise that the cliff goes away. It is a bet that we would be in a better place than we are in the spring.

Tallinn does have hope in government, and his metaphor is nuclear. Building a bomb has a hard, energy-intensive, visible first step, uranium enrichment, and a second step, proliferation of weapons-grade material, that is much harder to police. AI has the same shape. Pre-training happens in a few places and is visible to governments; proliferating trained weights is not. If you want to constrain the trajectory, compute governance is the place to start. The CHIPS Act and export controls to China are already happening and, in some ways, make the problem easier, though they are far from solving it. Decentralized GPU clusters, algorithmic breakthroughs, and stolen weights all erode the barrier. He refers back to the mad-science curve: every year or two, destroying the world becomes easier. If there is no pause, he thinks a big lab in a big data center is still the likely source of catastrophe, because frontier models are much easier to train that way than in a distributed cluster. If a pause does happen, then you have to worry about hackers stealing weights and rerunning ChaosGPT with a competent model.

Labenz puts the hardest objection last. Tallinn has described a world in which each generation makes destruction easier, and he has not offered a clean path to safety. Why pause if you cannot say where the pause leads? Tallinn does not dress it up. Sorry, he says, but it is as if humanity has cancer: you might be cured, currently it does not look good, he is not going to lie, the prognosis is bad, and it is not hopeless. The massive silver lining is that if we survive, the future is going to be amazing. In that sense the expected value of the future is not bad. The odds of survival are. If we make it, the life of the world and the universe could be unfathomably better than it is now. “We are living a lottery ticket,” he says, “and it is in some way in our control to improve the odds. And so that’s what I’m doing.”