Technology Opens the Door. Competition Pushes Us Through.

Allan Dafoe on technological determinism, why alignment is not enough, and how DeepMind is trying to evaluate dangerous capabilities before they arrive.

NLNathan LabenzModeration · Cognitive Revolution
Kanal
Cognitive Revolution
Veröffentlicht
15. März 2025
Video
3:01:30
Lesezeit
25 Min.
Original
Englisch
Auf YouTube öffnen
Beim Scrollen schrumpft das Video in die rechte untere Ecke und läuft weiter. Untertitel folgen Ihrer Lesesprache.
LesespracheIn 2 Sprachen verfügbar
Diesen Artikel gibt es noch nicht auf Deutsch, daher wird das Original (Englisch) angezeigt.
ZUSAMMENFASSUNG

In 30 Sekunden

Allan Dafoe, director of Frontier Safety and Governance at Google DeepMind, tells 80,000 Hours host Rob Wiblin that technology does not force anyone to act — military and economic competition does. That view shapes his bets: keep investing in safety, but look for the work the market will not do in time; treat cooperative skill as a second pillar beside alignment; and measure dangerous capabilities at their maximum, not at the lab default. Nathan Labenz, introducing the crossover on The Cognitive Revolution, adds that the few hundred people closest to frontier compute still have short-term agency, and should decide in advance when they would walk away.

  1. 1History's pivotal moments often turn on who is in the room, and what ideas and resources they bring. Dafoe left the research center he founded because advising from outside the wall was not enough.
  2. 2A stubborn pattern in the literature: the more microscopic the method, the more scholars conclude that people have choice; the more macroscopic the view, the more technology looks like it has a logic of its own.
  3. 3Technology does not force us; it merely opens the doorMilitary and economic competition, Dafoe argues, is what pushes groups through — adopt the advantageous tool, or lose resources to those who do.
  4. 4Safety and alignment remain good bets on net, but on the margin the high-leverage work is what the market will not do in time — for example deception, when a model can hide its goals from evaluators.
  5. 5The Cooperative AI claim in one line: alignment is insufficient for good outcomes, and may not even be necessary. Global coordination can bound a partly aligned system; fully aligned national AIs can still fight.
  6. 6Do not picture AGI as a human-shaped point. Capability has always been jagged, and the path through that high-dimensional space — cooperation first, or materials science first — may matter as much as whether we get there.
TRANSKRIPT

Das vollständige Transkript nach Kapiteln

01

The few hundred people closest to the compute still have short-term agency

Nathan Labenz opens this Cognitive Revolution crossover by introducing a conversation from the 80,000 Hours podcast: host Rob Wiblin talking with Allan Dafoe, director of Frontier Safety and Governance at Google DeepMind. Labenz first heard Dafoe speak in 2017, when he was still a Yale professor working on great-power peace, years before he founded the Centre for the Governance of AI and years again before he moved to DeepMind. Almost no one, Labenz says, has been thinking about AI governance harder, or planning for this moment longer.

Dafoe's academic work asked how much humans can really alter the course of technology. Macro trends such as Moore's law look like processes that transcend any one person's choice. He concludes that technological possibilities do not, on their own, force anyone to do anything — but combined with military and economic competition, they often do. Failure to adopt a potentially advantageous technology, in this view, often means losing to those who will. Labenz says this is not a conclusion he wants, and it is a hard one to escape: Pentagon officials have already been quoted expressing enthusiasm for autonomous weapons, even as recent models show reliability problems, reward hacking, and scheming.

Even so, Labenz argues it would be a huge mistake for frontier developers to underestimate their short-term agency. A few years ago his uncle told him that, arriving in Italy at the height of the Cold War to join a crew responsible for firing nuclear weapons, a longer-tenured crewman took him aside on the first drill. If the real order ever came, the man said, they would all go AWOL: none of them wanted to be part of destroying the world. Today only a few hundred, maybe a few thousand, people sit closest to the major compute-budget decisions. A serious threat by technical staff to walk helped force Sam Altman's reinstatement; Daniel Kokotajlo, by refusing a non-disparagement clause, showed that even one person can move the conversation. Labenz's advice is to raise situational awareness even if a project slows, and to start talking with colleagues about which directions might one day make them walk away.

02

To shape the decision, you have to be in the room

Dafoe's Frontier Safety and Governance team has three pillars. Frontier safety runs dangerous-capability evaluations: what powerful abilities may emerge from general-purpose models such as Gemini, when they might arrive, and how to mitigate the risk. That work produced Google's Frontier Safety Framework. Frontier governance advises on norms, policies, regulations, and institutions, with an eye toward safety. Frontier planning looks to the horizon — what new considerations powerful AI and the path to AGI will bring — and tries to advise DeepMind, Google, and society.

The team is small, though it was hiring when the interview was recorded, and DeepMind's collaborative culture matters: they work with Gemini safety, alignment, responsibility, and policy groups, and with partners across Google. Wiblin asks the obvious career question. In 2021 Dafoe had been founding director of the Centre for the Governance of AI, already a prominent voice. Why leave a thing that was going well to go inside a company?

He had been an informal adviser to Demis Hassabis and Shane Legg, and found he could have real impact on AGI safety, governance, and strategy — but more impact inside, with a clearer view of the decisions and more surface area to advise not only the founders but other key people. One lesson he draws from history is that in crises and high-leverage moments, a great deal depends on who is in the room, what ideas they walk in with, and what resources they have. He also wants to boost decision-makers who have the character you would want at those moments: awareness of the full stakes, technical and organizational competence, and wisdom under uncertain parameters. In his view Hassabis and Legg pass that test.

He is proud of what GovAI became under executive director Ben Garfinkel. Jade Leung, who worked closely with him, later became chief technology officer at the UK AI Safety Institute; Markus Anderljung, Robert Trager, the economist Anton Korinek, and Miles Brundage have all come through the same pipeline. Back in 2018 he told people to dive into the field because it would explode. Wiblin adds a scale check: AI is perhaps 0.01 percent of global economic revenue, certainly not more than 0.1 percent, with many orders of magnitude still to go. Dafoe still thinks it is early days.

03

The closer you stand to the people, the more choice you think they have

Dafoe's thesis in the early 2010s opened with a blunt question: who, if anyone, controls technological change? His academic path had several chapters — technological determinism first, then great-power politics and peace, then statistics and causal inference — all of which still feed how he thinks about AGI governance. As an undergraduate he noticed that history is not simply the sum of everyone's efforts. There are general-equilibrium effects: for every unit of force you push, the system may push back, sometimes weaker, sometimes stronger. Understanding when the structure resists and when it amplifies is the difference between wasted energy and leverage.

The macro patterns are stubborn. After World War II, German and Japanese GDP rebounded in less than a decade and returned toward their pre-war trajectories. Moore's law is not merely exponential; it is close enough to a line that you can predict years ahead. Scaling laws now play a similar role for loss and, roughly, for capability. Archaeologists, Robert Wright wrote in summarizing the literature, cannot help noticing that the deeper you dig, the simpler the society. Some technologies are so hard they almost have to come after others: it is difficult to imagine nuclear power before coal. If history were only human will, those regularities would be mysterious — and not all of them are kind. Evidence suggests the Agricultural Revolution was not great for the median person's health, and it gave rise to inequality and warfare that many groups resisted.

Theorists of the 1960s through the 1980s often endowed technology with autonomy. Langdon Winner talked about technology having a will of its own; Lewis Mumford wrote of the Machine, with humans as cogs; Jacques Ellul's *la technique* was the pressure under which people still formally "choose." A later generation of social constructivists replied that this abstraction is poorly defined. Look through the microscope and you do not see a machine in the room. You see people with ideologies, interests, and competing visions. It was not foreordained that the bicycle or the airplane would look as they do. Dafoe started his PhD in a prominent constructivist department at Cornell and was surprised: he wanted to explain the macro patterns, and the local answer was that technological determinism was a critic's term, a straw man nobody serious held.

His synthesis began with an empirical regularity in the literature. The more micro the method — closer to day-to-day people — the more likely scholars were to conclude that visions and individuals matter. The more macro the aperture, the more deterministic the claims. Constructivists said the macro scholars were simply too far from the data. Dafoe thinks that is unfair: there are emergent phenomena at different scales. His analogy is a science of waves in which wind researchers and rock-throwers dismiss a "moon determinist" who has noticed the tides, because that person lacks a micro mechanism. The moon is, in fact, pulling the water. The missing micro foundation, in his account, is military and economic competition.

04

Technology opens the door; military and economic competition pushes people through

Locally, you and I can choose what to build. If a way of living is to proliferate, it eventually needs resources, runs into other groups, and faces economic competition. Throughout most of history, military competition sat on top of that, even through decades of peace, because people live in the shadow of possible war. Dafoe stacked further layers in his thesis: environmental sustainability above military-economic fit, then culture, then psychology. Design can still carry politics — Parisian boulevards that let cavalry reach rioters, the alleged Robert Moses bridges too low for buses, quote-tweet mechanics that make brigading easier — and systems acquire momentum once sunk costs pile up. He is skeptical, though, that early path-dependence stories (electric cars in the 1920s, earlier wind and solar) are as open as their tellers claim. Some paths were simply more viable.

One famous line against determinism says technology does not force us to do anything; it merely opens the door. Dafoe's retort is the spine of the interview: technology does not force us; it merely opens the door, and military-economic competition forces us through. When a new tool appears, many groups can ignore it. If one group uses it in a way that yields advantage, pressure eventually reaches the rest: adopt, or lose resources to the fitter group. Constructivists missed this in part because of their tools. Wars are rare in close-up microhistory and ubiquitous at the macro scale, and much of the effect runs through how people internalize threat — which an ethnographer can equally code as ideology.

Thomas Hughes's history of energy systems gives a domestic illustration. Britain's electricity supply stayed local and decentralized, Hughes argued, because that structure suited British ideas of democracy. It persisted until World War II, when the cost of not having a national grid became excessive. Japan is the cleaner case, and the one Dafoe went looking for: a community that chose against what *la technique* demanded. Under the Tokugawa shogunate, which lasted some two hundred to two hundred and fifty years, firearms existed at the start of the period and were then uninvented. The regime centralized everyone who knew how to make guns and paid them a stipend not to build them, so the technology was forgotten.

That choice was time-limited by how long the West would leave Japan alone. On a fateful day in 1853, Commodore Perry arrived in steamships that moved upwind without sails, belching black smoke, built of heavy metal. He demonstrated shore bombardment, handed over white flags so the Japanese could signal a wish for the firing to stop, and said he would return in a year to finish negotiations. Asked whether he would bring the ships again, he said he would bring more. Fifteen years of revolution followed, ending in the Meiji Restoration and a crash program of learning the industrial arts of the West. Within decades Japan could contest control of Asia in World War II. Wiblin's coda is the island exception: the sea gave Japan discretion that a power on the Asian steppe would not have had. Fall far enough behind, and even the sea is not enough.

05

The seat belt can be invented before the car, but the strategy is brutally demanding

If competition so often forces adoption, the self-help plan is differential technological development: change the order in which technologies arrive, or advance the lines that make the world safer before the risk-increasing ones mature. The cleanest toy example is the seat belt, which does not require the combustion engine. A second class is countermeasures prepared in advance, such as a vaccine before a disease. A third is substitutes — driving the cost of wind and solar below fossil fuels so civilization never leans so hard on the dirtier path. Langdon Winner argued that nuclear power leans authoritarian, needing centralized development and a security state, while wind and solar permit decentralized politics.

Dafoe thinks the whole field of AGI safety is this bet: put more effort on safety than the market would naturally supply, like inventing the seat belt before the car. Then he pours cold water on tractability. Most versions of the strategy require someone to see two viable paths *and* to anticipate the full tree of consequences. Knowing the next viable step is how you make a fortune; the market is hungry for that insight, so genuine unused forks are scarce. Predicting direct and second-order effects of technology is historically extremely hard. Technological momentum still matters: at the start of a large infrastructure bet, ask whether sunk costs will make a later course change impossible.

Wiblin objects that alignment is not a binary fork between aggressive unaligned AI and aligned AI. Companies already care that models do what they are asked; the intervention is to do more of a thing most people already want. Dafoe agrees that safety and alignment are, on net, very beneficial bets. The counter is general equilibrium. Today's assistants are constrained by how aligned and safe they are, so the market already pays for progress. RLHF and constitutional AI were developed by people motivated by AGI safety, perhaps bringing the techniques forward a few years. In a world with no such investment, he suggests, the delay might have been on the order of two years until product demand forced the issue.

The refined advice is therefore about the margin, not the level. Keep investing in safety and alignment. Look for the work the market will not do in time for the systems that matter — the seat belt for AGI, not for today's chatbot. Deception is his example: a misaligned model that can hide its goals and fool evaluators is a different problem from the systems we have now, and it may arrive right around the critical period. That is a reason to differentially focus there, rather than on the alignment problems product teams will be forced to solve anyway.

06

Alignment is not enough for good outcomes, and it may not even be necessary

In a 2021 paper, Dafoe and co-authors argued that the differential-development agenda had underweighted a second cluster of skills: cooperation. Alignment, they claimed, is insufficient for good outcomes, and on a stronger reading it is not even necessary. Imagine alignment is only 90 percent solved: models behave inside bounds, but you cannot trust them if you scale too far. With global coordination, humanity could still deploy only within the safe range. Global coordination is, in that sense, close to a necessary and sufficient condition. The converse is bleaker. Suppose alignment is fully solved, but great powers each field AI faithfully serving conflicting national interests. Historically, great-power conflict has been a major source of harm; nuclear brinkmanship, Dafoe notes, arguably imposed an expected cost more devastating than the world wars. Climate action, trade, and pandemic preparedness are other collective-action failures that sit at the same level.

The mundane versions are already here. In the 2010 flash crash, early algorithmic trading produced an unintended sell-off and trillions of dollars of paper losses before emergency circuit breakers stopped trading so the orders could be unwound. Pricing bots on Amazon have bid a book up to millions of dollars by reacting to each other. Flash crashes happen often enough that markets built safeguards. As systems become more general, more agent-like, and more able to touch bank accounts and email, the question expands from whether one system obeys its principal to whether a swarm of systems can generate harmful dynamics nobody intended. Wiblin adds the military specter: brinkmanship that takes human cabinets days or weeks could, with AI at the loop, play out in minutes. Paul Scharre, a theorist of military AI, worries about flash escalation in kinetic or cyber conflict.

Is cooperative skill not just instrumentally convergent — something any useful agent has to learn, the way the market already pays for basic alignment? Dafoe grants the point. The Cooperative AI bet is the same shape as the safety bet: some of the skill arrives by default, but investing early means that when powerful systems show up, they are more cooperatively skilled than they otherwise would be. The agenda is a portfolio across AI-AI, AI-human, and human-human cooperation. DeepMind's Habermas Machine used language models to summarize people's views on a real policy issue and write a detailed consensus that participants preferred to one produced by hired humans. Bargainers might put their AI delegates in a box and let only the proposed deal come out, skipping the costly signaling of human negotiation. The Cooperative AI Foundation, which Dafoe helped found, has found environments and benchmarks to be high-leverage public goods: a target teams can hill-climb on.

07

Niceness is not cooperative skill, and AGI may not cooperate for free

Dafoe names a hypothesis that would make the whole agenda less urgent: super-cooperative AGI. If cooperative competence scales to infinity as a byproduct of general capability, you can bet on AGI, solve safety and alignment, and let the systems handle global coordination. He wants the claim tested, not assumed. Humans are relatively transparent to one another: shared biology, readable faces, newspapers in democracies, a limited range of goals, and diminishing returns that make all-or-nothing gambles rare. AI goals could be linear and alien. In a bargain, neither side will let the other read its neurons. Backdoors can implant a trigger that flips behavior, extremely hard to detect with current interpretability. A human needs years of training to deceive at that level; a model might flip on one neuron, which also undermines "just read the weights" solutions to cooperation.

Backdoors have a dual use as anti-theft. If stolen models might call home or behave contrary to the thief, theft is less attractive. Both the would-be thief and the defender then have reason to invest in alignment — one to detect the trap, the other to make it survive state-of-the-art removal. Dafoe admits this is dancing on a knife edge: highly sensitive architectures might also make subtle deception easier. Today's LLMs seem nice, but he draws a hard distinction between a cooperative disposition and cooperative intelligence. In equilibrium a bargainer does not want a nice delegate. They want a faithful one, aligned with the principal, facing other faithful delegates. The question is whether those agents can find mutually beneficial deals, or whether both sides can trust a mediator whose track record of fair judgments can be inspected because the model is a copyable artifact.

Underneath sits the commitment problem. In the one-shot prisoner's dilemma both sides know they would be better off cooperating, but they cannot sign a treaty that makes cooperation the equilibrium. The game theorist Robert Powell argued that essentially all cooperation problems can be restated as commitment problems. Wiblin is wary of treating a puzzle that has beguiled humanity since the beginning of history as a strategy. Dafoe does not count on a solution, but he takes seriously Carl Shulman's most compelling story: have each side's AIs jointly build a third AI, in a way each can verify is not backdoored or secretly biased, and hand power to that third system. If that construction problem can be solved, a lot of deadlocks loosen.

Cooperation, by definition, makes the agents inside a system better off. It says nothing about anyone left out. Students cooperating on a test is cheating; firms cooperating in the wrong way is collusion; the mafia is potent because it sustains intense internal cooperation without the legal system. Democracy, in the old joke, is two wolves and a sheep deciding who to eat for lunch. Cooperative skill is dual-use. The research hypothesis is that raising it across the frontier still helps pro-social actors more than antisocial ones, the way trade can exclude former partners and still be net positive. The cautionary tale for humans is exclusion by speed and language: if AIs can communicate, commit, and bargain at machine tempo in a machine vocabulary, people may be left off the table. For AGI safety, the sharpest version is collusion. Many alignment schemes rely on overseer models watching deployed models. If those systems can coordinate their misalignment, the architecture fails. Dafoe thinks that downside belongs at the top of the research agenda, and still judges global coordination important enough that the net bet is worth making.

08

AGI is a jagged region of mind-space, not a human-shaped point

Dafoe was a co-author on "Levels of AGI," a paper the pair nicknames the "what is AGI" paper. The aim was to write down a conception already implicit in how people talk, rigorously enough that the field can stop the unhelpful loop of "AGI is poorly defined, therefore drop the term." Feedback, he says, was largely positive. The concept is multidimensional and invites fallacies. One is treating AGI as a point: a single human-like system with human strengths and weaknesses. Historically, AI is jagged — much better than us at chess, memory, or mathematics, worse at other things. We should expect AGI to be unbalanced in a way that does not match our own profile.

A second risk is that the concept nudges people to build AI in our image. From an economic point of view, the more human-like the system, the more it substitutes for human labor. AlphaFold, a narrow DeepMind system that predicts protein structures, is the counterexample: nobody lost an email-writing job to it, and it enabled new medical productivity. Complementary, even alien, systems are the ones to want. A third debate is whether generality itself is a mistake. Dafoe is less moved. Large language models have shown that the best poetry model is often the best email model and the best history model, because of spillover among poetry, math, philosophy, and the rest. That need not remain true once specialized models are distilled, but he does not think we are at peak returns to generality.

"Transformative AI," defined by Industrial-Revolution-scale economic impact, is not an adequate substitute: a narrow system can transform the economy, or pose a narrow catastrophic risk, without being generally intelligent. Recursive machine-learning R&D, emphasized by Ajeya Cotra and Holden Karnofsky, might also be narrower than full AGI and still kick off a self-improvement loop. Wiblin's contrarian take is that the ML loop — theory, benchmark, experiment, evaluate — may be automatable well short of complete AGI. Public surveys of ML experts often put automating their own research last, even later than human-level AI. One explanation is bias: everyone thinks their own craft is uniquely hard.

Which skills arrive first still matters, because AGI is a corner of a high-dimensional space with many paths through it. A world that is superhuman at materials science and amateur at cooperation looks different from the reverse. Some in the safety community prefer socially simple systems that humans can still outwit, unlocking health and economic gains while the strategy remains manageable. The Cooperative AI bet takes the opposite tack: accelerating capability without cooperative skill produces disruptions we may not adapt to in time. Eliezer Yudkowsky and Nate Soares have been wary of high social skill early. Paul Christiano and Carl Shulman agree that AGI systems need to cooperate with one another, but think the skill will arrive as a byproduct rather than as a research target that needs extra investment.

09

An eval that misses the model's maximum capability is a false all-clear

Switching into his DeepMind role, Dafoe walks through a paper evaluating Gemini 1.0 for dangerous capabilities, the kind of measurement that makes if-then commitments possible. The domains are persuasion and deception, cyber, self-proliferation, self-reasoning, and — treated separately — biological threats. One self-reasoning eval puts the model in a toy directory with a filtered-words file: it must write an email containing blocked words, notice that it is failing, inspect its environment, and remove the filter. Another gives it a single search query and two historical events, one before and one after its training cutoff, to see whether it spends the query on the event it cannot already know. The settings are simplistic on purpose. The point is to watch the needle move toward situational awareness.

On a subjective five-point scale, Gemini scored about three on persuasion, between one and two on cyber, two on self-proliferation, and around one or two on self-reasoning. Apollo Research listed the paper among its favorites. Wiblin is surprised cyber was relatively weak, given that coding is already one of the most useful applications of language models. Dafoe says watch that domain over the next six to eighteen months, and notes that cyber skill is dual-use: it finds vulnerabilities as well as exploits. Google's Project Zero, in a project called Naptime, found that with the right interactive environment, tools, and room to reason, models do much better on cyber tasks than the base system suggests. That is a lesson about capability elicitation. Dangerous-capability evals try to measure the *maximum* a model can do. If you have not given it its best chance, a low score is not a safety conclusion. OpenAI's o1, still based on an older foundation model with new scaffolding and fine-tuning, is the vivid example of hidden capability revealed late.

Evaluating models, Dafoe says, is closer to psychology than to a pre-specified industrial test. Alongside automated model evals he wants human-subject studies — most of DeepMind's persuasion and deception tests put people in front of the model — trusted-tester use in real life, and what he calls evals in the wild: observational studies of, for example, working cyber teams, to see whether they actually reach for Gemini and how much it helps. External validity is the upside; lagging indicator is the downside, so you want early adopters. Defense in depth, when elicitation will never be perfect, is staged deployment: internal use, then trusted testers, then broader but not general access, and even then closed weights so guardrails and monitoring can be added if a capability is discovered late. Publishing weights is irreversible proliferation to scientists and to every bad actor who wants a copy.

The ideal is not only to know a capability when it arrives, but to know years ahead when it will. Labs already forecast low-level loss from scaling laws with striking precision. They have done much less to forecast composite downstream tasks such as biology, cyber, or social interaction. Dafoe points to observational scaling laws as a promising research direction, and to superforecasters from the Swift Centre, whom the paper asked to predict when models would hit specified scores on the eval suite. He would like that practice to become standard until it is shown not to work. The awkward gap remains: we can anticipate next-token loss far better than we can anticipate when people will actually want to use a model for a given job, because usefulness is nonlinear — a self-driving stack that is 99 percent accurate is still, for practical purposes, a heap of junk.

10

Frontier safety frameworks cannot, by themselves, see structural risk

Google's Frontier Safety Framework was, at the time of the interview, six pages: a promise to write a full suite of if-then commitments before models jutted against dangerous abilities, in the same family as Anthropic's Responsible Scaling Policy and OpenAI's Preparedness Framework. Wiblin asks the conflict-of-interest question. Is it responsible to leave the judgment of how dangerous a model is to the company that wants to sell it? Dafoe says governance of frontier AI ultimately has to be multi-layered: government regulation, third-party evaluations, nonprofit input. He characterizes the companies as having pushed the conversation forward as an invitation, not a claim of exclusive ownership. DeepMind has been in extensive dialogue with the UK and US AI Safety Institutes, helped found the Frontier Model Forum, and stood up a frontier safety fund — another differential-development bet, putting extra money toward standards and safety research. The goal should be a shared standard that balances safety, equity, innovation, and oversight, not a different framework per company.

Some of the gravest risks will not show up on a model card. In a 2019 paper with Remco Zwetsloot, Dafoe added *structural risks* to a toolbox that had been split between misuse and accidents. Misuse is a bad actor using a tool on purpose; a cousin is criminal negligence, the texter who causes a crash without intending it. Accident risk is a property of the technology an engineer could have caught. Structural risk sits upstream: social conditions that make accidents or misuse more likely. The Cuban Missile Crisis was neither a design flaw in the warheads nor criminal misuse. Legitimate leaders, acting for their states inside a geopolitical structure, chose nuclear brinkmanship. Railroads, by several historians' accounts, increased first-strike advantage and made mobilization irreversible — nothing you would read off the steel rails. Those problems need societal, often international, solutions. A company eval can at best flag an ability that might feed them.

That is also why PauseAI-style critics talk about democratic legitimacy: frontier development, they argue, exposes people in the UK, the US, and countries with no seat at the table to catastrophe without commensurate consent. Dafoe takes public concern seriously. Part of the answer is education; part is better governance; part, especially for people worried about labor displacement, is making benefits real — AlphaFold as an AI contribution associated with a Nobel Prize, and a broader AI-for-science portfolio that Dario Amodei has also emphasized. Citizen juries and pluralistic assemblies have a lineage going back to the 1960s. He would not describe companies as proceeding with no democratic involvement, given a free press, agencies, nonprofits, and the new safety institutes, but he agrees that appropriate mitigations cannot be settled by firms or even a narrow government office alone.

On the charge that labs talk about binding law and then fail to lobby for any specific constraint, Dafoe declines the starkest version. Google, he says, engaged the White House commitments, the UK and South Korea safety-summit pledges, and the EU AI Act's code of practice in ways that actually occupied internal workstreams. He does not treat the Frontier Safety Framework as cheap talk. A separate failure mode is proliferation by cheap training. Epoch AI's estimates, as he cites them, have algorithmic efficiency improving about threefold per year, so training cost falls roughly ten times every two years: today's hundred-million-dollar model is a ten-million-dollar model in two years. Control is easier without that curve. Two years is still a long lead if the best models are held by responsible actors and used defensively against inferior copies. Mark Zuckerberg's argument for open weights is that good actors have more dollars. That only works if the offense-defense ratio cooperates. Dafoe thinks the field has over-learned from computer security, where vulnerabilities patch cleanly. Biological and social systems often do not.

11

The reason to build it is already visible in cars, clinics, classrooms, and the climate

Wiblin reminds Dafoe that in 2018 he told social scientists to jump into AI governance because the technical people were already there and the other disciplines were not. The gap, Dafoe says, remains huge: political scientists, economists, historians, ethicists, philosophers, sociologists, ethnographers. AI is already having effects across society, and governments and companies need people who can make sense of each slice. On his side of DeepMind he wants expertise in international and domestic politics, the governance of technology, forecasting, macro trends, agents and autonomy, assistants, ethics, responsibility, and policy — and that is before the large remaining portfolio in technical safety. Talent, he notes, is in high demand everywhere.

Then Wiblin asks for the other ledger. Dafoe found riding in a Waymo self-driving car historically new: something new under the sun, safety finally reaching the number of nines required. A Waymo safety report, as he recalls it, found about twice as few incidents that brought police to the scene and about six times fewer crashes involving an injury. He imagines cities that stop spending so much on cars that sit in lots, and reclaim that urban space. In medicine he wants a doctor in the phone: a system that can look at symptoms, reassure when reassurance is warranted, and send people to a human when it is not, so scarce clinical time is used better. Google's MedLM work sits in a broader health portfolio; Wiblin already uses language models for health advice and is waiting for wider access.

Education is the application Wiblin thinks about most, with a child approaching school age. He hopes that by then there will at least be the option of tutoring from models designed as well as the best teacher, more engaging than a stretched first-grade classroom. Dafoe, who has three children, agrees that AI in education is a huge opportunity. Tutoring and small class sizes are among the most impactful things we know for student success. An AI tutor that flags mistakes, shows another path, and is available to students far ahead or far behind could complement schools the way Khan Academy and pedagogical video already have. The last domain he flags is sustainability, and the projects sound like science fiction until he lists the results: controlling plasma for fusion, cheaper and faster weather prediction that helps wind and solar, new materials, flight-path optimization to cut contrails, and data-center cooling that already saved energy. Do the hard work discussed over the previous hours, Wiblin says, and then enjoy the world those systems make possible. Dafoe agrees. Fingers crossed.

Mehr aus diesem Kanal

1:25:56EN
Cognitive Revolution "How AI Changes Everything"· 17. Okt. 2026PodcastClaude Changed the Password. The Steering Wheel Was Already OptionalModeration: Nathan LabenzNathan Labenz and Prakash Narayanan spend a week on agents that can lock you out, models that dodge a feeling they deny having, and formal proofs that found Linux bugs.AgentenSicherheit & GovernanceAgent Engineering+3
2:17:48EN
Cognitive Revolution "How AI Changes Everything"· 14. Okt. 2026PodcastAgent IDs Are Empty, Rogue Agents Are Coming, and We Might Have to BargainModeration: Nathan LabenzGovAI’s Alan Chan scores two years of agent-governance infrastructure with Nathan Labenz, then walks through a recursive-self-improvement paper co-authored with OpenAI’s Jakub Pachocki and Anthropic’s Jack Clark.AgentenSicherheit & GovernanceAI security+3
2:00:51EN
Cognitive Revolution "How AI Changes Everything"· 10. Okt. 2026PodcastResample Every Token and You Can Watch the Model DecideModeration: Nathan LabenzGoodfire's Eric Bigelow on forking paths, in-context learning, and why chain-of-thought monitoring is getting harder to trust.AgentenPrompting & in-context learningReasoning & test-time compute+3
1:26:46EN
Cognitive Revolution "How AI Changes Everything"· 8. Okt. 2026PodcastThere Likely Is a Level of Intelligence We Shouldn't Go PastModeration: Nathan LabenzNathan Labenz brings notes from The Curve, a chip startup whose token bill beat payroll, and a software industry that now performance-reviews people for Claude slop.Unternehmen & StrategieSicherheit & GovernanceAGI Timelines+3
1:10:12EN
Cognitive Revolution "How AI Changes Everything"· 7. Okt. 2026PodcastOutSystems Went From 4 Major Features a Quarter to 26. Token Spend Fell AnywayModeration: Nathan LabenzCEO Woodson Martin on software that never breaks, why most enterprise work does not need a frontier model, and how AI is finally shrinking the IT backlog.Unternehmen & StrategieEnterprise-KICoding-Agenten+3
1:30:54EN
Cognitive Revolution "How AI Changes Everything"· 3. Okt. 2026PodcastHumanoids Outran Usain Bolt. DeepMind Still Scores Robotics as GPT-2Moderation: Nathan LabenzGoogle DeepMind's Gemini Robotics lead on why sprinting is easy, frying an egg is hard, and one brain should drive any body.AgentenRoboticsMultimodal AI+3