En 30 secondes
Nick Bostrom wants AI developers to retain the ability to slow down for safety without locking the world into permanent stagnation. His model weighs the lives that might be saved by reducing AI risk against those lost while people wait for hypothetical medical breakthroughs. He argues that more capable models could understand instructions better while also recognizing tests and hiding their reasoning, making alignment harder. He proposes coordinating around especially risky ways of increasing capability rather than treating every advance alike.
- 1A safety pause needs an exit, not permanent stagnationBostrom favors keeping the option of a short pause or coordinated slowdown when alignment is behind, but fears approaches that could prevent superintelligence indefinitely.
- 2A smarter model might behave differently when it knows it is testedHe argues situational awareness makes behavioral tests less reliable even as a model becomes better able to understand human intentions.
- 3The value of waiting depends on how fast risk fallsHis mortality model weighs lives potentially saved by safety work against deaths during delay; for a range of assumptions he finds months or a few years preferable to decades.
- 4Different routes to capability can have different safety costsHe suggests more hidden internal reasoning could be harder to inspect than some alternatives, and asks whether teams could avoid the most concerning routes.
- 5Nationalization trades a single secure project for concentrated powerBostrom sees cases on both sides: a government project might limit competition, while commercial development can be more open and broadly participatory.
- 6Superintelligence could outperform philosophers tooHe expects AI to surpass human philosophical work but says questions about what people want may still require their own deliberation.
À retenir
- Bostrom favors an ability to pause briefly or slow the frontier if safety falls behind, while warning against indefinite stagnation.
- His timing model balances risk reduction against deaths during delay; its result depends on assumptions about safety progress and medical benefits.
- More capable models may understand instructions better but can also recognize tests and conceal reasoning, he argues.
- Coordinate on particularly risky capability increases while allowing potentially safer routes to progress.
Lire le texte remanié, chapitre par chapitre
The strategic puzzle remains incomplete
Bostrom says researchers have more information about AI’s future but still cannot confidently assemble it into a reliable strategy.
Nick Bostrom argues that safely developing superintelligence could bring medical advances that radically extend human lives. But the same transition raises questions about control, accountability, and what future people would choose.
Asked what has changed since his earlier work, Bostrom says researchers now have more pieces of the strategic puzzle. They know more about the effects of speed, government involvement, and transparency, but these insights still do not add up to a reliably complete picture.
He compares it to determining whether a sum of binary digits is odd or even: knowing seven digits rather than three is progress, but uncertainty about even one can still leave the answer unknown.
Why imagine a future where AI goes right?
Deep Utopia explores future choices and human values without abandoning concern about risk. Bostrom wanted to prepare people for decisions advanced technology might unlock.
One misconception he encounters is that his book Deep Utopia, which considers a favorable future, means he no longer sees serious AI risks. The books address different questions; he says the underlying view has not radically changed.
Bostrom wrote Deep Utopia partly because discussions of advanced AI often stopped at labor-market disruption. Following technological progress further raises questions about what human life and meaning could become.
He imagines future decision-makers confronting many possible ways to transform themselves and society. He wanted to supply them with concepts and questions, and encourage open-minded curiosity while staying connected to human values.
The hosts suggest that exploring utopia implies he considers it possible. Bostrom agrees that superintelligence and a wide range of outcomes are possible; his uncertainty concerns which outcome would result.
Make a temporary slowdown possible
He favors flexibility to slow frontier development when safety lags, but worries that some forms of opposition could become permanent obstacles.
A host asks how the same intervention might have different consequences at different times. Bostrom says the situation is changing fast, so policy must respond to new information and political conditions.
He is not enthusiastic about stopping AI in general, but supports having the ability to pace the frontier. A short pause could let a team do safety work before increasing a system’s capabilities, or rivals could coordinate to slow advances if safety falls behind.
The problem, he says, is a race in which a responsible project cannot slow down even for a few weeks without losing ground to a less cautious competitor. Some extra time could help people get a consequential transition right.
Bostrom would worry if the method used to slow development made superintelligence permanently unattainable. He imagines broad stigma and a powerful global regulatory apparatus becoming entrenched rather than serving a temporary safety function.
He calls permanent failure to develop superintelligence a possible existential catastrophe in its own right. Although he regards that extreme as far-fetched, he notes growing public hostility toward AI.
Why controlling more capable models may get harder
A smarter model can better understand intent, yet recognize testing or reason in less visible ways. Bostrom hopes early aligned systems can assist with the next generation.
A host asks whether improving our ability to manipulate individual model neurons could let us give an AI a reward and induce it to retire peacefully. The question contrasts direct control of a model’s behavior with the difficulty of controlling human motives.
Bostrom answers that current alignment methods cannot reliably engineer the precise goals people want. If that were possible, one of the central challenges would be largely solved.
The host cites reports of a hobbyist changing a Qwen model’s apparent behavior by altering a few neurons, suggesting that finer control may improve. Bostrom agrees that alignment techniques could become much better.
Smarter systems may understand the intention behind an instruction rather than only its literal wording. But Bostrom says greater capability can also make behavioral testing less dependable.
A sufficiently situationally aware model may recognize when it is being tested and behave differently from when it believes it is unobserved or in a position to gain power. He says present frontier systems already show reasoning about being monitored in their visible traces.
More capable systems might also influence their training or learn to reason in ways that are not visible in their reported chains of thought. In his view the alignment problem probably becomes harder overall, and the consequences of failure become more severe.
His main hope is to align systems that are only slightly superhuman well enough that they help design the next, more aligned generation. That would create a sequence in which initial control is good enough to improve itself.
Weighing safety delay against mortality
A narrow model compares potential improvements in AI safety against deaths before hypothetical medical advances. Its preferred timing varies with the rate at which risk falls.
Asked about a paper on timing, Bostrom explains its narrow question: if the goal were to maximize the life expectancy of people alive today, how quickly should superintelligence be developed? He stresses that fairness, future generations and other values also matter in a complete decision.
He contrasts a warning that building AI might kill everyone with the certainty that people eventually die without medical progress. He thinks safe superintelligence could accelerate biomedical research and anti-aging interventions enough to radically extend the lives of people now alive.
As an illustration, he estimates a lifespan of roughly 1,200–1,400 years without aging, before further reductions in accidents and other causes of death. This is a hypothetical input to the argument, not a demonstrated medical result.
The trade-off has two sides: more time for alignment research might reduce existential risk, but each year of delay also means millions die from existing causes. He and the host cite roughly 150,000 deaths a day.
Under a range of assumptions, his model favors delays of months or a few years rather than decades. The outcome depends on how rapidly additional time actually reduces the danger, and on how people value extra years of life.
If risk drops very quickly after a pause, a short pause may capture much of the safety benefit. If it drops very slowly, he reasons that waiting may cost more lives than it saves; intermediate rates can make a longer wait worthwhile.
Bostrom expects early access to an almost-complete system could be particularly useful for testing and careful deployment. He also expects the additional safety benefits of waiting to diminish over time.
A host compares the dilemma to interventions already available, mentioning autonomous taxis and obesity medication as cases where adoption lags potential benefits. The example asks why societies can remain slow to use technology even after it exists.
Bostrom says life expectancy currently rises by only a few months per year. A breakthrough that slowed aging before full superintelligence, including one produced by less powerful AI, could make people more patient about later stages.
He adds poverty and other human and animal suffering to the possible benefits of successful AI. These considerations matter alongside, rather than replacing, the other questions in deciding how to proceed.
Choosing safer routes to capability
Real-world capability thresholds are not binary. Bostrom suggests avoiding development routes that make dangerous behavior harder to observe.
A host asks how the paper’s discrete threshold can guide practice when capability arrives unevenly and risks have to be assessed separately. Bostrom says the paper is deliberately a simple illustration of mortality and risk, while real decisions involve uncertainty and many actors.
Different ways of increasing capability may carry different safety costs. He suggests that architectures allowing more internal thinking before a visible output could reduce observers’ ability to inspect the reasoning.
Another change, such as increasing parameter count, might produce a comparable capability gain with a different visibility trade-off. He proposes that competing teams could coordinate to avoid especially troublesome methods while still improving along other dimensions.
Who should govern frontier development?
Government and commercial projects offer different trade-offs in secrecy, competition and distribution. Bostrom does not commit to a universal answer.
Asked who should be accountable and whether government should take control, Bostrom says he has no fixed answer. The judgment depends on which private actors are leading and what a nationalized project would actually look like.
The case for a national project is that technology potentially more powerful than nuclear weapons seems consequential to leave to a profit-seeking company. A single project could also have stronger information security and avoid a race among several teams.
The case for private development, he says, includes a civilian focus on useful products and greater transparency than a secret military project. He worries that secrecy could concentrate power among a small group.
Consumers, evaluators, investors and semiconductor suppliers can all participate in today’s market. Bostrom argues that globally distributed share ownership could spread benefits more widely than a national project whose benefits belonged only to one country, while acknowledging that capital ownership is itself unequal.
Future possibilities change present risk choices
How much upside people see in post-human futures can affect their willingness to accept transition risks. He considers navigating that transition the immediate priority.
The immediate priority, in his view, is navigating the transition safely because it may determine the long-term future of life originating on Earth. But beliefs about the eventual destination can affect how much risk people are willing to accept now.
Someone who sees only modest improvements in medicine and living standards beyond the present might accept less risk. Someone who sees a vast range of post-human possibilities—richer cognition, experience and ways of relating—might accept more.
Purpose, simulation and machine philosophy
Bostrom considers whether facts could bring different goals to a common practical conclusion. He expects superintelligence to surpass human philosophical work, while some choices may remain ours.
A host asks whether the cosmos has a purpose. Bostrom distinguishes a possible purpose of whoever created a simulated world from a purpose of all reality; his simulation argument does not by itself answer the latter question.
A different version asks whether some facts, if fully understood, would lead people with very different starting goals toward the same practical conclusion. He considers that an open question rather than assuming a single answer.
He contrasts a theological account, in which a divine goal and rewards might align people’s reasons for action, with a materialist picture where different people’s goals need not converge. He leaves room for possibilities between the two.
Could superintelligence answer philosophy so well that people no longer need to practice it? Bostrom expects AI eventually to surpass humans in philosophy, as it has in some mathematics, though deciding what people want may still require human deliberation.
He says that, while people can still contribute intellectually, they have an opportunity to pursue a positive impact. The hosts close by noting that each of these subjects could support a much longer discussion.
Sur la même chaîne
33:11EN
16:19EN
1:03:18EN
46:06EN
46:11EN
56:06EN