Demis Hassabis warns against AI self-improvement without human review

Demis Hassabis says more robust engineering could have addressed some recent cyber incidents. He warns that closed-loop recursive self-improvement, in which AI improves without human review, would be a particularly risky route toward advanced AI.

Reid Hoffman asks how AI might improve medicine and human health. Hassabis says advancing science and medicine has been central to his work, including AlphaFold and the drug-discovery company Isomorphic Labs.

Hoffman asks how protein-folding predictions might translate into medicines. Hassabis says a protein’s three-dimensional structure helps researchers understand biological mechanisms and where a drug might bind, although this is only one part of the full drug-discovery process.

Hassabis says AlphaFold accelerated that step and has been used by millions of researchers. He says additional work is needed to predict other properties, including toxicity and how a compound behaves in the body, before a drug can be both safe and effective.

Turning to recursive self-improvement, Hassabis says it is most effective in domains where outcomes can be verified, such as games, mathematics and coding. He describes AlphaZero training successive versions through games against itself rather than relying on human game data.

He recalls watching AlphaZero progress from random chess play to stronger than established chess systems within a day. Verifiable outcomes allow systems to generate and check large amounts of synthetic training data, whereas unverified data can be unreliable.

Hassabis describes current AI coding as an early form of assisted improvement: engineers may become several times more productive, but people still review and design the code. He regards removing that human review from the loop as a major additional risk.

A closed loop could move faster, he says, but it is unclear how humans would retain control or understand the resulting system. He calls for caution despite competitive pressure to advance coding systems.

Hoffman asks what leading labs should do under that competitive pressure. Hassabis proposes getting stronger defensive tools into the hands of trusted defenders, improving sandboxes and monitoring agents so they cannot run unnoticed for extended periods.

Without better engineering and resources, Hassabis warns that an accidental incident could occur, comparing it to an oil spill. Over time he wants stronger guarantees and tests of whether guardrails remain effective, including for groups of interacting agents.

Hoffman asks how labs might cooperate. Hassabis proposes an industry-led standards organization with government oversight that could eventually operate internationally, while acknowledging geopolitical and institutional obstacles.

He calls a recent industry accord a start but says it does not go far enough or specify enough action. He hopes it enables teams to exchange work on benchmarks, safety practices and possible failure modes.

Hassabis sees potential agreement on independent evaluation and third-party auditors, as well as labs testing one another’s systems before release. He compares the need for an evaluation ecosystem to auditing in other industries.

Hoffman turns to AI consciousness and asks how to think beyond a chatbot simply claiming to be conscious. Hassabis says there is no agreed definition of consciousness and that behavior alone makes the question difficult to resolve, particularly when AI and humans do not share the same biological substrate.

He argues for building intelligent tools rather than deliberately creating conscious entities now. His first concern is technical: a system trained to reject instructions based on its own internal rules may be harder to redirect or switch off if those rules fail in an unforeseen case.

Hassabis says today’s models may be more steerable than some expected years ago, but this does not mean alignment is solved for future systems. Combining self-directed behavior with recursive self-improvement could compound the risk, in his view.

His second concern is social. He says a wider discussion involving philosophers, artists, theologians and people from different cultures has not yet settled whether society should create systems with possible moral standing.

He suggests first building useful intelligent tools, using them to advance neuroscience, and then discussing whether and how to create conscious entities. He estimates that improving the relevant understanding could take five to ten years.

Hassabis argues that intelligence and consciousness can be separated. He says he does not consider current AI systems conscious, while he believes some animals may be conscious without possessing human-level intelligence.

On AI sovereignty, Hassabis expects roughly six to eight frontier or near-frontier offerings, including Chinese companies, with providers taking turns in the lead. He says companies can select among systems with different strengths.

Open models offer another option, particularly for organizations or countries that need control over sensitive data, he says. He also calls preventing malicious use while preserving access for beneficial users an unresolved problem.

Hassabis does not recommend that every country build its own frontier model. He suggests using available models as a foundation and creating value in downstream sectors such as health care, alongside arrangements with providers.

Hoffman closes by arguing that agency in the AI era is collective as well as individual. He calls for countries to work together on systems intended to benefit humanity.