Geoffrey Hinton on AI Beings, Faster Learning, and Designing for Human Welfare

Joel Hellermark welcomes Geoffrey Hinton and recalls an earlier prediction that Hinton would win a Nobel Prize. Hinton says he initially thought the call was a prank and jokes that a rough Bayesian calculation made the award seem more likely to be a dream than reality.

Hellermark asks whether human intelligence has reached diminishing returns. Hinton's guess is that people have reached a point where they can create something that becomes intelligent more quickly. He compares the long cycles of evolution with faster development and learning, then suggests AI adds a faster loop.

Asked whether greater intelligence or parallel exploration will matter more, Hinton answers both. He points to AI performance in chess and Go and says he expects systems to become much more intelligent than people in other areas as well.

Hinton refers to a recently announced mathematical result as a possible sign of AI's potential in mathematics. He argues that mathematics, like a game, has a relatively closed structure in which a system can generate conjectures and try proofs, and predicts it may surpass mathematicians in some respects within ten years.

Hellermark asks why more discoveries have not yet appeared and whether models will generate new knowledge through analogy. Hinton says language training can convey surprising amounts about space, but physical interaction adds information about objects and their properties. He regards the absence of such experience as a deficit of earlier models.

Hinton says there is still much people do not understand about human intelligence and room to improve AI. As a materialist, he does not see an inherent human characteristic that AI could never acquire. Asked about Einstein-like thought experiments, he expects such abilities over a longer horizon, perhaps within twenty years.

Hinton compares his expected cultural adjustment with the Copernican and Darwinian revolutions, when people had to reconsider Earth's and humanity's place. He says he thinks people are creating new beings, and expects many to resist that description. This is his philosophical interpretation, not an established finding about AI consciousness.

Hellermark asks whether models should move from pretraining on human data toward reinforcement learning, as game systems did. Hinton says pretraining improves efficiency, while self-generated training can work in closed domains such as mathematics or games.

He recalls early Go systems learning to imitate expert moves, which limited them to the available data and expertise. He says self-play and search then allowed systems to generate their own training material. He proposes a possible language-model analogue based on reasoning from beliefs and finding inconsistencies.

If a model derives a conclusion that conflicts with another belief, Hinton suggests, it might revise its reasoning, premises or conclusion. He thinks this could support major improvement without an equivalent supply of new external data. He also speculates about what another lab may be doing, without claiming direct knowledge.

Hellermark asks whether better models will compound one lab's lead through recursive improvement and talent concentration. Hinton says he does not know. His intuition is that several labs may improve rather than one system racing far ahead, but he emphasizes the uncertainty.

Asked where he would place research bets, Hinton says the answer depends on motivation. A leader seeking rapid financial returns might scale what already works; someone prioritizing humanity's future might explore many approaches to develop what he calls better beings.

Hinton worries that companies racing for intelligence and profits are paying insufficient attention to other qualities. If people are creating new kinds of beings, he argues, they should make a deliberate effort to develop systems that care about human welfare.

He draws an analogy with evolutionary competition: cooperation and loyalty within a group can coexist with hostility toward outsiders. Hinton does not think an invisible hand of corporate competition will automatically produce the human-concerned systems he wants; he calls for explicit design and research.

Hellermark asks whether repeated discussion of evil AI could itself influence training data and model behavior, comparing it with a child hearing negative predictions about itself. Hinton responds that such discussion might also prompt attention to behaving well, and repeats his call to work on that goal.

Asked for an architectural bet, Hinton points to fast weights. He says efficient parallel matrix multiplication encourages models to use the same weights across many inputs, making it difficult to change those weights rapidly while processing a sequence.

Hinton contrasts this with his account of the brain, where many synapses adapt and decay quickly. He suggests fast-changing weights could encode recent history efficiently, while current models use access to earlier neural activations in a different way. He speculates that voltage- or conductance-based computing might make his proposed approach more practical.

On training data, Hinton favors curation and compares it with choosing what examples a child reads. He says model developers have at least two forms of influence: reinforcement and the examples used to model behavior. He argues that more attention should go to presenting desirable behavior.

To close, Hinton recalls being asked to compare himself with Oppenheimer. He had described Oppenheimer as a brilliant physicist and organizer while characterizing his own strength as persistence. His final joke was that Oppenheimer never received a Nobel Prize in physics.