Geoffrey Hinton on Neural Networks, Intuition and the Brain

Geoffrey Hinton recalls arriving at Carnegie Mellon and finding the lab full of students on a Saturday night, excited by the future of computing. The intensity felt very different from his previous experience in England.

At Cambridge, physiology and philosophy did not answer the questions he had about how the mind works. AI research in Edinburgh was more satisfying because simulations allowed theories to be tested.

Books by Donald Hebb and John von Neumann influenced him. Hinton believed learning must involve changing connections in neural networks rather than programming every rule in advance.

He drew inspiration from simple nonlinear neurons and the question of how to adjust their connection weights. At Carnegie Mellon he collaborated with Terry Sejnowski on Boltzmann machines, work he loved even though he now doubts it describes the brain.

Hinton also learned about speech and hidden Markov models from student Peter Brown. He recalls borrowing the word “hidden” for neural-network layers whose internal variables are not directly observed.

He tells the story of Ilya Sutskever arriving at his office and asking to work in the lab. After reading a backpropagation paper, Sutskever questioned why its gradient was not passed to a more sophisticated optimizer—a question Hinton says took years to explore.

Hinton remembers Sutskever as mathematically strong, independently minded and unusually quick at implementation. In one collaboration Sutskever built a programming interface in a morning when Hinton expected it to take a month.

Sutskever argued early that larger networks and datasets would improve results. Hinton initially expected more algorithmic ideas to be necessary, but says scale proved more powerful than he had anticipated, alongside useful architectural advances.

They experimented with character-level prediction on Wikipedia using GPUs and were surprised by the results. Hinton recalls finding it hard to believe the system might have learned meaningful structure.

Hinton explains how embeddings and backpropagation can predict the next symbol. In his view, predicting an answer well can require representing the question’s meaning and performing some reasoning, although the nature of model understanding is debated.

He considers prediction a plausible ingredient in how brains learn from sights and sounds too. His claim is an interpretation of learning, not a complete account of how the brain works.

Hinton gives an example of GPT-4 drawing an analogy between feedback processes in a compost heap and a chain reaction. He thinks finding shared structure across different domains can support creative generalization.

To illustrate going beyond existing examples, he cites AlphaGo’s move 37 against Lee Sedol. He agrees that self-play helped produce novel strategies, while arguing that learning from imperfect data can also lead beyond individual labels.

He recalls a handwritten-digit experiment with deliberately corrupted labels in which the trained model generalized better than the noisy labels. The example shows that a model can exploit shared patterns despite label noise; it does not guarantee reliable learning from arbitrary bad data.

On reasoning, Hinton imagines systems using deliberation to revise their initial intuitions, analogous to a game-playing system using search results to improve its evaluation function.

He expects multimodal training on images, video, sound and interaction to improve spatial understanding and provide more learning signals. He says language alone can convey much, but other modalities can make learning easier.

Hinton considers whether language adapted to brains or brains adapted to language, suggesting both. He contrasts symbolic-rule accounts, vector-only accounts and his preferred intermediate view in which symbols are associated with interacting embeddings.

He recalls being advised by Rick Szeliski to use graphics processors for matrix operations. Early gaming GPUs sped up experiments substantially, and Hinton publicly urged researchers to use Nvidia hardware in 2009.

The pair discuss how compute evolved alongside neural-network research and what might come next. Hinton returns to the brain as a source of ideas, while not assuming machines must copy every biological detail.

A key idea for him is multiple timescales of weight change. He suggests that temporary changes in connections, or fast weights, could help account for short-lived effects such as recognizing a recently heard word.

He says conventional batch processing makes input-dependent temporary weights difficult to use efficiently. In his view, more flexible hardware and learning methods may eventually make such approaches practical.

Hinton argues that large neural networks have shown complicated behavior can be learned from data without encoding every rule in advance. He applies this lesson to longstanding debates about how much language structure must be innate.

Asked whether a long-lived AI assistant could have feelings, Hinton says he thinks machines could. He offers a behavioral account of feelings in terms of dispositions toward actions; this is his philosophical interpretation, not evidence that a particular system is conscious.

He revisits his earlier skepticism of symbolic reasoning. He now accepts that natural language provides a symbolic medium for reasoning, while arguing that learned vector representations play a major role in how it operates.

Hinton credits collaboration with students for much of his work. His stated method is to find a widely accepted idea that feels suspicious, then build a small demonstration that can expose why it might be wrong.

He gives dropout as an example: silencing neurons during training seems as though it should hurt performance, yet can improve generalization. A simple simulation can reveal the effect and prompt a deeper explanation.

His longstanding open question is how the brain obtains useful learning gradients and whether it approximates backpropagation. He does not claim the mechanism has been established.

Looking back, he says he may have been wrong to see Boltzmann machines as a brain model, but does not regret the time spent on them. He values their elegant learning algorithm and the collaboration that produced it.

Curiosity about how brains learn drove his research more than a detailed plan for social applications. More recently, he has become concerned about potential harms as well as benefits of the resulting technology.

He sees possible benefits in healthcare, materials and understanding biology, while worrying about surveillance, manipulation and weapons. These are broad scenarios rather than established outcomes of a particular system.

Hinton says slowing the field could delay benefits but doubts a unilateral pause is likely amid international competition. He expects AI assistants to make research more efficient by helping with programming and reasoning.

On selecting researchers, he recalls recognizing strong questions from Ilya Sutskever and David MacKay. He values both intuition and technical strength, and says a lab benefits from people with different combinations of skills.

He advises questioning claims instead of accepting everything, while acknowledging that strong prior intuitions can also lead to serious mistakes. His own earlier commitment to Boltzmann machines is one example.

He sees scaling multimodal models as a promising research direction but welcomes other ideas too. Whether many different learning algorithms can reach similar capabilities remains an open question for him.

Asked what makes him proudest, Hinton chooses the elegant learning algorithm for Boltzmann machines developed with Sejnowski, even if it proves impractical or does not describe the brain. He ends the interview with a joke about choosing something to watch on Netflix.