Nick Bostrom on Superintelligence: Why the Upside Deserves as Much Attention as the Risk

Bostrom, a philosopher at Oxford and director of the Future of Humanity Institute, defines intelligence in the common-sense way: the ability to solve complex problems, learn from experience, plan and reason. He thinks something could be fairly intelligent without being conscious. Superintelligence would be a system with much more general cognitive capacity than humans, a faster learner and better reasoner whose plans are more effective across a wide range of challenging environments.

Asked whether such a system must exist in the physical world, he says digital-only is fine. Computers and memory are physical anyway, and even with a limited set of actuators, such as typing text on a screen, its ability to affect the world could be very strong.

Before discussing attack vectors, Bostrom stresses that he sees huge positive potential in machine intelligence. He wrote his book Superintelligence in 2014 because he felt there was neglect of what happens if AI succeeds. Since then the risks have gotten wide recognition and research groups now work on alignment, so he thinks the upside is now relatively neglected in public discourse.

Bostrom separates two contexts: near-term issues such as algorithmic discrimination, self-driving cars and drones, and the long-term questions he addresses in his book. They are blurry and overlap in the medium term, but mixing them produces confusion, with overhyping of the near term and underhyping of the long term. Keeping them separate, he says, gives two good conversations instead of one bad one.

On the long-term upside, he says there is no single killer app. The world has many problems, from health to the economic system to investment, trade and foreign policy, and a friendly, aligned superintelligence would help with nearly any of them. He calls AGI the ultimate general-purpose technology that applies wherever human creativity and problem-solving are useful.

The main thing it would give us is far more control over nature. It would not automatically fix fundamentally political problems, since some require coordination that technical capability does not supply. Bostrom adds that it is consistent to hold that the ultimate potential is enormous while also thinking some near-term technology gets overhyped.

Asked about intelligence explosion, Bostrom says we do not know how explosive it will be, but it seems fairly likely to him that at some point there will be a period when AI progress becomes extremely rapid, roughly around human-equivalent cognition. He admits the concept of human-equivalent breaks down on close inspection, and that what counts as an explosion could be overnight or a few years. Compared with the average person, he leans somewhat more toward an explosion scenario.

On whether we can proceed past human level, he says it seems very unlikely that human cognitive capacity is a ceiling, though some uncertainty always remains because we have never seen it. When Lex raises the fear of losing control and our place as the smartest thing on Earth, Bostrom replies that many people are working on projecting higher levels of problem-solving ability while keeping systems aligned with human values.

He also downplays the loss of specialness: if the universe is big, other civilizations probably already have superintelligences. He compares a superintelligent infrastructure to the scientific community as a whole, a mind-like thing far more capacious than any individual that we already accept as part of the fabric of existence. In a posthuman world, he imagines such intelligence working more as a protective background rather than an intrusive presence.

Lex asks whether the collective intelligence of Google, Twitter and Facebook recommender systems might already be superintelligent. Bostrom answers that the vagueness of the concepts dominates, since Google search is clearly superhuman at retrieving text and images from a key but cannot do a vast set of other things at all. Current systems tend to have one spike of superhuman ability and near-zero capability elsewhere, like a simple computer multiplying huge numbers.

Lex adds his intuition that AlphaZero is much more intelligent than Deep Blue, because self-play learning crosses into a different kind of intelligence. Bostrom agrees that learning ability is an important facet of general intelligence: taking a new domain you were not programmed for, figuring it out and getting good at it. AlphaZero does this far more than Deep Blue did, and systems with more general-purpose learning may trigger a stronger intuition that they are getting smart.

Asked to pick a utopia, Bostrom says it is a deep and difficult question to which he does not have all the answers. If AGI arrived, material and resource constraints would shrink dramatically, giving a big pie, and a much larger design space of possible modes of being, including what we could do to ourselves. From this he draws two conclusions.

First, we would need a fundamental rethink of what we ultimately value, going back to first principles on the meaning of life, happiness and fulfillment, since the world would be too different from the familiar to extrapolate. Second, we should aim to do well by more than one value system, instead of scoring very high on one criterion such as hedonism, with wire-headed brains full of pleasure, and zero on beauty or achievement.

With the expansion of options, he thinks we could possibly get something like 98 percent of the best by several criteria simultaneously. Lex likens this to multiple parties balancing each other, but Bostrom says that is not exactly what he means: the first instinct should be to try to do very well by all value systems before deciding whom to shortchange. There may still be hard trade-offs in the end, he says, but we should start with generosity and inclusiveness.