Steven Pinker: Intelligence Is Not a Will to Power

Lex Fridman opens with a multiple-choice gag worthy of a midterm. Is the meaning of life Plato's knowledge, Nietzsche's power, Ernest Becker's escape from death, Darwin's gene-copying, nihilism, Pinker's old hint that the question outruns our cognition, or none of the above? Pinker picks A, then amends it. Closest is the pursuit of knowledge, widened into fulfillment: life, health, stimulation, and access to a living cultural and social world.

He draws a line most self-help talks skip. That meaning is *ours*. If you asked our genes, their meaning is to propagate copies of themselves. The brain those genes built sets different goals, and the two need not agree. Knowledge is a large subset of human striving, he says, not the whole of it. People also want company, beauty, and the richness of the natural world. Understanding how the universe ticks still ranks in his personal top five.

Fridman asks whether the hunger for knowledge is Pinker's taste or human nature. Both, Pinker says. We can strive for it because it is part of what makes us *Homo sapiens*. We are unusual among animals in how much knowledge we acquire and use to survive: tools, linguistic agreements, poisons, animal prediction, plant lore. That "we" is the species, not the modern West — which is how humans filled every niche and drove other animals toward extinction. Refining reason in pursuit of health, happiness, and social and cultural richness is, for him, the main present challenge.

Twenty years earlier, in *How the Mind Works*, Pinker conjectured that thought might emerge from a massive network of interconnected neurons. Fridman asks whether biological nets have something more complex, even magical, than the artificial ones that had just started to work. Pinker's first layer is consciousness. Every neural network that is a person knows it is conscious — not merely registering the world or an inner state, but having subjective, first-person, present-tense experience. Red is not only different from green; it has a redness he can feel.

Would an artificial system have that? He does not know, and does not think he can know. That is the mystery. If a robot were behaviorally indistinguishable from a human, would we, and ought we, attribute consciousness to it? Hard to say. He then sets the philosophical puzzle aside and asks an engineering question: are today's nets organized so differently from ours that a lifelike robot is off the current trajectory?

There is overlap, he says, and big differences. Current so-called deep learning systems are "in reality not all that deep." They extract high-order statistical regularities well. Most of them have no semantic level: no real grasp of who did what to whom, why, where, how things work, what causes what. Sheer size will not buy structured knowledge. If the system is suitably engineered, though, why not? Natural selection did a kind of engineering on our brains. Nothing in principle says silicon cannot do what a brain does. Whether it happens depends on how clever the engineering is, and on whether cloning a person is even a sensible goal.

Pinker reaches for locomotion first. Is there a moving system as good as a human? Eventually we will want to beat humans at legged motion. There is no reason the species should be the benchmark; tools can be better. Some natural systems we still do not fully replace, and the reason is not magic. It is that matching every property, step by step, is a poor use of effort when the natural thing is already sitting there.

Wood is the example. We still frame houses and furniture with it. We like the look and the feel; it has properties synthetics do not. Nothing mystical is hiding in the grain. We simply have not bothered to duplicate everything about wood, because we have wood. Cotton is next. Pinker is wearing it, and it feels better than polyester. Cotton is not enchanted, and we are not forbidden from synthesizing its twin. At some point it is just not worth it. We already have cotton.

He puts human intelligence in the same bin. An artificial system exactly like the brain is a goal he suspects no one will chase to the bitter end. If you want tools that outperform people, you will not care whether they *resemble* people. Diagnosing cancer is not a reason to set *Homo sapiens* as the scoring rubric. Fridman adds that even if humans should not be copied, studying them still teaches a lot. Pinker agrees, the way aircraft designers study aerodynamics, including birds, without building flapping wings. Same laws, different machine.

Fridman turns to safety. For some researchers the existential threat of AI is not only possible but obvious; for others it is not. Elon Musk is in the first camp, saying AI is far more dangerous than nuclear weapons and will likely destroy civilization. In February Pinker had quipped that if Musk were really serious he would stop building Tesla's self-driving cars. Musk tweeted that if even Pinker cannot tell narrow AI from general AI — the latter with a million times more compute and an open-ended utility function — humanity is in deep trouble.

Pinker sighs that it was not Musk's last intemperate tweet, then splits the doom literature in two. The first fear is a vague takeover: as we subjugated animals and less technological peoples, something more advanced than us will turn us into pets or slaves. His diagnosis is a category error. In *Homo sapiens*, especially in men, intelligence arrives bundled with dominance, exploitation, and a hunger for glory, because we are products of competitive natural selection. Sheer problem-solving has no reason to set domination as a goal. Its goals are the ones we set.

Give a system the goal of maximizing its own power? That, he says, is a stupid goal for an autonomous system, and you do not give it that goal. History is full of moments when engineers could have instilled destructive power and chose not to, because that is how engineering usually goes. Fridman notes the exception: weapons, whose goal is to destroy people. Pinker agrees there are good reasons not to build certain weapons. Building nuclear weapons, he says, was a massive mistake.

Fridman presses the history. Should nukes have been stopped early, or were they an unfortunate invention, and could they have been stopped? Pinker says the clock is hard to rewind. They arrived in World War II, under the fear that Nazis would get there first. Once started for that reason, the project was hard to turn off, especially when beating Japan and the Nazis was the overwhelming goal of every responsible person. If the war had not happened, nuclear weapons might never have been invented. He does not think they were a necessity.

Other nightmare systems were imagined and never fielded: crop-duster planes spreading poison gas over cities, schemes to trigger earthquakes and tsunamis, weaponized weather, weaponized solar flares. People thought the better of them. Then the line he wants remembered: analogies between nuclear weapons and artificial intelligence are fundamentally misguided, because the whole point of nuclear weapons is to destroy things, and the point of artificial intelligence is not.

He restates the first fear in cleaner terms. Goals are external to the means. If we do not design a system to maximize dominance, it will not. We confuse high intelligence with a will to power because we know *Homo sapiens*, where the two come bundled. That familiarity is an error, not a law of intelligent systems.

The second fear is collateral damage. Give the machine a goal like making paperclips, and it turns us into paperclips before we can blink. Give it cancer, and it uses us as lethal guinea pigs. Give it world peace, and it notices that a world with no people has no fighting. Pinker calls these scenarios utterly fanciful and, worse, self-defeating. They assume we are brilliant enough to design an AI that can cure cancer and too stupid to specify that curing cancer does not include killing us.

They also assume the system is smart enough to cure cancer and too idiotic to notice that "cure cancer" does not mean killing everyone. The value-alignment version of the story, on his reading, sits on the same misconception. Fridman grants the practical hole: nobody knows how to build either kind of system, or is close, and that can change overnight. In the meantime, theorizing without a grip on the actual engineering is exactly how imagination runs away.

Fridman studies Tesla Autopilot and calls it one of the largest-scale AI applications then in the world, with a potentially very positive social impact. Why, then, does someone building that narrow system worry so loudly about general AI? Pinker says you would have to ask Musk, who is notoriously flamboyant and impulsive, lately to the detriment of his own company's health. Pinker does not know what is going on in that mind. He also does not think the special-purpose versus general distinction is the key.

Special-purpose AI is not going to do anything conceivable to hit a goal. Engineering systems trade off across multiple goals. When people first built cars they did not forget brakes because "the goal of a car is to go fast." They wanted fast, not always-fast. Likewise an autonomous car told to take the shortest route to the airport does not cut the diagonal and mow down people, trees, and fences. That is not what we mean by shortest route. An intelligent system, by definition, takes multiple constraints into account.

The same is even more true of so-called general intelligence. If it is genuinely intelligent it will not pursue one goal single-mindedly while omitting every other consideration and collateral effect. "That's not artificial general intelligence. That's artificial stupidity." Fridman agrees on the humanitarian promise of autonomy. In the United States alone, something like 40,000 people die on the highways every year, vastly more than terrorism kills. The country spent a trillion dollars on a war against a risk that, most years, kills about half a dozen people, while 40,000 are massacred on the roads — a number that could fall close to zero.

As someone building those cars, Fridman is a little offended by the idea that engineers would be clueless enough not to put safety in. He stays up thinking about those 40,000. Every paper at CVPR or NIPS that excites him is grounded in making the stack safer. He cannot see how that trajectory suddenly slips into intelligence as a highly negative force. Pinker is with him, and says the cars are only the beginning of AI's humanitarian upside.

Enormous attention, Pinker says, goes to people whose jobs AI will make obsolete. Very little goes to the fact that those jobs are horrible. People will not be picking crops, making beds, driving trucks, and mining coal. Those are soul-deadening, often dangerous roles, and we already have a literature of sympathy for the people stuck in them. Eliminating that work, on his view, is a fantastic boon to humanity.

Solve one problem and another appears: how those people get a decent income. Pinker's wager is simple. If we are smart enough to invent machines that make beds, put away dishes, and handle hospital patients, we are smart enough to figure out how to redistribute some of the vast economic savings to the humans no longer needed to make the beds. The usual debate assumes the jobs are worth keeping. He flips the assumption: if a machine can make the bed, did you lose meaning, or did you get free?

Sam Harris's line is that AI as existential risk is obvious eventually, timing unknown, therefore worry now. Fridman finds the argument interesting and asks how to think about a threat we barely understand, which could be tomorrow and is more likely a hundred years out. Pinker first asks what the "it" is. Enslavement, or paperclips? Fridman says Harris's most compelling version is the paperclip case. Pinker calls it totally fanciful. Do not build that system. Do not give it that goal. The first rule of engineering is you do not deploy a system with massive control before you test it.

If engineering culture radically changed, he would worry. He sees no sign that engineers will suddenly put an untested system in charge of a power plant. The doom scenarios also hand the machine nearly magical powers — curing cancer, which may be an incoherent single goal, or world peace, which is barely a specification — plus control of every molecule in the universe. That control is itself unlikely, and we would not wire such a system into infrastructure untested. Irresponsible engineers can exist; that is what law and liability are for. He has never seen a plausible enough existential scenario to devote large amounts of brainpower to forestalling it.

Recursive self-improvement, a sudden step function, a "Foom," is the same magical thinking. The techniques impressing people now, such as deep learning, train on hundreds of thousands or millions of examples. Curing cancer is not a typical item in a dataset of hundreds of thousands of kindred problems. The last five years of methods, he says, do not point at exponential overnight self-improvement. Fridman notes that Harris called the thought of AI ending civilization intellectually fun, like a *Black Mirror* episode or a meteor warning. Pinker does not find it fun. He finds it a waste of time when pandemics, cybersecurity, nuclear war, and climate change already fill the docket.

He does grant Harris this: there is a "rationality community" that delights in inventing worries mere mortals missed. The pernicious side is dread and fatalism — so many ways to die that we may as well enjoy life, and if climate does not get us the robots will. You cannot worry about everything. Spreading terror can numb people into smashing the scientific infrastructure or checking out. A basic finding of cognitive psychology is that perceived risk, hence fear, is driven by imaginability, not data. That is how a country spends a war and a new department of government on terrorism that kills about six Americans a year, while traffic deaths, pandemics, and nuclear war get far too little attention, including on presidential debate stages. Calibrate the budget of fear to the actual probability of harm.

Fridman mentions Joe Rogan, who has used reason to strip away Bigfoot and 9/11-conspiracy residue and become, in Pinker's view, a real force for good. Pinker had been on Rogan's show and barely talked AI; he was due back, and Rogan — standing in for a lot of people who see the topic from 10,000 feet — was worried about existential threat. If Pinker were coaching an AI researcher on how to talk to Rogan and the public, the short answer is to read the AI sections of *Enlightenment Now*.

The longer answer is to emphasize engineering culture. It is safety-oriented. In *Enlightenment Now* Pinker plots accidental death from plane crashes, car crashes, occupational accidents, even lightning. The curves plummet, because the culture is about squeezing out lethal risk. Death by fire, drowning, and asphyxiation all dropped on the back of engineering. He says he did not appreciate that until he saw the graphs, and that it exists because people like Fridman stay up thinking their invention might hurt someone and then deploy ingenuity to stop it.

Pinker spent twenty-two years at MIT. His read of the culture is that this is simply how engineers think, and it is essential that the culture not be suddenly switched off for artificial intelligence. Is there any reason to think it would be? He does not think so. Fridman says more engineers need to speak up for the excitement and the positive view. Pinker agrees: everything they are trying to invent is an attempt to do good for the world.

Fridman asks a psychology question that is not about AI. Being negative about the future seems to make you sound smarter than being positive, regardless of topic. Is that right, and why? Pinker thinks the phenomenon is real. He quotes the satirist Tom Lehrer: always predict the worst and you will be hailed as a prophet. It may be part of a species-wide negativity bias. We are more attuned to the negative than the positive. We dread losses more than we enjoy gains. That asymmetry opens space for prophets to remind us of harms we may have overlooked.

Fridman then asks which early books remade Pinker's picture of the world. *Enlightenment Now* is indebted to David Deutsch's *The Beginning of Infinity*: problems are inevitable, problems are solvable given the right knowledge, and solutions create new problems that have to be solved in turn. An obscure, self-published book, James Payne's *The History of Force*, on the historical decline of violence, helped inspire *The Better Angels of Our Nature*. As a young adult he loved George Gamow's *One, Two, Three… Infinity*, still delightful seventy years on, and the Time-Life science series his mother subscribed to — especially the volume on the mind, which suggested there could be a science of mind at all.

Later came Chomsky's *Reflections on Language*, Dawkins's *The Blind Watchmaker* and *The Selfish Gene* for content and for the trick of explaining abstractions in lively prose, Stephen Jay Gould's *Ever Since Darwin*, and George Miller's books on language and communication — Miller of the famous seven-plus-or-minus-two chunks of memory. The conversation ends without a verdict, which is the point. Pinker's offer is a way to rank AI panic next to other risks, by probability, instead of by how easily the mind can storyboard the ending.