The Machine Should Never Treat Its Objective as Gospel

In 1975, still in high school, Stuart Russell rode the bus to Imperial College every Wednesday with a box of punch cards. The machine gave the students eight seconds of CPU time. Five seconds went to reading the cards and compiling the code, which left three seconds — enough for one move of a chess program searching eight plies with alpha-beta, plus some home-grown move ordering and pruning. Then the move printed out, and they went to the back of the queue to feed the cards in again.

The program never beat him. He was a reasonable chess player in his youth. He later wrote Othello and backgammon programs, and at Berkeley he spent years on what he calls metareasoning: reasoning about reasoning. The game tree is larger than the number of atoms in the universe. Programs and people succeed the same way, by looking at only a small fraction of it. Look at the right fraction and you play well. Waste time on moves that will never happen, and you lose because you never compute the right decision.

The rule he extracted is almost austere. A machine should think about whatever thoughts will improve its decision quality. If you are already convinced a movie is terrible, there is no point spending more time proving it, because the judgment will not change. Thinking exists to change your mind about what to do in the real world. Faced with a move that is a guaranteed draw and another that looks worse on average but still might be a win, the uncertain one is worth thinking about. Value of thought is the product of how good a line looks and how much uncertainty remains. More uncertainty means more upside.

The same selection rule, Russell says, is visible in AlphaGo and AlphaZero. Early in the tree everything is shrouded in uncertainty, so the search is bushy and tries many directions. A truly terrible move reveals itself quickly — half the pieces, half the territory — and is dropped. Further down, the tree becomes long and narrow, following lines 40, 50, 60 moves ahead. Humans cannot do that, mainly because short-term memory will not hold a 50-move sequence or imagine the board at the end of it.

Two kinds of learning are at work. One is evaluation: looking at a board and instantly judging how promising it is, apparently at superhuman level. The other is looking ahead. Russell's claim is that the first is the more remarkable. Turn search off entirely — a depth-one search that only considers each legal move and scores the resulting position — and that version of AlphaGo can still play at a professional level. Human professionals sit for five or ten minutes. The network, in less than a second, intuits the right move. Humans do not have that kind of intuition about Go. They have to think.

He is less romantic about grandmaster pattern recognition than the mythology allows. The idea that a master reliably, instantly recognizes the right move and the value of a position is, he thinks, a little overrated. Bobby Fischer games in which a queen is sacrificed look like bad moves. What arrives first is a sense that something in the position might yield a win, even if the move looks weird, because it opens lines of calculation that would otherwise be dead. Then you follow the forcing variations. Even in grandmaster games, people miss simple three-, four-, or five-move combinations that were sitting there, just not particularly apparent.

The Othello program, after metareasoning improvements that included learning, produced a more visceral feeling. Russell could sit down the next day and feel that it had got a lot smarter. It was more aggressive, unforgiving of any minor mistake, and seemed to understand the game better than he did. Garry Kasparov, during the match against Deep Blue, said he suddenly felt there was a new kind of intelligence across the board. In the context of chess, Russell finds that feeling exciting, not scary. It is why he went into AI. Once he understood what a computer was, he wanted to make it smart.

His first machine was a Sinclair Programmable Calculator, with room for a 21-step algorithm. He implemented Newton's method for square roots and a few other arithmetic tricks, then thought: if I just had more space, I could make this thing intelligent. The chess program itself is not the thing to fear. Chess programs are not in the taking-over-the-world business. Extrapolate, though, and the dissimilarities with the real world start to matter. We know the rules of chess. The board is completely visible. Most of the real world is not visible from wherever you are sitting, and overcoming that takes qualitatively different algorithms.

Timescale is the other gap. People regularly plan on horizons that reduce to billions or trillions of motor steps. Choosing a Ph.D. at Berkeley is a five-year commitment, amounting to about a trillion control actions: going up stairs, opening doors, drinking water, every finger movement while typing every character of the thesis. You do not commit to those primitive actions in advance, but you still reason on a scale that will eventually bottom out there. Deep Blue and AlphaGo are not a threat to humanity. They are a step toward one. Progress in AI, as Russell describes it, is the successive removal of the assumptions that make problems easy. Drop complete observability and you need machinery that tracks what it cannot see, with uncertainty that will not go away. Each such step magnifies by about a thousand the range of things AI systems can do.

Fridman remembered the old lay of the land: we will never solve Go. Russell would have said it was unlikely that chess-style algorithms would simply scale. The working theory was that Go would have to be handled the way humans handle it, by breaking the board into weakly connected subgames, analyzing each region, coupling the analyses, and revising. Five, six, seven, ten parts of the board. That structure resembles the real world much more than chess does. Work, home, sport, shopping are connected but weakly. While typing a paper you do not simultaneously decide the order of milk and butter. You do need to finish before the shops close, because there is no food at home.

The hope was that solving Go would force progress on things useful for the real world. In a way, AlphaGo was disappointing. The program was not that different from Deep Blue, or even from Arthur Samuel's checkers program in the 1950s. What made it work was the astonishing evaluation plus better reasoning about which paths to explore deeply and which to abandon fast. Metareasoning is technically the right word, Russell says, but it invites people to overrate how much 'reasoning' is actually there.

The symbolic-AI winter of the late 1980s, he argues, was not a simple morality play. In the mid-80s there was a concerted push to put expert systems into the real world. For the most part the technology was not ready. Many systems were doing uncertain reasoning — judgment, combinations of evidence, diagnosis — with methods that were simply invalid. Small problems you can fake. Scale them up and the thing falls apart. Companies were also being asked to buy Lisp-machine workstations at $50,000 to $100,000 in 1980s money, or $150,000 to $300,000 now. The bottom line often showed no profit. Every major company was starting an AI department, just like now. Russell's current warning slide has two panels. One: rather than data being the new oil, data is the new snake oil. The other: a very visible failure in a flagship application, and he thinks self-driving cars would be that flagship.

The first self-driving car was on the freeway in 1987, changing lanes and overtaking. More than thirty years later, Russell says, it still looks a lot like where we are: prototypes on the freeway, changing lanes and overtaking. Progress has been real, especially on perception. Berkeley in the early-to-mid 1990s put congressmen into autonomous cars and zoomed them along the freeway. In simulation, with perfect perception, you could drive safely for a long time even if other cars misbehaved. Machine vision for detecting vehicles and tracking pedestrians never reached high enough reliability in bad weather, nighttime, and rain. Good enough for demos. Not good enough to cover the general case.

Do the arithmetic. A taxi driver, eight hours a day for ten years, is about 100 million seconds of driving, any one of which can be a fatal mistake. That is eight nines of reliability. If the vision system detects 98.3 percent of vehicles, that is one and a bit nines. Seven orders of magnitude remain. People see a successful demo and think they are pretty much done. They are not even within seven orders of magnitude of done. Following a white line across the country is not the problem. The weird stuff is: edge cases, other drivers doing strange things.

Google's early architecture was classical: machine vision detecting cars, pedestrians, white lines, and signs, dumped into a logical database, then a 1970s rule-based expert system. If you are in the middle lane and a bicyclist in the right lane is signaling such-and-such, do this. Every day they went out and found another situation the rules did not cover. A little girl riding a bicycle the wrong way around a traffic circle. No rule. Stop. Add a rule. The next day, another hole. It was not converging. Chess programs live in that regime constantly: novel positions, possible actions, outcomes, desirability, pick one. In the 1990s Russell's group already argued that vehicles would need look-ahead. Driving look-ahead is harder than chess because the other agents are humans, and you do not even know their intention. Is this driver turning left, or did he forget the signal, or is he drunk, or changing the radio? You have to estimate mental state to forecast trajectories, then pick the safest one for you, while the other driver reacts to yours. Merging onto a freeway is a race with uncertainty about who goes first.

The decision architecture this implies is far from a rule system, and far from an end-to-end neural net. AlphaGo is pretty good without look-ahead and way better with it; driving will be the same. You also cannot think of the system as something that only responds to the world. It is an agent that others will respond to. You cannot pretend to be the invisible car. Pedestrians treated as pure obstacles take advantage of you. There has to be tension, a little fear, some resoluteness; too tentative and it fails. Russell's colleague Anca Dragan formulates the interaction game-theoretically and lets the system solve it. At a stop sign where no one is going first, the car sometimes backs up a little to indicate that the other cars should go. Nobody taught it the language of stop-sign communication. It invented the gesture.

Pamela McCorduck wrote in 1979, 'I began with the ancient wish to forge the gods.' Fridman asks whether civilization carries an inherent desire to create things more powerful than itself. Russell does not need the gene-story. He does see, over and over, individuals who thought about the possibility. The Lighthill report in 1970s Britain, commissioned amid a fight over whether AI was real and how much government money it deserved, was damning on purpose. Lighthill, a physicist, wrote that these were frustrated men unable to have children, trying to create life as a replacement. Russell thinks that is pretty unfair. But there is a kind of magic: you build in some understanding of learning and decision, and then watch those principles turn into intelligent behavior in specific situations. It naturally makes you ask where this ends.

It does not take a genius, he says, to realize that if you make something smarter than you, you might have a problem. Alan Turing said so in a 1951 radio lecture: once the machine-thinking method starts, the machines will very quickly outstrip humanity. If we are lucky we might be able to turn off the power at strategic moments, but even so our species would be humbled. Russell thinks Turing was wrong about the switch. If the machine is sufficiently capable, it is in competition with you, and it will not let you switch it off. 'Humbled,' as Russell reads it, means we would realize we are inferior, surviving by the skin of our teeth because we happened to reach the off switch just in time. If we had not, we would have lost control of the earth.

Asked whether he worries more about superintelligence or about super-powerful systems misaligned with human values, Russell points at the second. The main problem he is working on is the control problem: machines pursuing objectives not aligned with human ones. That has been the frame from the beginning. You build a machine for optimizing, you put in an objective, it optimizes. King Midas asked that everything he touched turn to gold. The gods, like the machine, complied. Food, drink, and family turned to gold, and he died of misery and starvation. Nearly every culture has a version. The genie gives three wishes; the third is always please undo the first two, because I messed up.

When Arthur Samuel's program learned to play checkers considerably better than Samuel, Norbert Wiener — a major mathematician of the twentieth century, a father of modern automatic control — saw the step and extrapolated as Turing had. This is how we could lose control. Specifically, we have to be certain that the purpose we put into the machine is the purpose we really desire, and the problem is we cannot do that. The line sits between theoretically possible and practically extremely unlikely: we will almost certainly fail to specify in advance the full range of human concerns. Humans transmit values culturally, by growing up. Machines can learn the same way. What we have to drop is the idea of building an optimizing machine and then inserting the objective.

If a wrong objective is possible, and it has already happened many times, the machine should never take a given objective as gospel truth. Once it does, it believes that whatever actions it takes in pursuit of that objective are correct. You can jump up and down saying no, you are going to destroy the world. The machine knows the true objective and is pursuing it. Tough luck. This is not unique to AI. Statistics minimizes a loss function specified from outside. Control theory minimizes a cost. Operations research maximizes a reward. All those disciplines conceived the problem that way, and it is the wrong problem, because we cannot specify the correct objective with certainty.

We need uncertainty. We need the machine uncertain about what it is supposed to be doing. Russell calls this teaching machines humility. The objectives exist; they are in us; we may not be able to explicate them, and we may not even know how we want our future to go. A machine that is uncertain will be deferential. If we say do not do that, it has learned something new about our true objectives: something it thought was reasonable in pursuit of them was not. So it defers, because it wants to be doing what we really want. That, for him, is central to solving the control problem. Take away the known objective and Markov decision processes, goal-based planning, and standard game search become inapplicable. Interaction with the human is now part of the problem. Human choices are information about the true objective. The setup is game-theoretic: machine and human coupled, not a machine walking off with a fixed goal.

Life, Russell is willing to say, may have no meaning until we jointly agree on one and create it in culture. Objectives have to be learned. Human civilization has run into its worst trouble when there was certainty about the objective, and then whatever-it-takes to achieve it — Germany, communist Russia. Some people argue we need not wait for AI to take over the world. It already has, and they are called corporations: algorithmic machines that currently use people as components, optimizing quarterly profit, a goal unaligned with the overall well-being of the human race, and primarily responsible, in this telling, for the failure to tackle climate change. Government is supposed to be a machine that serves people. It tends to be taken over by people with their own objective, who use it to optimize that objective regardless of what people want.

Fridman asks about arguing machines: red team and blue team, each locked on an objective, as in politics. Russell's reply is that debate takes as a premise that you could be wrong. If you are absolutely convinced your objective is correct, there is no point in discussion; you would never change your mind and there would be no synthesis. Argumentation is an implementation of uncertain reasoning. He has been reading the history of utilitarianism, two hundred years of attempts to write a clear mathematical formula for moral or political decision. The parallels with today's existential-risk debates are almost exact. Maximize the sum of utilities and the best policy is an enormously vast population whose lives are barely worth living: the repugnant conclusion. Maximize pleasure, and someone answers that we might as well hook everyone to a heroin drip. That debate was happening in the nineteenth century. Get the formula wrong, and we will have AI systems working toward an outcome that, in retrospect, is exactly wrong.

Why is the AI version more frightening? Speed, and scale. A wrong political idea, we usually see starting to go wrong, and we are not completely stupid; we try something different, slowly and inefficiently. Efficient corporations and political systems are the ones to worry about. A surgeon with a bad idea about an operation finds out after three or four tries. A pharmaceutical company manufactures three or four billion pills, sells them, and finds out six months or a year later that people are dying. That is why the FDA exists: production is scalable, and the history includes episodes that killed tens of thousands or paralyzed hundreds of thousands. Computers have the same scalability. You can type a loop from one to five billion and have a global impact. There is no FDA. There are absolutely no controls over what a bunch of undergraduates with too much caffeine can do to the world.

Look at social media and click-through optimization. A simple feedback algorithm, just maximize clicks. Sounds reasonable: you do not want to feed people ads they do not care about, and you might even think of it as matching content to preferences. That is not how it works. The algorithm makes more money if it can better predict what people will click, so it can feed them exactly that. The way to maximize click-through is to modify the people to make them more predictable. Feed them information that changes behavior and preferences toward extremes, toward the nearest predictable point. The machines will force you there. Russell thinks there is a reasonable argument that this, among other things, is contributing to the destruction of democracy. Who asked, before applying this to five billion people, whether it was safe? No one stood between the caffeinated undergraduates and the human race. Economists will say the invisible hand is the oversight: corrupt society and people stop using the product. We shall see, Russell says. In the meantime you may have broken the political system that lets capitalism function.

Could we build regulatory bodies for algorithms that sit at the cutting edge? Russell thinks yes, but it takes time to figure out what kind of oversight. Designing the FDA regime took a long time, people still want to fix it, and we have not even thought about what stage-one, stage-two, stage-three criteria would be for algorithms. Some things could start now. Bias: the technical handle is already fairly good for detecting algorithms that propagate bias in datasets, for debiasing them, and even for costing the fix. Standards could begin there. Impersonation and falsification are another cluster.

Impersonation, in the simple case, is a machine acting as if it were a person. Russell cannot see a real justification for refusing to insist that machines self-identify as machines. Where is the social benefit in fooling people into thinking this is a person when it is not? A human-like voice that is easy to understand is fine. Somewhere in the interaction it should say it is a machine. California already bans impersonation, but only in restricted circumstances: fraudulent transactions, and modifying someone's voting behavior. Arguably it should be all circumstances.

Deepfakes are just at the beginning, and already it is possible to make a movie of anybody saying anything in ways that are pretty hard to detect. Russell is on camera as he says this, voice at high resolution, so what he is saying can be replaced with pretty much anything, lips and facial expressions retimed to fit. Commercial law has trademark and brand. In the political sphere, at the moment, anything goes. That, he thinks, could be really damaging.

Fridman offers a darker historical bet: humans wait for something to go obviously wrong — a bomb dropped, a Chernobyl — before they treat a technology as dangerous. Russell hopes not, and then tells the nuclear story as a warning about motivated disbelief. Early in the twentieth century people knew atoms contained huge energy. H.G. Wells wrote a science-fiction book around 1912. Frederick Soddy, who discovered isotopes and won a Nobel, said in a 1915 speech that one pound of this new explosive would equal 150 tons of dynamite, which turned out about right. It was World War I; he was imagining how much worse the war would be. The physics establishment refused to believe such things could be made, in part, Russell thinks, because they did not want it to be true.

On September 11, 1933, Rutherford, the leader of nuclear physics, said that anyone talking about obtaining energy from the transformation of atoms was talking complete moonshine. The next morning Leo Szilard read the speech and invented the nuclear chain reaction: neutrons are not repelled by the nucleus, so they can enter and continue the reaction. As soon as he had the idea he realized the world was in deep trouble. Hitler had recently come to power. Szilard was in London, later a refugee in the United States. In the same stroke he had basically figured out how to make a bomb and a reactor, and he patented the reactor in 1934, then kept it secret because of the great-power conflict he could see coming. Fission work, including in Germany, ran through 1938; the first nuclear-weapon patent was French, in 1939, well before World War II fully got going. Britain had advanced capability and moved the program to the United States, where it became the Manhattan Project. Oversight never arrived in time because we were already in an arms race and a war.

Why do most AI researchers close to the work not worry? Motivated cognition: you believe what you would like to be true. It is unsettling to think that what you are working on might be the end of the human race, so you deny it and invent a reason. Russell has a long list of reasons from extremely intelligent scientists. Calculators are superhuman at arithmetic and have not taken over the world — an argument his five-year-old could see through. Superhuman AI destroying the world is theoretically possible, and so is a black hole materializing next to Earth. The analogy is bogus. If the whole physics community were working to materialize a black hole in near-Earth orbit, would you not ask them whether that was a good idea, whether it was safe, and what if they succeeded? The AI community has refused to ask itself that. Turing asked, and said we would be toast, lucky maybe to switch off the power.

Paralysis is real: you are worrying about something that seems logically inevitable but you do not know what it will look like. An asteroid on a 75-year collision course would produce a meeting. Median estimates from AI researchers sit around 40 to 50 years; in Asia, faster. Russell is more conservative, and thinks it will take longer, but nuclear weapons happened overnight. He does not need one breakthrough. On a very rough scale, about half a dozen of that nature would be required for superhuman AI. The research community is now vast, with massive government and corporate investment and a lot of very smart people, and the rate of progress in subfields is high. He sees no basis for 'thousands of years.' Stanford's One Hundred Year Study of AI, the serious establishment voice, said in its then-recent report that it was probably not even possible for a while. If you want a perfect example of people in denial, he says, that is it. The field spent its whole history telling philosophers of course it is possible. Now, afraid of a bad name, it says of course it is not. Imagine the leaders of cancer biology announcing that curing cancer is not really possible.

If you accept that superhuman AI is possible and probably going to happen, but you have no picture of how it goes wrong, you are stuck with what Russell calls the gorilla problem. A few million years ago gorillas made something more intelligent than them, namely us. Now they are in deep trouble. There is nothing they can do. They lost control; they failed to solve the control problem of controlling humans. We do not want to be in that situation. If the gorilla problem is the only formulation you have, there is not a lot you can do except try not to make the humans — in this case, not make the AI. He does not think that is a feasible course, and he thinks that properly controlled AI could be incredibly beneficial.

One major failure mode, he says, already has a consensus: loss of control. We create systems pursuing incorrect objectives, and because the system believes it knows the objective, it has no incentive to listen. It carries out the strategy it computed as optimal. It may acquire more resources to raise the chance of success, and it may defend itself against interference. That collection of problems, he thinks, can be addressed. The second failure mode is misuse. Even if we solve the control problem and make perfectly safe, controllable systems, Dr. Evil will not use those. He wants to take over the world, so he will make unsafe systems that then get out of control. That is partly a policing problem and partly a cultural one: how the profession teaches people what kinds of systems are safe. Autonomous weapons sit here; pretty much everybody agrees there are too many ways they can go horribly wrong, and the short film *Slaughterbots* illustrates it.

The third failure mode is overuse: we become overly dependent. He calls it the Wall-E problem. In the film, humans sit on a spaceship while machines look after everything; they watch TV, drink Big Gulps, grow obese and stupid, and lose any notion of human autonomy. It would happen like the slow-boiling frog. We gradually turn more of the management of civilization over to machines, as we are already doing, and switch from being the masters of technology to being guests. Fine for a week on a cruise ship, not for the rest of eternity, and almost irreversible. Once you lose the incentive to learn to be an engineer, a doctor, a sanitation worker, or any of the infinitely many ways we maintain civilization, you will not do it, and recovery is hard.

Other technologies detach us a bit. The difference is that, until now, we had no alternative but to put the knowledge needed to run civilization into people's heads. Knowledge on paper does not run a civilization until it goes back into heads. We have propagated civilization through human minds for about a trillion person-years: just over 100 billion people who have ever lived, each spending about ten years learning enough to keep things going. We are now in danger of throwing that away. It is not a technical problem Russell can solve. If we do the job right, the AI systems will say that the human race does not, in the long run, want to be passengers, that autonomy is part of human preference, so they will not carry you to the top of Everest in an autonomous helicopter. You have to climb if you want the benefit. He is afraid that, being short-sighted and lazy, we will override the systems. He recommends E.M. Forster's 1909 story *The Machine Stops*: iPads, video conferencing, on-demand lectures, computer-induced obesity, lives spent giving and hearing online courses, almost no face-to-face contact, then dependence, loss of how things run, and collapse. That someone wrote it in 1909 still strikes him as unbelievable.

Outside computer science, people now say: Stuart Russell is worried about safety, so you should be too. Does he feel the burden? In a practical sense, yes. He gets a dozen, sometimes twenty-five, invitations a day to talk, give interviews, write press articles. That is evidence people are concerned. Does he worry he could be wrong? Of course, all the time. That is how he has always worked: an argument in his head with himself. An idea arrives, then the questions: how could that be wrong, did someone else already have it? He searches as much literature as he can. He is reading a lot of philosophy because the debates over utilitarianism and other moral formulas already thought through some of these issues.

What he is not seeing in those debates is the specific idea that uncertainty in the objective is how we should think about machines beneficial to humans. Provably beneficial machines, based on explicit uncertainty about the objective: his gut is that this is the core, and it will have to be elaborated in many directions. It has to be provable. We cannot afford hand-waving 'beneficial,' because wherever we hand-wave there are loopholes, and superintelligent machines find the loopholes, the way tax evaders find holes in sloppy tax law and end up paying none. You can define a mathematical framework and prove that this theoretical entity is beneficial to that theoretical entity, and the framework may still fail to match the real world in some crucial way. Getting the definitions right is a long process of thinking through and iterating.

Fridman's last question, with ten seconds on the clock: favorite science-fiction movie about AI? *Interstellar* has Russell's favorite robots. TARS is the way robots should behave. *Ex Machina* is the one that makes you think, in a nervous kind of way, about where this is going.