You Can't Have Both Kinds of Fairness, Says Peter Norvig

Lex Fridman opens with a stack many researchers know by color: the red, green, and blue editions of *Artificial Intelligence: A Modern Approach*, the field-defining textbook Peter Norvig wrote with Stuart Russell. Norvig is director of research at Google. The first edition appeared in 1995, with later editions in 2003 and 2009; they are working on a fourth. Fridman asks how the book, and Norvig, have changed across those versions.

Hardware is the first shift Norvig names. The first edition treated predicate logic as something that would explode into millions of expressions that could not fit in memory, so first-order logic looked more concise. They soon realized fast solvers made those millions—and later billions—of expressions cheap. A similar hardware jump is happening again with GPUs and custom ASICs for deep learning. The more interesting change, he says, is philosophical. The first three editions assumed AIs maximize expected utility and you tell us the utility function; twenty-seven chapters then taught how to optimize it. This time they are saying the optimization may be the easy part. The hard part is what I want, and what a collection of agents or a society wants as a whole.

The book has always had a philosophy chapter. It used to linger on arguments such as the Chinese Room. Now it spends more time on fairness, bias, and aggregating utilities. There is no settled method for encoding human values. One technical beginning is inverse reinforcement learning: instead of taking actions and collecting rewards, you watch someone act and infer what they must have wanted. The catch is that people take self-destructive and otherwise suboptimal actions. The field has to recover something like ideal behavior, not a copy of whatever was observed. That is still an open challenge.

Norvig's cleanest illustration is commercial software used to predict recidivism and to inform parole or bail. Fairness across protected classes such as race splits into two demands that both sound obligatory. First, calibration: a score of six out of ten should mean the same thing regardless of race—about a 60% chance of reoffending. One vendor optimized for that balance and says it achieved it. Second, mistakes should not fall on one group. In the system he describes, harmful errors are about twice as likely for a Black defendant as for a white one. That also looks like unfairness.

You would like both. The analysis shows it is theoretically impossible. You have to trade one goal against the other. Norvig finds that result useful precisely because it kills a fantasy: you cannot have everything, but you can stop arguing for an impossibility and start deliberating, as humans, where the tradeoff should sit. At least the debate is then informed. For anyone shipping a machine-learning system that scores people, the lesson is to name which fairness definition you optimized, which you sacrificed, and to hand the remaining choice to public argument rather than hide it inside a single metric.

Fridman pulls a fuzzier example from a commencement talk: games such as *Angry Birds*, and feeds that optimize for likes, that chase a dopamine hit and a return visit rather than any long-term improvement. Norvig agrees the issue matters and then refuses to treat it mainly as an AI problem. Society built an attention marketplace and decided it likes things to be free. Phone apps are therefore free, which means they compete for attention and later extract money through ads or in-game sales. The only way to win is to beat every other app by stealing focus. In that market they look as if they are working against you, not with you.

He would rather change the playing field so the software feels like it is on your side: fun in the short term *and* helpful in the long term. Those two objectives, he says, are not in conflict. The conflict is the current incentive structure, which sits on the easier side of the fork—the dopamine rush—while we are still figuring out a new world. Algorithms look antagonistic because they are loyal to the market we designed, not because antagonism is a property of the math.

In the mid-1990s Norvig was still at Berkeley, lunching with colleagues who all agreed the field was moving and the textbooks were behind. Everyone said someone should write a new one. Nobody did. He left for industry at Sun Labs and assumed academic publishing was over. A year later, at a conference, he needled Russell: that book you were always talking about—you must be halfway done. They had kept talking and never started. So they decided to write it, because they thought the field was turning in two directions at once: from Boolean logic toward probability, and from knowledge engineering—interview an expert, write down everything by hand—toward machine learning. Other books had not fully made that turn. Starting from the new point of view could produce a different kind of text.

The collaboration was primitive by later standards. Norvig had moved to Boston; Russell stayed in Berkeley. They worked over dial-up Internet, typing `cat` on a file and hoping the other end captured it, with no practical way to send figures. Asked whether they foresaw deep learning and big data, Norvig is blunt. They knew learning mattered. They underestimated how much. They missed big data. Deep learning had not been invented. They could have written the whole book from a machine-learning stance; instead they walked axes of representation and environment—fully versus partially observable, deterministic versus stochastic—and put the learning axis later. That choice looks imperfect now, and it is also why the book still covers a field rather than a fashion.

In the fourth edition the deep-learning chapter is a guest essay by Ian Goodfellow, who compressed his entire deep-learning book into one chapter. They handed him the old neural-net chapter and told him to modernize it. He told them about half of it was still fine. The core had not moved as much as the outer ring. Norvig does not know yet how the craze will settle: convolutional nets began next to perception and later reached actions and some longer-term planning, but representation, reasoning, and one-shot learning still need more.

Fridman asks whether the 1980s symbolic systems, after their partial demise, still hold kernels worth bringing back. Norvig says representation and reasoning remain crucial. Sometimes there is not enough data to learn the world de novo; you need a representation—programmed in or told—and then steps of inference. The failure of good old-fashioned AI was not that it used symbols and logic. It was two mismatches. First, the symbols were atomic. That is fine for the necessary and sufficient conditions of a triangle. The real world is messy, without sharp edges, and atomic symbols do not fit it. Second, the reasoning was universal: applicable anywhere, which sounds like a virtue and means there is no guidance about where to apply it.

The classic heap paradox follows: remove one grain of sand and you still have a mountain; repeat, and at some point you do not. Logic will not stop you from applying the rule ten thousand times. It has no notion of *enough*. Norvig does not even have a name for the fix he wants, but the direction is: give "mountain" something more like a word embedding than an atom, and learn when a rule may be repeated. Induction on the integers can run infinitely. Taking sand from a heap is done after tens of times. Strategies should not be completely neutral and available everywhere; they should learn the domain in which they are allowed.

Neural nets learn representations from data and erase a lot of structure humans would want to inspect. They succeed brilliantly and fail on tiny perturbations, with no introspective story. Fridman asks whether more data will fix that or whether symbols must return for explainability. Norvig would rather talk about trust, validation, and verification; explanation is one tool toward those goals. Europe's GDPR can require a bank to explain a denied loan. An explanation alone is not enough. People and institutions already give reasons that may have nothing to do with the real cause: not enough collateral, or they disliked your religion. You cannot tell from the sentence, whether a person or a machine decided.

What he wants is a conversation: you gave this reason; what if the facts had been different; what would I need to change? The second layer is testing across cases. Staring at your own file cannot reveal whether the decision was collateral or skin color; looking at everyone else's can. The third layer is adversarial testing. Image recognition looked close to human until adversarial examples: a perturbation no person sees, and the model is completely wrong. Humans have their own holes—the dress whose color no one can agree on—but the attacks differ. The machines' failures are striking enough to change what you think you have built.

The seduction, he says, is a low-dimensional metaphor. Textbooks draw cats on one side, dogs on the other, and a tiny fuzzy strip in the middle, so you conclude we are nearly done and adversarial images are a few exceptions. The real geometry is a million-dimensional space in which cat is a very thin curve shooting out in some crazy direction. Step a little off the path and you are in nowhere, with no idea what happens. That is why adversarial examples are structural. The model has only seen points on the line; a millimeter off it still answers with high confidence. Fridman wonders whether we are too hard on robots compared with the fast trust between two people who have just met. Norvig agrees, and cites the naturalist Mark Moffett: the amazing human trick is walking into a coffee shop full of strangers and not killing one another. Chimpanzees cannot do that, especially with food around. Humans still go to war, but they have mostly learned to share space. AI systems will need some version of that. Many problems labeled "AI," he adds, are really communications problems: we can build these systems because we can collect data and reach people worldwide. That change is bigger than the algorithms.

Norvig's Udacity introduction to AI drew about 160,000 sign-ups, one of the first large MOOC successes. The lesson reversed his prior belief. He had thought the job was information: explain the material clearly and good things would happen. After teaching, he decided motivation is the scarce resource. If students do not stick, content quality does not matter. He is unimpressed by low completion rates as a verdict of failure. He himself has started perhaps fifty courses and finished none, and got exactly what he wanted, because he never intended to finish—only to sample a topic or watch someone else's pedagogy.

The first wave of courses also harvested external motivation: it was new, and being part of it was exciting. Students arrived already driven. The people who most need help often lack that drive and cannot see what finishing would buy them. Technology alone will not close that gap. Community will, and some of it has to happen in person. Fridman suspects he would know less CS if he had not been trapped in hard classrooms, sometimes bored, sometimes asleep. Norvig splits that experience in two. Physical location is one variable. Affiliation is another. You persist partly because you see everyone else suffering, and partly because you paid tuition and parents, peers, and society expect you to continue. Those can be separated: expensive tuition, public commitment, and everyone in different rooms—or a shared classroom full of people whose minds are elsewhere.

What decides outcomes is the commitment, not the screen. Informal talk after class still matters for motivation and for the learning that actually sticks. Higher bandwidth, AR, and VR might someday reproduce that without a shared building. He does not expect Stanford or MIT to go online-only. They have huge financial and cultural investments in campuses, and some work cannot be virtualized: a CS student can work with a laptop and paper, but a biology lab's equipment is not at home. Georgia Tech's online master's is the sort of creeping exception he already sees, usually in further education with working students' constraints.

In 2001 Norvig wrote "Teach Yourself Programming in Ten Years," a retort to the twenty-one-day books. He would now qualify it. He had in mind a career as a professional software engineer, which really is lifelong. Plenty of people now need programming only as a means: a biologist, physicist, or historian with a data set and a question. That does not take ten years. He likes teaching kids to code and wishes the slogan would change. Code is not the point. He does not care if you know JavaScript syntax or can snap blocks in a visual language. He cares whether you can analyze a problem, invent a solution, build a model, run it, test it, check that the results are reasonable, then ask and answer the next question. The practice is modeling and problem-solving. Code is the tool, used to answer questions you actually care about.

He tells on himself. On a project with a younger colleague, a new package looks useful. Norvig starts the manuals. Two hours later the colleague is done: downloaded, installed, first call failed, second call worked. Norvig still has questions about internals and parameters. *Who cares?* I answered my question. The package is huge; I do not understand the rest; I got the right answer. He finds that mindset hard. If there is a manual, he feels he should read it. He is trying to get more comfortable with not knowing everything. Fridman names the spectrum: Donald Knuth, who will not speak until he is at machine code, versus twenty-year-olds who solve in hours what would take Norvig weeks of reading. Abstraction keeps rising. You can do a lot with a deep-learning framework without knowing the internals. Norvig thinks we should never have centered the identity "programmer." Skill is a means; the result is what counts. Early in his career a library was square root and you built the rest. Now you download most of what you need. "It's more a question of assembly rather than manufacturing."

Google's hiring evolved with that. Early interviews asked, in effect, how close is this person to Jeff Dean? Most people were far; they took the less-far ones and got a homogeneous group of excellent programmers. A larger company should not want everyone the same, so they hire biologists for health, physicists, mechanical engineers, social scientists, ethnographers. In code review Norvig looks at flexibility of design: this API solves today's problem, but who else will call it, and how are you making that easy? Documentation helps; design dominates. If the design is right, people will figure it out; if it is wrong, good docs will not save it. His own biggest change as a programmer is letting go of small-scale efficiency. In a 1990s book he showed objects as hash tables, then warned that real Common Lisp was far more complicated and that nobody would ship the toy. Python pretty much did exactly that: objects are dictionaries, plus a few tricks. A style that would have been a hundred times too slow in the 1980s is now fast enough for almost everything. What matters is total time to finish the project, and most of that is programmer time. Slightly less efficient and easier to understand and change is the right tradeoff.

What is beautiful in Lisp? Simplicity. Half a page can define the whole language, so you can hold it in your head. Other languages pile up a hundred or two hundred syntax rules. Lisp's answer was roughly eight rules, and you define the rest. That is the idea Norvig still finds most powerful: start from your problem and your data, build the language you want for them, and let Lisp implement it. You are simultaneously a programmer in a language and a language designer, so the code can sit closer to the problem. He thinks Lisp did that better than other languages. Why no mass adoption like Python? Two reasons. It was designed for a single programmer or a small team with good taste. Good language-design choices are wonderful; bad ones hurt you and confuse everyone else. There is a limit to the number of people a Lisp project can absorb, and the industry grew past that scale.

Then there are the parentheses. The joke acronym is "Lots of Irritating Silly Parentheses"; his is "Lisp Is Syntactically Pure"—parentheses and atoms. The 1990s textbook used pseudocode in the body and Lisp in an appendix, because that was still AI's language. A student complained that he could not see how the pseudocode mapped to the Lisp. The symbols corresponded one-to-one; the only difference was the parentheses. Norvig's conclusion: for some people, past a certain number of left parentheses, the brain shuts off. A ten-to-fifteen-week course cannot spend two weeks teaching brackets. Java was the popular alternative and too verbose for a one-to-one match with his pseudocode. He surveyed for the language most like that pseudocode and found Python. "Somehow I had channeled Guido," he says—he had not heard of Python when he designed the pseudocode. His GitHub repo pytudes is a pile of puzzles and exercises that began as write-ups from a Udacity problem-solving class and continued because he likes puzzles and teaching.

From 2001 to 2005 Norvig was Google's director of search quality, in a company doubling every year. The work was to define a right answer, implement it, keep it efficient, and keep testing. A good answer meant that whatever keywords people typed, results were ordered by how much they actually wanted them—the first click should often be the thing they were looking for. They tracked first-click metrics, whole-page metrics, top-three metrics, and they carved out query classes they were failing and tried to fix those. The philosophical shift came when they realized they were not a library card catalog, standing to the side and reflecting the books. Every change they made, webmasters made a change. It was game-theoretic. You have to ask not only whether a move is good now, but what the countermove will be, and skip the move if the resulting position is worse.

They also changed the shape of the web. Early on, Larry Page and Sergey Brin had PageRank, and Jon Kleinberg had hubs and authorities: pages of "my dog links" pointing at the authority pages everyone else linked to. People no longer bother to maintain those hub pages, Norvig says, because Google took over the function. Asked whether he foresaw today's internet, he says he is bad at predicting. As a graduate student around 1980 he used the ARPANET. When commercialization and the internet were proposed, and Senator Al Gore thought it a good idea, Norvig's reaction was that you could not expect commercial companies to understand the technology. A `.com` domain would not go anywhere. "I was wrong. Al Gore was right."

What would it take to approach human-level intelligence? Norvig's first reply is that human-level intelligence is not one thing—it is many tasks and capabilities. His second is that it should not be the goal. He would not want a calculator that multiplies at human level; that would be a step backward. On many abilities we should aim far beyond humans. On some, human level is a sensible threshold. On others, skip the work because humans already do it. He prefers to ask whether a tool is useful. Sometimes crossing human level is exactly the threshold that makes it useful. Voice assistants on phones and speakers are the example: you want a conversation the way you have one with a person. The industry has not found the right product model. You can ask the weather, play songs, and maybe five or six other things, then you run out. There is no deep, meaningful connection.

Could a person fall in love with an assistant, the leap from weather to the film *Her*? Norvig thinks that is what people love to do. At a panel after a screening, someone asked what other movie *Her* resembled. He said *Life of Brian*—not science fiction. Both films are about wanting to believe in something that is not necessarily real. Kids fall in love with teddy bears that barely interact. People working on autonomous vehicles told him many drivers already invest deep feeling in inanimate cars. Get to dog level and a lot of people will pour in love. As a test of intelligence he is already impressed by Go and StarCraft. Short conversation matters, but many chatbot tests never force a moment deep enough to reveal whether anything intelligent is there. If Turing were alive, Norvig suspects he would say he did not mean the imitation game that seriously. The point was having a test, so philosophers could not just declare AI impossible. The test need not be conversation. Better tests as the technology evolves are the right path.

Does existential risk occupy him? He thinks about threats. Powerful technology cuts both ways. He is not worried about a robot apocalypse or *Terminator* scenarios. He is worried about employment: whether society can react fast enough. Plenty of people are already angry about income inequality, and automation could accelerate that. Powerful tools become weapons—robots, drones—and many threats need no AI at all. He does not know whether autonomous drones or widely available CRISPR is worse. Asked whether he is optimistic that technology will also relieve the damage it causes, he refuses the prediction. Society has survived nuclear bombs, but only surviving societies get to have the conversation, so that may be survivorship bias.

What is exciting to work on next? Conversational assistants that combine common-sense reasoning with the power of data. Programming tools: not only making frameworks such as TensorFlow easier for more people, but applying machine learning to ordinary programming. When you misspell a Google query, it asks "did you mean" because many other people made the same error and then corrected it. We should be able to mine codebases and bug-fix histories so that when you type a line it can say did you mean this?—and if you type this today, you will probably type that fix tomorrow. He also wants those capabilities in more people's hands. None of that is a grand prophecy. Across the interview Norvig keeps saying he does not know, and keeps moving the problem from the model to the institution: fairness tradeoffs for public debate, attention markets for incentive redesign, education for motivation and community, AI risk for jobs and weapons. The man who spent decades putting a whole field into a textbook ends by admitting the urge to understand everything, including the whole manual, is not always the right move. Answer the question first.