The SS Soldier Who Hugged a Jewish Boy, and the Systems That Still Fool Us

Lex Fridman opens with a story Kahneman has told before. In Nazi-occupied Paris, early in the war, an SS soldier picked up a Jewish boy, hugged him, and showed him a photograph of another boy — probably his son — without realizing, certainly not, that the child in his arms was Jewish. Decades later that child is Daniel Kahneman, Nobel laureate in economics for folding psychology into judgment and decision-making, and author of *Thinking, Fast and Slow*. After the story he adds a sentence Fridman does not soften: the soldier was a good man, perfectly capable of killing a lot of people, and Kahneman is sure he did.

Asked what the war taught about psychology, Kahneman puts the surprise in one place: that the German people made systematic genocide a matter of state policy, and that enough people willingly took part. It is no longer a surprise, he says, but it remains strange. The Eichmann trial taught him that if it could happen in Germany, it could happen anywhere. The Germans were not special. He does not like the word *evil*. The mechanism is plainer. You can take a group of people out of the category of humans and put them in the category of animals. People slaughter animals without feeling much. Combine that with uncontrolled power over others, and you will not bring out the generous side of human nature.

In-group and out-group is basic equipment: loyalty and affection toward the in-group, and a willingness to dehumanize the out-group. The Holocaust did not have to teach that; it was a very sharp lesson in what people can do. Empathy, if it was there at the start, disappeared quickly. The fact that everybody around you was doing it — everybody shooting Jews — made it permissible. Whether every culture would have been as efficient and disciplined as the Germans is an interesting historical question; the underlying capacity, he thinks, is human nature. War also brings the other face: loyalty among soldiers, male bonding, a thrill to friendship under shared risk, up to the point where trauma leaves little behind. People will take a bullet for a comrade and fire on the unarmed with the same nervous system.

Kahneman says himself that calling them systems is a bit misleading. The word suggests two modules in the brain; there are not. It is also very useful. System 1 is easier to think of as a family of activities, distinguished by how ideas come to mind. Some arrive automatically: two plus two. Others you have to work for: 27 times 14. The second case means running an algorithm in your head. It takes time, occupies short-term memory and executive function, and makes you incapable of doing other things at once. The mark of System 2 is mental effort, and capacity for that effort is limited. The mark of System 1 is that it is essentially effortless. That is the major distinction.

Evolutionary stories about this, he says, are cheap and easy. His crude picture is that animals already have perceptual systems that cannot explain anything but can anticipate what will happen next — and prediction is a form of understanding the world. System 2 grew out of that, via language, a much larger brain, and the ability to manipulate ideas, imagine futures, imagine counterfactuals, and do conditional thinking. Those abilities would be impossible without language and a brain this size. System 1 is more like what animals have, but System 1 can also talk. The words coming out of his mouth are not each a deliberate process. They come out automatic and effortless. Language, for System 1, is a learned skill, in the same family as driving a car.

System 1 is not instinct. Driving is not innate; you go through a period when it is not automatic before it becomes so. Heuristics are the patterns that form after that automation: a new experience matched against an old pattern. When the match is poor, biases appear. Most of the time the match is good, so most of the time the anticipation of what happens next, and the plan for what to do, are correct. The sharper point is that in the domains it is good at, System 1 is much better than System 2. A strong chess player's candidate moves are all strong moves. The selection happens unconsciously, automatically, and very fast. System 2 verifies. Along that line, humans are machines that construct a pretty effective System 1 — which is close to a description of what current AI is doing.

What is happening in deep learning, Kahneman says, is more like a System 1 product than a System 2 product. It matches patterns and anticipates what will happen. It is highly predictive. What it does not have — and here he puts something ahead of the usual complaint about reasoning — is causality, any way to represent meaning, any way to represent real interaction. Until that is solved, what can be accomplished is marvelous and exciting, and limited. The speed is what has impressed him most: chess to Go, AlphaGo to AlphaZero, faster than anticipated, bewildering. There are problems you can solve that way and problems for which you need something else, something like reasoning.

He takes Gary Marcus's criticism as having a point. Children do not need a million examples; they need two or three. That gap is fundamental. To make a machine learn quickly you have to build some expectations or structure into it so it is ready. At the moment that seems unsolved. He is pretty sure people are working on it; if they have solved it, he has not heard. Fridman relays Yann LeCun's image: we can only see one or two mountain peaks ahead, and there might be a few more after that or thousands. Kahneman likes that better than Ray Kurzweil, which strikes him as off the wall. He stands with the majority who think neural networks will hit a limit in capability, while conceding that pattern matching is so much of intelligence that networks will almost certainly be a huge part of the final answer.

Translation systems already do a very good job, and they really do not know what they are talking about. For words to have meaning they have to get in touch with some perception or some action. Fridman calls that grounding. Kahneman is not sure a body is required. A completely paralyzed human brain can still learn a lot about the world; a perceptual system seems essential, a body merely helpful if you are mimicking humans. A machine that could accumulate world-knowledge through perception would be a big deal. Babies learn by connecting the movements of their hands with what happens in the world; people learn to operate an artificial limb impressively quickly. The brain is a system ready to learn the world through action. How hard is that to build? He kicks the ball back to LeCun: we do not know how many mountains there are. That is the honest answer.

On modeling pedestrians for autonomous vehicles, Kahneman is unusually optimistic, and the reason is scale. Every vehicle carries a huge amount of information, and anything one vehicle learns becomes part of what the whole system knows. With a multiplier like that, you can do a lot. Humans are complicated; the system will make mistakes the way humans make mistakes. Cars already sneak into roundabouts, so they must be anticipating how people react. Then Fridman notes the current systems treat pedestrians as things that cannot be hit, not as agents you interact with in a game-theoretic way. Every time someone tries to solve that dance, it looks harder.

Kahneman describes his own negotiation. When he crosses and a vehicle is approaching, he looks the driver in the eye. That is a signal, and an autonomous vehicle had better understand it. Fridman, after hundreds of hours of video, adds the next move: you look before you step into the street, and once you step in you look away. You are trusting a car that has not slowed yet. You are saying you are committed, as in a game of chicken; looking away is the visible commitment that forces the other player to stop. Does the machine need to understand mortality? Kahneman's first instinct is no — it has to anticipate. Chess and Go have a lot of anticipation and zero understanding. Fridman's follow-up surprises him. If you need a model of the human mind, it is a lot harder than he had thought.

Almost every robot–human collaboration, Fridman says, is harder than people realize. Kahneman's judgment is pessimistic and clear. In any system where humans and machines interact, if the machine has advanced enough to really help, the human will be superfluous within a fairly short time. Chess is the example. There was a period when Kasparov said human–machine combinations would beat everybody. Stockfish does not need people; AlphaZero certainly does not. The interesting exception would be problems the machine cannot solve and people can. Then you have to build into the machine an ability to recognize that it is in that kind of situation and call the human. That cannot be easy without understanding. To recognize the full scope of problematic situations, you almost need to be smart enough to solve all those problems. Every problem probably ends up like chess. The real question is how long the transition lasts, and how you divide labor while it does. Driving, being open, is a lot more complicated than Go; you need a hierarchical system that recognizes a situation and then brings up the relevant knowledge, more complicated than what we currently have.

A lot of people think driving is pretty simple because they think of their own experience, Fridman says, and they tell him humans are terrible at it. He thinks humans are incredibly good at driving and that driving is terribly difficult. Kahneman is honest about the limits of his own work: it has contributed nothing to understanding the ecology of situations or the complexity of problems. What is clear is that Go is endlessly complicated and very constrained, while the real world has far fewer constraints and many more surprises. People thought reasoning was hard and perceiving was easy. They learned that modeling vision was tremendously complicated and that proving theorems was relatively straightforward. "Quickly" took several decades, Fridman notes, and most people still have to learn it. The public's intuitions have not changed radically. They still evaluate the complexity of problems by how difficult it is for them to solve them, which has very little to do with the complexity of solving them.

That mismatch has consequences. The combination of hype, investment, and public intuition is what led to AI winters: media hype, companies putting money in, the tech not making money, a crash. Is there a way to educate people out of that System 1? Kahneman's answer is simple: no. It is going to take a long time before an understanding of what those systems can do becomes public knowledge, and before expectations catch up.

A device that cannot explain itself is a major, inescapable difficulty, and it is already here. In the judicial system, models that predict parole violations are clearly better than judges, but they cannot explain their reasoning, so people do not want to trust them. Do humans explain things? Kahneman's answer is no, with an interesting twist. Humans *think* they can explain themselves. Ask why you believe something and reasons will occur to you. His own belief is that in most cases those reasons have very little to do with why you believe what you believe. They are a story that comes to mind when you need to explain yourself. People traffic in those explanations anyway. Human interaction depends on those shared fictions, the stories people tell themselves.

Fridman draws the unflattering inference: maybe explainable AI does not need to explain the model. It just needs to tell a convincing story. Kahneman: absolutely. The story does not necessarily need to reflect the truth; it needs to be convincing. You can say that in a way that sounds cynical or in a way that does not. The brilliant objective of an explanation is to tell a story people will accept, and for it to be robustly acceptable it has to have some element of truth. The objective is still for people to accept it. That, he says, is quite brilliant.

Kahneman distinguishes two selves. Most of the time we just live. Experiences are better or worse, it goes on, and we forget most of what happens. Occasionally, when something ends, we evaluate the past and form a memory. The memory is schematic; there is no roll of film. You construct the elements of a story about an episode. There is the experience, and there is the story created about it — the remembering self. One self does the living; the remembering self is all we get to keep. Decision-making and everything we do is governed by memory, not by what actually happened, by the story we told ourselves or the story we are keeping.

He studied how we construct those stories. One finding is especially sharp: in stories, time does not matter. There is a sequence, a highlight, an ending, "they lived happily ever after," and three years later something happens. Events matter; duration does not. That creates a pile of paradoxes, because time is all we have to live. Time is the currency of life, and it is basically not represented when we evaluate memories. He abandoned happiness research because he could not solve the problem. What makes the remembering self happy and what makes the experiencing self happy are different things. Suppose you are planning a vacation and you are told that at the end you will get an amnesic drug, remember nothing, and all the photos will be destroyed. Would you still go to the same vacation? People go on vacation in large part to construct memories, not only to have experiences. The vacation you want if you will not remember is probably not the same vacation you want if you will.

Fridman offers a contemporary version: social media magnifies the remembering self. Many people live for the picture they post, which thousands or millions can share and which they can relive far beyond those minutes. Kahneman laughs that he is too old; he has never seen Instagram and cannot speak intelligently about it. He will follow the logic. Devices are supplanting functions. You do not have to remember phone numbers. You do not really have to know facts; conversations now include "let's look it up." Knowing things used to be important and is becoming optional. Sharing itself is not new. There was a time you could assume your friends had read the same books, the same *New York Times*, the same reviews; with three television channels you had probably seen the same one. Sharing was always there. It was just different. The psychology of this magnification, he thinks, has not been described yet, because it is new.

Fridman mentions Sartre, Camus, and Buddhism as philosophies that lean on the experiencing self: just experience, do not evaluate, do not keep score. Kahneman does not know enough philosophy to answer cleanly, but he can see the family resemblance. If you could imagine a life in which people do not score themselves, it feels as if that would be a better life. "How am I doing?" is not a very happy question to have. He left the field because his intuition was that the experiencing self is reality, and then it turned out people do not want experiences. They want memories and a good story about their life. You cannot have a theory of happiness that does not correspond to what people want for themselves. When he saw where that was going, he got out.

Looking back, Kahneman says, is definitely a feature, not a bug. You have to look back in order to look forward; without it you could not intelligently predict. Fridman brings up Viktor Frankl's *Man's Search for Meaning* — purpose as what can save a person from suffering in the camps. Kahneman can see that a feeling of purpose and meaning could sustain someone. He does not have that feeling. He is pretty sure that if he were in a concentration camp he would give up and die. Frankl is a survivor. Kahneman is not sure how essential that stance is to survival. There is a sort of character that manages to survive in those conditions, and because they survive they tell stories. It sounds as if they survived because of what they were doing. We have no idea. They may have survived because of the kind of people they are, the kind who tell themselves stories of a particular kind.

He does not think seeking purpose is a significant driver of behavior. Ask people whether meaning in life is important and they say it is the most important thing. Ask what kind of day they had, which experiences they remember, and you do not get much meaning. You get social experiences. Taking care of your own children versus someone else's makes a very big difference, and he thinks that is entirely true — more because of a story we tell ourselves, a very different story in the two cases. On whether the lab can capture real behavior, his view is that you can learn a lot, then you have to verify, because conclusions are limited to the experimental situation. The inductive leap to the real world is where good psychologists separate from mediocre ones: a sense that the experiment captured something important and real, rather than just running experiments. Skilled intuition tells you an idea is going somewhere before you can explain it. Advice to others? Follow your heart. Explaining it is like trying to explain what it is like to drive; you break it apart and lose the experience.

Kahneman reads — Fridman quotes — his own account of the years with Amos Tversky: twelve or thirteen years of interpersonal and intellectual bliss. Almost everything was interesting, almost everything was funny, and there was a current joy of seeing an idea take shape. Many times they shared the magical experience of one of them saying something which the other understood more deeply than the speaker had done. Contrary to the old laws of information theory, it was common to find that more information was received than had been sent. He has almost never had that experience with anyone else. If you have not had it, you do not know how marvelous collaboration can be.

How do you find that? It is like asking how one finds love. You have to be lucky, and you have to have the character for it. He has had many collaborations, none as exciting as with Tversky, and he thinks he is good at it; not everybody is. Serious collaborations, in his experience, are very personal. You have to like the person. There is a commercial kind of collaboration, an exchange of this for that. The real ones are between people who like each other, who like making each other think, and who like the way the other person responds to their thoughts. In the room, the process happens again: Fridman mentions pedestrian prediction and Kahneman's curiosity lights up. There is no trick. He is curious about anything to do with AI and robotics, he knew Fridman was dealing with that, so he asked. Follow your curiosity.

On the replication crisis, Fridman is polite. Kahneman says he is being polite; it is worse than that. His theory starts with a distinction. Within-subjects experiments put the same person in two conditions. Between-subjects experiments put different people in different worlds. Between-subjects work is harder to predict, more expensive, and where the problem is. Researchers' intuitions about it are wrong for a subtle reason: the researcher is always in a within-subjects situation. You imagine both conditions, you see the causality, you feel it. The subject lives in one condition; the other is nowhere. So intuitions about between-subjects experiments are very weak, and the same manipulation is much more powerful when you are in both conditions than when you live in only one. Experimenters systematically overestimate the power of their manipulations.

Almost all psychological hypotheses are true, he says, in the directional sense. A does not cause the opposite of B; A just has a much weaker effect than you think when you are imagining it. Friends of his funded 53 studies of behavioral change by 20 different teams, with a precise objective: change how often people go to the gym. The success rate was one. Those are the best people in the field, and they are not calibrated. They think it will be powerful because they can imagine it. Actually it is weak, because you are focusing on your manipulation and it feels powerful — the focusing illusion: when you think about something, it looks more important than it is. The way out is to trust intuitions less. Experiments have to be preregistered; by the time you run one you have to be committed, run it seriously, and in public. That is already happening. Mechanical Turk makes it cheap and effective, and whenever psychology gets an important new method, the field is rewritten. Classic vision results were done on three or four people, one of them partly color-blind. For weak between-subjects effects, the old standard of 30 or 40 subjects is not enough. You may need a couple of hundred — an order of magnitude.

Asked about lasting theoretical disagreements, he names one long, unpleasant controversy with the German psychologist Gerd Gigerenzer, who has been after the Tversky–Kahneman work for thirty years or more. He does not particularly want to talk about it, and he thinks it will never be resolved. You build a system and you live inside it; other systems of ideas look foreign; there is very little contact and very little mutual influence. Changing your mind, once you are committed, is extremely difficult, and that is true of scientists too. On things that matter — politics, religion — people just do not change their minds, and there is very little you can do about it. What does happen is that leaders change their minds. The American public does not really take climate change seriously; if some religious leaders decided it was a major threat to humanity, that would have a big effect. We have the opinions we have not because we know why, but because we trust some people and not others. It is much less about evidence than about stories.

A good test of intelligence? The standard definition of AGI is that it can do anything people can do, and better. What we have are domain-specific programs that beat people easily in a specified way, very far from general-purpose intelligence. AlphaZero was much more general than AlphaGo and is still extraordinarily narrow. The Turing test's natural-language conversation is too easy to pass under constraints. What would impress Kahneman in a conversation is not factual chat, which he thinks is easy. Humor would be more impressive. Allusions would be interesting. Metaphors would be interesting — new ones, not practiced ones. That is a high bar: humor and a fresh metaphor both require mapping two distant semantic spaces, generated now, fitted to context, which is exactly what current systems are worst at. Human-level or superhuman intelligence: fascinating, exciting, also terrifying. He will not be around to see it. Predictions about thirty years from now are silly. "We really have no idea."

Fridman asks the meaning of it all. Kahneman does not dodge. There is no answer he can understand, and he is not actively looking for one. There is no answer we can understand; he is not qualified to speak about what we cannot understand. We cannot understand reality, and we can still do a lot. Gravity waves are a big moment for humanity. An ape able to go back to the Big Bang is astonishing. The *why* is bigger than us. The why is hopeless. Fridman closes with Kahneman's own line: intelligence is not only the ability to reason; it is also the ability to find relevant material in memory and to deploy attention when needed. System 2 reasons. System 1 retrieves and aims attention. The second job is most of what intelligence is.