Intelligence Is Not Just Prediction: David Ferrucci on Explanation, Persuasion and Shared Meaning

Asked by Lex Fridman what intelligence is, Ferrucci says he thinks of it in two ways. The first is the ability to predict what happens next, whether that is the answer to a question, market dynamics or what someone walking into a room will do. The challenge is to do so in a dynamic environment full of uncertainty, taking a small amount of prior data, learning the pattern and predicting accurately and consistently.

He compares this to what deep learning does: given enough prior data and a defined output variable, find the function that predicts it. Getting it right more often than not is smart, doing it with less data and training time is smarter, and working out what is worth predicting is smarter still. That raises the question of goals, which he ties to what humans are pre-programmed for, such as survival, which in a social species includes understanding subtle social dynamics, finding a mate and reproducing.

Ferrucci says predicting accurately is certainly a form of intelligence, but it is not necessarily enough for others to perceive you as intelligent. If you keep being right but cannot say how, I may call you a savant, find what you do weird and uninteresting, and struggle to relate to you. In that case you are an alien intelligence relative to me, intelligent but unrecognized.

Once we can communicate and start to understand each other, my respect and my ability to relate to you change, and you become a human intelligence. He applies the same idea to animals, which do things we cannot comprehend and cannot put in our terms, so we regard them as lesser and treat them differently. He notes that nobody really knows what is going on inside them.

Fridman observes that in debates and on Twitter, people mostly tell stories, and that the task of explaining your reasoning is to give a convincing story, true or not. Ferrucci agrees and goes further: even a mathematical proof ultimately depends on other mathematicians being convinced. Some proofs have been studied for a long time before the community decided they actually proved anything.

He concludes that the notion of understanding is ultimately a social concept, since you have to convince enough people that you reasoned in a way they can follow and replicate. For AI, he says, one can define intelligence objectively as learning a function and getting it right often enough, and that can be very useful. But humans also need to take responsibility for decisions and relate to the process, so an agent, human or machine, is obliged to make others understand how it reached an answer.

He gives the example of a staff member who says only "I have a good feeling about it" when you are putting the company on the line. You want an explanation, and that explanation could be convincing without being true. He adds that if the listener is not good at objective, logical reasoning or at weighing probabilities across different paths, then collectively we will not do well.

Asked how hard it is for computers to explain their reasoning steps, Ferrucci says it is hard, partly because it is also hard for humans, and it is not clear we have a recipe. Nor is it clear what data or judgments we would use to learn it. He points to science as the enterprise meant to be objective and reasoned, and says that the most intelligent people are the think tanks, scientists and philosophers who work through the details with logic and the scientific method, rather than savants who give you a number.

Training someone to do that is a hard process, and society works hard to get others to understand its thinking. People can be persuaded by emotional means, but getting them to follow a logical argument is difficult. Scientists, journalists, artists, writers and teachers all try, with significant training, so asking how to get a machine to do it is a vexing question.

Fridman asks about algorithms at companies like Facebook and Google that persuade you to buy things, sometimes through emotional manipulation rather than reason. Ferrucci describes what they do more neutrally: certain features of things attract your attention, and the algorithm learns the function that predicts what you will click or buy, then gives you more of it. Whether that is good or bad depends on your goals.

He offers an optimistic reading: if you are searching for something, you have already reasoned that you need it, and the algorithm leaves it to you to decide whether it is an unhealthy addiction or a sound choice. The algorithm is not judging or reasoning, just showing more of the same, and he says the same logic applies to social media. Fridman notes the dichotomy that people expect such systems both to meet an extremely high bar and to bring out the better angels of our nature.

Ferrucci says the easy part is finding patterns, and the harder part is interpreting data against prior models of what is important, sensible, moral and valuable. Recommenders see superficial features such as words, length, colours or price, and do not ask why you are clicking, what assumptions you bring or whether the material leads to thoughtful conclusions. That judgement is not happening in AI today, because it requires getting at meaning and how humans interpret content relative to their values and deeper thought processes.

Fridman suggests meaning is itself a social construct. Ferrucci says meaning is often relative but implies going beneath the surface of the artifact, like a painting that means different things to different viewers and to the artist. As we move toward precise mathematics or engineering drawings, we narrow the space of interpretations, and aligning understanding means specifying much that is not in the artifact: your perspective, experience, assumptions, values and reasoning ability.

Humans can intuit much of this without being told because of shared experience: similar culture, education, prior models, brains, goals and emotions in similar circumstances. Fridman then returns to an earlier question about how biological and computer systems differ, and Ferrucci begins to answer that humans come with a great deal of pre-programmed material, which is why we can communicate.