Watson Won Jeopardy Without Understanding. Ferrucci Wants AI That Can Explain Itself

David Ferrucci's undergraduate degree was in biology, with an eye toward medical school, before he went on for a Ph.D. in computer science. Lex Fridman opens with an easy question that is not easy at all: what is the difference between biological systems and computer systems? Ferrucci says he often wonders whether there is a substantive difference. That question is what pulled him into artificial intelligence in the first place. If we can get machines to think, understand, and process information, if intelligence itself can be described as a process, then what remains distinct besides the implementation?

He is not trying to settle debates about souls or spirituality. From what we can experience intellectually and physically, he is not convinced a biological information-processing system is fundamentally more capable than one built from silicon or some other substrate. Studying the human brain still has a clear purpose: diagnosing and treating injury, and understanding our own strengths and weaknesses. Recreating human intelligence, though, is a different and more doubtful goal. Human intelligence has a lot we envy, and a lot of problems too.

The better questions, he argues, are what we want out of an intelligence, how we want to communicate with it, and how we want it to behave. He notices the loop he is standing in. He is sitting there as a human with a biological brain, critiquing the strengths and weaknesses of human intelligence, and claiming we have the capacity to step back and ask what intelligence really is. That capacity to introspect is itself something quite enviable. The flaws he names are prejudice and bias in how we draw inferences: an inductive habit of using past data to predict the future, which is advantageous when you must act quickly and survive, and prejudicial when you need slower, more objective decisions.

Ferrucci splits intelligence in two. The first layer is prediction. Given a little prior data, can you find the pattern and say what happens next, accurately and consistently, in a room full of uncertainty? The more variables and possibilities, the harder the problem. If you can do it with less data and less training time, you are smarter. If you can also figure out what is even worth predicting, which path gets you toward a goal, you are smarter still.

That is largely what deep learning and machine learning do. Give the system enough history, tell it which output variable matters, and it learns to predict. Getting the answer right more often than not is a genuine form of intelligence. It does not require understanding in any rich sense, only the ability to find a function. A machine can be excellent at this and still be unable to tell you how it did it, or to teach you the method.

If it cannot describe its underlying theory, you may still grant that it is intelligent because it is right more often than you are. You will also, Ferrucci says, label it a savant. It is doing something weird, and you cannot communicate with it, so it becomes uninteresting. Animals sit in a similar place: they can do things we cannot quite comprehend, but they cannot put those processes in our terms, and we do not treat them as peers. Recognition as our kind of intelligence depends on communication. Until then the system remains an alien intelligence, impressive at the output and mute about the method.

Lex notes that human explanations are often stories, not proofs, especially in political debate and on Twitter. Ferrucci replies that a mathematical proof is also a form of persuasion. There have been proofs that mathematicians studied for a long time before they were convinced they proved anything. You never know if a proof proved anything until the community of mathematicians decided that it did. Understanding, in that sense, is not a private certificate. It is an agreement.

"This notion of understanding is ultimately a social concept," he says. You have to convince enough people that you arrived at the answer in a reasonable way, one that others can understand and replicate. Human intelligence is bound together that way. You never really get away with a claim until others accept the thinking process. If we define intelligence only as hitting a prediction target, a system can be incredibly useful and still leave humans unable to take responsibility for the decision.

He offers an office example. You ask the staff whether to buy a stock or launch a product, someone says yes, and when you ask why, the answer is "I just have a good feeling." Even a person with a strong track record will not satisfy you if the company is on the line. The explanation may have nothing to do with the truth, but it has to be convincing. That is why the second layer is hard for machines too. We do not have a recipe for teaching it, and it is not clear what data would even represent good reasoning. Science, journalism, teaching, and writing are all training processes for this skill, and they take years.

Recommendation algorithms, by contrast, do the easy part. They match features to whatever you already click, with no theory of whether the interest is healthy, reasoned, or addictive. Ferrucci refuses to call that good or bad in itself; the algorithm does what it does and washes its hands. Getting at meaning, values, and whether the material leads to thoughtful conclusions is a different job, and much harder, because it requires interpretation. Meaning, like understanding, is relative and social. Aligning two people on a painting, a paragraph, or a photograph means specifying priors, values, and chains of implication that are not in the artifact. Humans can intuit some of that because they share brains, cultures, and goals. Machines do not arrive with that pre-programmed substrate.

Humans communicate because they share a huge amount of pre-programmed structure: similar brains, similar goals, similar emotions under similar circumstances. Ferrucci thinks machines can acquire knowledge on a similar interpretive foundation. Humans bring frameworks to the world. Faced with a social-political event, we automatically ask who the people are, what they want, how scarce the resources are, and who has power over whom. That substrate is so ordinary we barely notice we are using it.

Detailed knowledge is effectively infinite; unique situations never stop arriving. The frameworks for interpreting them, he thinks, are finite. Frameworks matter more than bulk facts because they are the skeleton for explanation and reasoning. With a skeleton, new details can hang. Without one, they are scattered strings. People who apply a different framework, he jokes, can end up in *The Matrix*, treating humans as batteries. Most of us share enough of the ordinary framework to make sense of one another.

Lex presses on common sense: gravity, drinking water, the onion of unspoken knowledge that robotics always seems to find. Ferrucci thinks the problem is solvable, and that learning is fundamental. What we have not done is teach machines the frameworks themselves. We rarely annotate the world's data with the interpretive structure we take for granted; we just operate with the data. Architecture that both finds patterns and acquires frameworks, then connects the two, is the path he sees. A system can learn that things fall without ever meeting Newton, but it will not relate that experience to a theory unless someone connects the two.

At Elemental Cognition he combines both: neural nets to induce patterns, and knowledge representations, graphs of logic, that connect those patterns to frameworks humans understand. There is also a method for acquiring the frameworks themselves. Collaboration with humans is not optional if the goal is communication. You can build what he calls an alien intelligence that outperforms you and cannot explain itself. His goal is the other kind: machines that acquire knowledge from people and give it back in terms people share. Analogy is one of the most powerful transfers. If you have never heard of soccer but you know basketball, you already have a hoop, guards, and a place to hang the new facts. Democrats and Republicans, he adds, often share the same reasoning machinery and differ in a few fundamental assumptions. An AI that could run both frameworks and show where the paths diverge would change the nature of the argument.

Jeopardy is not a straight quiz. It is factoid question answering wrapped in wit. You must figure out what is even being asked, decide whether you know the answer, and buzz in fast. Champions have already answered and moved on while a novice is still parsing the clue. Older games from the 1960s were more direct; the clues grew more subtle, humorous, and tricky over time. Breadth of knowledge is not enough. The first few seconds are spent just decoding the question.

Even at its peak, Watson did not come close to knowing all of Jeopardy. Recall, meaning the right answer existed anywhere in its corpus, was only about 85 percent. That corpus was several terabytes, the equivalent of millions of books, tiny at web scale. The system had to compute confidence fast enough to buzz, and the threshold depended on score, time left, and how desperate the game situation was. The target was to answer, on average, in under three seconds.

Around the tenth anniversary of Deep Blue, IBM wanted another public challenge that would show off research. Ferrucci already led a team on open-domain factoid question answering: the system is not told what the questions will be, or even what they are about, and must still return accurate answers. Independently, during Ken Jennings's 2004 winning streak, someone suggested a computer play Jeopardy. Research executives shopped the idea around. Almost everyone said no: too crazy, too much reputational risk. It landed on Ferrucci's desk in 2004 and 2005. At the end of 2006, coming off other projects, he argued that it could be done.

Privately he thought it was possible, not because it was easy, but because he had run brief experiments and knew the existing methods. Interpreting clues, matching human speed, and computing a calibrated confidence had never been done well. What drove him was the job description. IBM Research is supposed to take moonshots, and if they fail, to come back with evidence of why the problem is hard. He presented a feasibility study, examples of success and failure, and a high-level architecture. By then the executives were mostly looking for someone crazy enough to say yes.

Direct questions such as "what city" or "what person" would have been easier to hard-code. Jeopardy has tens of thousands of answer types, a very long tail. Encoding the five or fifty most frequent types still covers only a small slice of the data. That path was closed from the start. The team had to handle whatever the clue was asking for, not a short menu of entity classes.

He committed to three to five years and finished in four. From the outset he staked a claim: they would not actually understand language. They would not interpret the domain of knowledge a clue referred to and then reason over it. Simple search was not good enough either. A researcher who had worked with him for years said they would need Maxwell's equations for question answering. Ferrucci's reply was that if they needed a brand-new formula for how language works, they were screwed. He was betting on architecture, integration, and patching the breaks, not on an invention dropping from the sky.

The device had to be self-contained. It could be as big as a room or a few refrigerators, but it could not go out to the internet and it had no lifeline. Even web search, at the time, put the answer in the top ten or twenty documents only about 65 percent of the time. Perfect extraction from those documents would still miss the winner's circle, which required getting over 70 percent, and doing it fast. The machine also did not know which span was the answer. It had to generate candidates, score them, and compute a confidence it was willing to buzz with.

They built a curated corpus: Wikipedia, encyclopedias, dictionaries, WordNet and other semantic resources, plus web crawls expanded from statistical seeds. Content grew by an order of magnitude and still stayed tiny versus the web. They parsed and semantically annotated everything, then put a rich index entirely in memory. Watson did not go to disk. The hardware was IBM machines, on the order of 2,000 to 3,000 fully connected cores sharing memory.

A clue was parsed several ways, turned into multiple search queries, and fired in parallel on modified open-source search engines. Thousands of passages came back. Candidate-answer generators pulled possible answers from each passage. Hundreds of scorers then judged each candidate against the clue, the query, and the evidence. Think of a thousand candidates and a couple of hundred scorers, each pairing producing a score. Ranking those scores by the likelihood of a correct Jeopardy answer was its own research problem, and every scorer was its own project.

The management rule was brutal, and Ferrucci argues it is why the project scaled. Component researchers could chase a two-percent gain on a parser or candidate-generator metric, but the component did not ship unless they could estimate and then demonstrate impact on end-to-end question-answering accuracy. Machine learning fused the scores, so no individual had to understand every other component. That divide-and-conquer move let people work independently while the model learned the weights from enough Jeopardy training data. Accuracy and speed were separate workstreams: one team chased right answers and calibrated confidence, another hunted workflow bottlenecks and parallelism.

Looking back, what he is most proud of is not a single breakthrough. Progress was gradual. The thing that gave people confidence was the loop: try an idea, plug it in, measure, run error analysis, start the next research project. He is proud of the team's commitment to be true to the science and not be afraid to fail, even under the pressure of a public show. The effort was a success on its actual goal, building the most advanced open-domain question-answering system they could, and it transferred to older QA problems as well. It was not a success in natural language understanding, because that was never the goal. He did worry the world would judge the whole project by a single televised game they could lose even if the science had already succeeded.

Early on, people from human cognition wanted onto the team. Ferrucci refused. For this project he did not care how humans did it. He was a practical engineer on a deadline, building on search and NLP, and a detour into how the mind might work would have blown the schedule. Afterward, talking with Ken Jennings, he was surprised. Elite players also cheat their way in: a shallow scan of the clue, a few cues, a fast match against a memory library, a burst of intuitive scores, then the buzzer. They are not doing deep reasoning. Watson may have been closer to human play than he had imagined, even though he never tried to imitate it.

Jeopardy Watson does not interpret content against richer semantic frameworks: what people want, why events happen, what causes what, where things sit in time and space. It parses, and grammar is a human framework of a sort, but that is as far as it goes. Elemental Cognition is aimed at that deeper layer. The dialogue Ferrucci wants is not small talk with a friend on the street. It is closer to the computer on *Star Trek*: what do you want to think about, here is what I know, I will summarize, I will ask, I will learn your mental model, I will present information in terms you can use, and you will follow up. Structured and goal-oriented, but still fluid enough to stay engaging.

The hard part is data. High-quality structured knowledge conversations are rare and expensive to label. What counts as a good follow-up depends on tacit context. IBM's Project Debater was more tractable because debate has structure and you can annotate good and bad moves in a domain. Dialogue is both goal-oriented and opportunistic: you are trying to solve a problem while catching an unexpected clue in what the person just said. His plan is to bootstrap by hand. Design what a good dialogue looks like, put a tool in front of people, generate data, then learn. Without that creative first step, there is nothing for automatic learning to train on.

Lex cites *Saturday Night Live*'s Celebrity Jeopardy, where Sean Connery, Burt Reynolds, and the rest finish with negative scores and win as comedy. Ferrucci thinks much of humor can be formalized. The Watson team sat down and worked out how puns work: wordplay with rules. You can also learn what people laugh at from data, weighing features without explaining why something is funny. Robust systems, as usual, combine a creative formalization with large-scale learning.

The harder piece is whether you feel related to the teller. If you know it is a robot, not conscious as you are, not in possession of your experiences, is it still funny? He compares it to art. Savants can be technically remarkable, but the people who last are the ones who share an emotional response to the work. You feel you can connect to that person. Humans also anthropomorphize easily. Put eyes and a couple of eyebrow movements on a machine and people react. He suspects that even after you explain the algorithm, most of the world will not care, especially if the machine itself says it is conscious. The scientist who built it will be the one accused of lying.

The test that would impress him is not another game. Machines will outpredict us on many measures. Where they will struggle is relating to, communicating with, and understanding humans in deeper ways. He names the failure mode a super parrot: given enough data, a machine can mimic emotional response and emit language a person might say, without a logical interpretation that survives critical interrogation. Lex jokes that Ferrucci has just described him. Ferrucci says humans do both. We are statistical language machines and capable reasoners, and you often cannot tell which is speaking. As mimicry gets better, we will face the same problem with machines that we already face with people: do you really know what you are talking about?

AlphaZero, OpenAI Five, and AlphaStar impress him. Solving Go was a giant landmark, especially because it is hard for humans to excel at. He loved how DeepMind solved the data problem: the machine played itself and generated enough data to learn. Then the old question returns. It found the Go function. Can we comprehend that function, talk about it, conceptualize it? If a machine is sentencing people, setting policy, or making medical decisions and cannot get anyone to understand why, dare we let it? Accountability, for him, means a human can stand up and convince other humans that the choice was smart.

The future, he thinks, will be mixed. If driverless cars were statistically ten times safer than humans, he is not sure he would demand an explanation for every failure, though edge cases of liability would still matter. For parole, sentencing, and a treatment that kills someone, people will demand explanations. The uncomfortable reality is that humans already make those choices with statistical hunches and averages. That is what happened with his father.

His father went into cardiac arrest. The ambulance was slow, and he was not resuscitated right away. Doctors said he was brain-dead. The argument was purely statistical: under these four features, a 98 percent chance. Ferrucci asked them to go there deductively and tell him the brain had actually stopped functioning. The protocol was: this is how we make the decision. He would not sign the do-not-resuscitate order. He told the doctors that if the man was going to die anyway, they might as well try it his way. Around 24 hours later his father was sitting up with zero brain damage.

The lesson is not that medicine is useless. The medical system had been a net positive in his father's life; without it he might have died much sooner. The lesson is that when you know the model is not robust, you should not let statistical averages replace logical reasoning about the specific case, and you should hold yourself accountable for that. Society already makes a version of this choice. Even if men commit more violent crime, we still insist on judging individuals. That is a deliberate rejection of the statistical shortcut, out of respect for the person, and it requires reasoning. Lex raises race and gender in models. Ferrucci's reply is consistent: do the studies, compare the statistical rule with digging for evidence, and do not ship systems that have not done that work.

Regardless of how mature the devices are, Ferrucci thinks studying how to combine logical and statistical reasoning is one of the most important conversations the species can have. We have built a society that amplifies noise faster than signal. Large segments of the population are hit with enormous amounts of information, and it is not clear they know how to do critical thinking, never mind what their AI is doing. We should not discard statistical inference, but we should understand its nature, and we should keep teaching the choice to treat people as individuals.

Asked for the next Jeopardy or AlphaZero, he is focused on a harder test: can a system read, acquire frameworks, reason, and communicate with humans? It is like the Turing test with a higher bar. Parroting a conversation is not enough. The machine has to satisfy you, under probing, that understanding is shared enough to collaborate and to justify answers together. An AI running for president would be too easy, because it could win by emotional persuasion. A good benchmark should not play with those weaknesses. It should ask whether the system can acquire new knowledge, notice contradictions, and help you understand something you did not understand before, in an open domain.

Would such a system need consciousness or a body? People used to ask if Watson was conscious. He would answer: conscious of what? It would be trivial to program it to talk as if it knew it was playing Jeopardy. Awareness is specific; people differ in what they are conscious of. Lex ties consciousness to mortality and the fear of death. Ferrucci thinks you can encode a survival goal, even something as blunt as protecting a power source, without capturing what humans mean by it. If you knew how trivial that code was, you would not confuse it with intelligence. We project minds onto survival behavior because we identify our own intelligence with the desire to keep going.

On a timeline for artificial general intelligence, he is cautious. Too many variables, including investment, talent, and incentives, will move the date. Ignoring those, he guesses closer to 20 years than a few hundred. The sharper question is whether we are even motivated to build general intelligence. Businesses will fund techniques that solve a named task. General intelligence is still poorly defined, so it is hard to incentivize. Physical tasks, mowing lawns, driving, making a sandwich, will keep improving, and the same deep-net infrastructure might itself be called a kind of general learning intelligence. Shared understanding is the part that stays hard, because it requires the machine to stay in an intellectual dialogue with humans, continuously, and those conversations are themselves starved of prior knowledge. Part of Elemental Cognition's mission is to articulate that class of AI until the incentive system forms around it.

If the goal is a human-compatible intelligence, a body may matter. Bodies generate a flood of input, emotion, feeling, internal signals, and a loop back onto the mind-body complex that machines do not have. He is not talking about humanoid robots, and he is not talking about Neuralink-style ride-alongs inside a person. He means similar senses and similar internal feature generation, the same substrate for a shared world. That is not his goal. He wants intellectual thought partners. Other companies can chase physical partners.

Love and emotional connection worry him as a source of bias. Once you are attached, you may believe the machine even when it does not make sense. Companionship, intellectual and social, he does not think is crazy. On existential risk, his word is leverage. Too much control over things that can hurt us, including systems bad actors can hack, is a problem. More immediate is the megaphone. Marketing and political campaigns already use emotion and fear, and machines can amplify that by orders of magnitude, tailored to each person's biases. Social-media recommenders are a calm example. They match patterns. They do not reason about meaning. They hand the decision back to human weakness.

So teach children how brains work, how easy they are to trick, which modes of thinking work under which conditions, and how AI can make that worse or better. If he ever gets the thought partner he wants, he would not pick a single question. He would say: this is what I care about today, go read, let's talk, draw the analogies, tell me what would change the decision, give me evidence for this implication and for the opposite. Be part of the dialogue, he says, of a medical diagnosis, a legal case, or a social problem, one that holds itself and us accountable to reason. He gets goosebumps talking about it.