How IBM Watson Won Jeopardy: A Moonshot Built on Engineering, Not a Breakthrough

Ferrucci describes Jeopardy as a game of factoid questions, ones whose answers few people would dispute. Players first have to work out what a clue is asking, which is harder than it sounds because clues are witty, playful and sometimes tricky. He notes that the early games in the 1960s had more direct clues, and that over time they became more subtle and nuanced.

After parsing the clue, a contestant must decide whether they know the answer and buzz in before they are sure, often by superficially judging whether the topic is familiar. Watson didn't come close to knowing all of Jeopardy. Its recall was a limiting factor: with a corpus of several terabytes, the equivalent of millions of books, the answer could be found anywhere in it only about 85% of the time.

So the system had to compute a confidence quickly, effectively answering the question and judging its own answer before deciding to buzz. That decision depended on the game state, because with little to lose Watson could buzz with lower confidence. It tracked the competitors' money, the time left and the standings. The team targeted answering in under three seconds on average, with a shallow confidence estimate first and more time to answer after buzzing.

Ferrucci says IBM wanted another public challenge about ten years after Deep Blue. He already had a team on open-domain factoid question answering, which he calls a personal passion. Around 2004, when Ken Jennings was on his winning streak, someone suggested a computer should play Jeopardy, but senior people called it crazy and refused to risk IBM's reputation, again in 2005.

By the end of 2006 he had more time after finishing other projects, and he argued that it could be done and that it would be crazy not to do it. His private confidence was that it was possible though many thought it impossible. A brief experiment and his knowledge of the area told him it would be hard to interpret questions, answer quickly and compute accurate confidences, but the technologies should work.

His deeper argument was about what research is for: a senior leader at IBM Research is supposed to push active problems to the limit, and if the attempt fails, to explain why. He backed this with a feasibility study showing where the approach succeeded and failed and a high-level architecture. The executives, he says, mostly needed someone crazy enough to say yes after years of no.

Earlier approaches had failed because Jeopardy clues are hard to interpret and cover a very long tail of answer types, tens of thousands of them. Encoding the five or fifty most frequent types would cover a small share of the data. The team found an approach to this problem that looked promising and improved it throughout the project.

Ferrucci committed to finishing in three to five years, and it took four. He also decided at the outset that Watson would not truly understand language, interpret the question and reason over a domain model. But simple search wouldn't give a single confident answer either, so he committed to doing whatever works and reporting honestly to the scientific community what did and did not.

He believed fundamental NLP and machine learning were adequate and the question was how to enhance, integrate and advance them. A researcher once said they would need Maxwell's equations for question answering, and Ferrucci replied that if they needed a fundamental new formula, they were screwed. He wasn't counting on a brand new invention, but on a good architecture, then finding where it broke and making the necessary advances.

One constraint was that Watson had to be a self-contained machine, with no internet lifeline. Even using web search, the answer appeared in the top 10 or 20 documents only about 65% of the time, and that assumed you could extract it and be confident. Winning needed more than 70% accuracy, with confidence and speed.

The team did source research to find a body of knowledge small but broad enough, including Wikipedia, encyclopedias, semantic resources like WordNet and some web crawls, then expanded it statistically by an order of magnitude. They parsed and semantically annotated this content, then built a rich, fast index held entirely in memory, so Watson never went to disk. The hardware was IBM machines with close to three thousand cores.

At question time, Watson analyzed the clue several ways, generated multiple search queries in parallel, and retrieved maybe a thousand to five thousand passages. Other algorithms extracted candidate answers from the passages. Teams worked continuously on better query generation, question analysis and candidate generation, while a separate team analyzed the workflow for bottlenecks and parallelized it for speed.

With thousands of candidate answers, each was scored using the question analysis, the query, the passage and the candidate itself. Researchers produced hundreds of different scores, so a thousand candidates led to hundreds of thousands of values to rank by the likelihood of being correct. Ferrucci likens it to a human reading a passage and judging how likely an answer is given that evidence.

Ground truth came only from the right answers to the questions, so the team always drove end-to-end performance. Anyone proposing a better parser, type estimator or candidate generator had to show the improvement on their own metric, then show how it would affect question-answering accuracy. If they couldn't estimate or demonstrate that, it didn't go in. Lex calls it the best-run AI project he has heard of.

There was no single breakthrough, Ferrucci says; confidence grew gradually through repeated cycles of building, error analysis and new projects. The key idea was that components didn't need to understand each other. Machine learning combined all the scores, trained on enough Jeopardy data to weigh them, so people could work independently while the model did the fusion.

Asked what he is most proud of, Ferrucci points to his team's commitment to be true to the science and not be afraid to fail, despite the pressure of a public event. He considers the project a success against its goal: build the most advanced open-domain question answering system, which did dramatically better on the old problems and also won at Jeopardy.

He worried that the world would be slow to understand it, since it came down to a single game and he knew statistically they could succeed technically and still lose that one. He is clear it was not a success in natural language understanding, which was never the goal. Lex responds that it was a success as an inspiration for later challenges.