How Alexa Learned to Hear From Across the Room: Rohit Prasad on Far-Field Speech and Intent

Prasad says the inspiration for Alexa was the Star Trek computer, but any product has to start somewhere. When he joined, the product was already in conception, and the first problem to solve was far-field speech recognition, meaning you can talk to the device from a distance. In those days that was not common practice, and in the research world he came from it was considered an unsolvable problem.

The first piece is the wake word. "Alexa" is a very common word with sound units that overlap with phrases like "I like you" or "Alex", and the device has to catch it with high accuracy amid household noise. The challenge is telling real requests apart from sentences such as "I love my Alexa" or "I hate my Alexa", where the word is mentioned but the speaker is not addressing the device.

Host Lex Fridman jokes that this very podcast may set off listeners' devices, and Prasad agrees it might wake a few times. He says Amazon has worked on telling human speech from other sources and has launched technology to filter out wake words in advertising, but it remains an unsolved problem the team cares about. Fridman calls it the world's best wake-word detector in a far-field setting, with devices working from 20 or even 40 feet away.

After the wake word comes large-vocabulary recognition, where audio arrives from 20 feet away in a busy house with a child playing music and others talking. Prasad joined in April 2013, when neural networks were only beginning to show promise in speech recognition. The team, working in a startup-like environment, doubled down on deep learning right away because it knew data volumes would grow quickly if the device succeeded.

The key question was how to scale deep learning. The team did some of the first work in training with distributed GPUs, with training time scaling linearly with the amount of data, through a mix of algorithmic and engineering improvements. This let them train on thousands of hours of speech.

Prasad says that by the time Echo launched, the combination of large-scale data, deep learning progress and the near-infinite GPUs available on AWS made far-field recognition useful to customers. He adds that it is still not solved in general, but it works very well in the settings found in homes.

Fridman asks how big the team was and how likely it was to fail. Prasad says the speech recognition team had six people when he joined and about ten soon after, and nine of ten believed the problem could not be done. Some wanted to ask management to switch to something like telephony speech for customer service calls, but he had far-field experience and says his eyes lit up at the problem.

He describes it as the killer use case speech researchers had long been looking for. Asked about the Amazon practice of writing a press release and FAQ first, he says he cannot divulge exact specifications, but one early magical feature was music, which suited him because he does not like earbuds and finds it hard to be a music fan with a phone.

Prasad says the working-backwards process has spread among his scientists, who love it. A press release is the closest equivalent to a thesis or grant proposal: you write what you expect and what it will change, and leave the results section open. Fridman likens it to writing the paper first and then making it happen.

The first breakthrough, Prasad says, was getting far-field training data when none existed and there was no customer base to supply it. The team also had to decide what magical would mean, since nobody had built this before, and put a stake in the ground for the accuracy and customer experience needed.

The second step was doubling down on deep learning as the way to learn from that data. Together the two cut error rates by a factor of five within six months of having the data, and that gave him the conviction it would work. The team reached a bar where they felt people would use the product.

That bar mattered because there would be only one chance. Prasad says that if the product had launched in November 2014 below the bar, the category probably would not exist. Fridman notes that voice interaction in cars is so frustrating that his own group has used it to collect data on frustrated faces.

Even with words recognized, the next problem was multi-domain natural language understanding across music, information, alarms, timers and other household tasks. Prasad says understanding in a single narrow domain was doable, but across domains it was daunting even then, and since there was again no customer base, there was no data. Alexa has since grown the confusion space by about three orders of magnitude.

At that time the industry still leaned on rule-based grammars to find intent. Alexa instead had a statistical approach from the beginning, with an entity recognizer and intent classifier trained on data, and deterministic matching only as a fallback to fix bugs. His point is that data-driven understanding wins in the end, but it depends on first getting the data.

Early dialogue was transactional, one shot at a time. If a user asks to play songs by "the Stones", the system must resolve whether that means the Rolling Stones or Stone Temple Pilots, using signals from other Amazon assets. Prasad says the goal is for the algorithm to handle the high 90 percent of cases without further questions, but asking "I did not understand you" is acceptable, as humans do, and a useful learning signal.

Alexa launched with about 13 big domains, with music the largest, and he notes it now has over 90,000 skills.