Nine of ten colleagues said far-field speech was impossible

Lex Fridman opens with the film *Her*, in which a person falls in love with an AI voice, and asks whether that purely voice-based connection is within reach. Rohit Prasad, vice president and head scientist of Amazon Alexa and one of its original creators, does not answer as a film critic. He notes the surge of assistants across ordinary life and says some of the *utility* shown on screen is absolutely within reach. A deep romantic bond is a different question, and he declines to collapse the two.

He then changes the frame. Assistants should be valued for what is superhuman, not only for what is human-like. A person can exist in one place; Alexa can be on a phone, at home, and at work at the same time. Human memory is limited and fuzzy; an assistant’s memory can be near-infinite and precise. Scoring the system only against a person, he argues, lowers the discussion. Fridman objects that users still feel a little creature in the room, not the whole infrastructure. Prasad agrees that people anthropomorphize — and still wants the comparison widened.

What people actually want depends on the moment. Walking through a house and saying turn on the lights is not a human-like conversation, and Alexa should not start chatting; it should complete the command. “This is not human-human alone. It is a human-machine interaction.” Some situations demand humanity; some demand machine-ness. Alexa can be companion, friend, assistant, or adviser, and the boundary, he says, is the context of the request.

Fridman raises the Turing test and its critics and asks how Prasad would define intelligence. Prasad does not hedge: the ability to converse is a sign of ultimate intelligence, “no question about it.” Humans collect data through sensors and decide with a brain; machines have analogues of sensing and acting. Where they already pull ahead is computation, near-infinite memory, and unambiguous retrieval — a machine does not forget who it saw.

Intelligence, in his account, is conversing plus reasoning over world knowledge and sensory input. It can also go beyond human-level, given what machines are becoming capable of. Asked what would impress him outside Alexa-as-product, he still points to Alexa’s conversational capability as a great test, while noting that AGI remains quite far off. He does not want the test limited to parsing an intent or extracting entities. The object is human-machine dialogue.

Self-driving cars and game-playing systems like Go and chess clearly require data-driven intelligence. Prasad still ranks them below open conversation. In games the end goal is defined and the state is known. In dialogue the goal is often undefined, you may not know what state you are in, and you can change your mind midstream — as when planning a weekend and swapping one event for another. That branching, he says, makes human-machine dialogue the ultimate test for intelligence.

Alexa Prize is a grand challenge thrown to universities: build a social bot that can converse coherently and engagingly for 20 minutes on evolving topics. Talking to a stranger — or even a friend — for that long is, Prasad says, super hard. Two years were complete at the time of the interview: the University of Washington won the first, a University of California team the second. A third year with ten teams was underway, after Amazon had spent the early contests building infrastructure so campuses could ship experiences and do research.

Failure is not only falling short of 20 minutes; quality matters. Before the finals, teams talk to millions of Alexa customers. In the lab, actors interact with the bots and three expert judges score them — two of three must agree the conversation “got it” to clear the 20-minute bar. In the field, customers answer a simpler question: how good was the experience, on a scale of one to five. Quitting mid-conversation is itself a signal of where the bot lost the room.

Fridman compares it to the DARPA autonomous-vehicle challenge, where nobody finished year one and a few finished the desert in year two. Prasad says this problem is harder. They are not close to a 20-minute coherent, engaging conversation. His horizon is five to ten years. Progress is still immense: response quality is rising, and humor — real humor, not goofiness — is starting to appear. Bots are learning when to inject a joke, how to keep talking when they do not know a domain, and, under the 20-minute clock, clever ways to mask missing facts.

Fridman asks whether contestants push edgy discourse — cursing, argument as a way to pull people in. Prasad first restates why the prize exists. Research talent was flowing into industry. Academia risked losing three things industry has: lots of data, massive computing power, and a clear way to test advances with real customer benefit. Alexa Prize bundles all three, which is why he calls it one of his favorite projects at Amazon. A side effect: saying “Alexa, let’s chat” became one of the most-used skills in the store.

There are guardrails. A sensitive filter goes beyond keywords into context, and some topics are off-limits because these are convenience devices in homes, often around children. The competition is limited to universities so that campuses keep a place to invent, and so graduate students and faculty have a reason not to all leave for industry. Customers who opt in are told they are talking to a contest social bot. Some come to help, some to play, and some to probe adversarially what the bot understands.

What does it take to win? Prasad’s diagnosis is blunt: systems still do more lookup of intents than reasoning about the conversation. If the topic is last night’s game, the bot has to understand the entities in context — how Tom Brady actually played — not dump a fun fact and call it intelligence. A lot of the time, he says, the system looks up a fact and returns something close enough, “but not really the answer.” That gap, true understanding and reasoning, is where the research has to go.

Researchers are used to a clean annotated corpus and then algorithms. That is not how Alexa Prize works. Teams must learn from live feedback: a one-to-five rating of how likely you are to talk to the bot again, open-ended comments, and the moment a user quits. “This is not a DARPA evaluation or NSF-funded study and you have a nice corpus.” It is the real world, at a scale he finds beautiful, and the optimization target itself is still a research question.

Fridman quotes Prasad as saying the only way to make a smart assistant really smart is to give it eyes and let it explore the world. Prasad says the line was pulled from a philosophical discussion about learning, and he still stands by the gist. Humans are not trained by being fed a ton of text and audio; they learn by experience, by seeing, and by being taught — and they are efficient. Machines, especially these assistants, are inefficient learners. The next wave is not “less data” in the sense of having little to learn from. It is learning faster: weak supervision, unsupervised methods, transfer from one domain or language to another. Other senses matter in that regime. Humans already learn from noisy data; that part should not change. The learning process should.

Embodiment, he argues, is the wrong obsession. Alexa already lives in third-party robots, microwaves, cars, toothbrushes, and Amazon’s own devices. It is a virtual assistant, superhuman because it is in multiple places at once. “The actual embodiment in some sense to me doesn’t matter.” Voice is essential but not everything — there are now screens — and being able to tell when it is Alexa versus not will matter for attribution. Identity of the AI, he thinks, is going to be crucial.

Close your eyes: what would make you recognize Alexa? The U.S., U.K., India, and Australian versions all speak English and still sound different. Recognition is not just the voice actor. Choice of words, tone, speed, how words are enunciated, how terse or lengthy explanations are — all of it is personality. Personalization creates a product bind: a personality for a million people can be tested; the more you personalize, the less you can know it is a great experience. Prasad’s answer is to give customers the control.

Remember This is explicit: you tell Alexa to store something. Prasad has stored his car specs and frequent-flyer numbers. Speaker recognition personalizes once you teach Alexa who is who in the house, and the experience is transparent when a personalization action happens. Music service preference — Spotify, Amazon Music, Apple Music — is set in the app, so “play Sting” routes accordingly. As for a metaphysical backstory, a birth, a family: early. Alexa may answer when it was born; a full historic self has not been decided. Prize social bots, he suspects, may grow identities first.

Fridman, drawing on psychology, notes that deep human relationships have fights, tension, and push-pull. Does that flavor belong in Alexa? Prasad names the missing ingredient: trust. Tension can be healthy; what is non-negotiable here is trust. “The bar to own customer trust for AI is very high in some sense, more than a human.”

It is not only privacy and personal data. It is daily actions: consistency, and how accurately the system understands you. Call a person about a broken internet connection, fail to be understood, and you lose trust immediately. With an AI that example is amplified by a factor of ten. Expectations swing both ways: if you expect brilliance, clumsiness makes you angry; if you expect a simple tool, a too-clever move also offends.

Prasad lives both sides. As a researcher he is occasionally frustrated that the bar is so high; as a customer he says it absolutely has to be. Society will spend years asking whether we hold AIs to a different standard for mistakes than we hold humans. Fridman, who works with autonomous vehicles, agrees the bar placed on these systems is unreasonably high — and Prasad treats that as both a challenge and a reason he still has a job.

Privacy, Prasad says, has to be designed from the beginning around two principles: transparency and control. On the first Echo, a light ring turns on when the wake word is heard and audio is sent to the cloud. A physical mute button cuts power to the microphones. Customers can inspect every recorded utterance in the app and delete it; later they could say, “Alexa, delete what I said today.” A further flag lets users keep human annotators from reviewing their data — a control aimed at the supervised-learning pipeline that needs labels.

Is Alexa listening? “Alexa listens only for the wake word on the device.” You pick one of Alexa, Amazon, Echo, or Computer. The cat-sweater anecdote — talk about cat sweaters, then see ads — is not, in his account, the device eavesdropping. Maybe it is winter and sweaters are popular. “We are not that unique either.” People still question the explanation, and Prasad says that is fine; Amazon has to keep earning trust, because customer obsession is the leadership principle he puts himself under as a researcher.

Fridman, as a machine-learning practitioner, hopes customers will someday choose always-on listening, because with control and trust more data makes the product more useful; he worries people are more sensitive about data than the life improvement warrants. Prasad is measured. Society is getting more comfortable sharing data for benefits, but not everyone is, and one size will not fit all. Two existing features show the pattern: follow-up mode reopens the mic so you can add eggs, milk, and bread without repeating the wake word; Alexa Guard, when you leave home, listens for events like a smoke alarm or breaking glass. Explicit permission in exchange for high-utility behavior.

Amazon’s process is Working Backwards: write the press release first. The inspiration was the Star Trek computer; the launch still needed a starting problem. When Prasad joined in April 2013, that problem was far-field speech recognition — talking to a device from across a room. In the research world he had come from, it was considered unsolvable. First came wake-word detection: “Alexa” is a common word, phonetically close to “I like you” and “Alex.” The device must tell whether the word is addressed to it. “I love my Alexa” and “I hate my Alexa” both contain the name and should not wake the speaker.

Fridman jokes that this podcast will say “Alexa” a million times. Prasad says the device may wake a few times. They have techniques, including watermark-like filters for ads and media, to tell human speech from playback. It is not fully solved, and they care a lot. After the wake word comes large-vocabulary recognition from 20 feet away, with a son playing music and a daughter running around — not a close-talking mic on a table.

Neural nets were only just showing promise in speech. The team was a startup inside Amazon and did not bet everything on deep learning on day one. The first job was data. Far-field training data did not exist, because there were no customers yet. They invented ways to collect it. Then they doubled down on deep learning and on distributed training so training time scaled linearly with data. Combining that data, deep learning, and available compute, they cut error rates by a factor of five within six months of having the data. That was when Prasad got the conviction it would work.

The speech-recognition team was six people. Prasad’s first meeting had about ten. Nine of ten thought it could not be done and wanted to tell management to switch to telephony speech for customer service. Prasad’s eyes lit up: speech researchers had been looking for a killer app, and this was it. They had to plant stakes for what “magical” accuracy would mean with no users to measure against. Echo launched in November 2014. “You really have one chance at this.” If they had launched below the bar, he does not think the category exists. Scientists on the team have since adopted Working Backwards — write the thesis first, leave the results section open.

After recognition came multi-domain natural language understanding. A narrow domain was doable; music, information, alarms, timers, and smart home together, with no customer base, was another order of magnitude. The industry still leaned on rule-based grammars. Amazon went statistical first: language understanding, slot and entity recognition, and an internal classifier were statistical. They then built deterministic matching to fix bugs the models produced. Pure statistics were not enough; pure rules were not either.

Entity resolution was a live problem. “Play songs by the Stones” might mean the Rolling Stones or Stone Temple Pilots. Prasad wanted the high 90s resolved without extra questions, but it is fine for Alexa to ask, the way a person would. Early Alexa was transactional — short utterances, one action. At launch they shipped on the order of 13 big domains; by the interview, Alexa had 90,000-plus skills. Music, information, smart-home control, and timers were the early winners. Hunches — you went to bed and left the garage light on — were already appearing as a more proactive layer.

At re:Invent that year Amazon launched Alexa Conversations. Developers used to author entire dialogue flows by hand, IVR-style: if the customer says this, do that. The new path is to provide sample interactions with your API, and a recurrent neural network trained on that data constructs the dialogue — no dialogue code. The deeper unlock is latent goals. “What movies are playing nearby?” might mean buy tickets, idle curiosity, or checking whether a film is still in theaters.

After a four-ticket purchase, Alexa can offer a nearby restaurant without re-asking party size and time — that information is already in the session. Cognitive burden shifts from customer to machine: anticipate the goal and take the next best action, including switching skills if the user says get an Uber. Short-term memory inside a session is in use. Long-term memory — which restaurant you prefer next week — is still an open problem, and Prasad calls these early days.

Fridman files a bug: he wears the same black shirt and does the same things, yet Alexa keeps playing a cover of “Stairway to Heaven” instead of Led Zeppelin. Prasad logs it as an official report. The larger point is self-learning. Alexa auto-collects millions of utterances with no human supervision, using behavioral signals: playing the wrong song, saying that is not the song, or repeating the request. Station names confuse M and N, especially with Prasad’s Indian accent; the system starts correcting those patterns automatically. Some users want an acknowledgement that the correction landed; others would be annoyed. Teachable AI, he believes, will only get bigger in the assistant space, and this loop is fully unsupervised.

“More natural” also means neural text-to-speech that sounds human in texture, timing, and tone, validated with listening tests. The harder problem is discovery. With some 95,000 skills, the old pattern — ask a named skill to tell you X — imports an app metaphor into voice. Voice should be a seamless surface. A deep-learning ranker shortlists skills for “get me a car,” uses preferences to choose among ride services, and returns the best response. Lists can be one shot: add cookies, milk, and eggs. Memory should unify calendar, Remember This, and reminders, so “when do I have to go to mom’s house?” just works.

Who owns that integration — skill builders or the platform? Both. Skill quality is enforced through Alexa Skills Kit tools; any developer, including Domino’s, can ship. Routing an underspecified request — “get me a pizza” rather than “open Domino’s” — is Alexa’s job. Zoom out and pizza may be a night-in with kids, not a SKU. That is conversational AI in the real world, Prasad says, where customers get frustrated quickly and the academic NLP problem is not the one that matters.

Challenges stack. First, get the basics right: one-shot transactions must work magically. Completing a night out but failing to turn on a light is unacceptable. Second, reasoning: infer the latent goal from current information and history, then pick the next best action — like a self-driving car, except the hypothesis space is far larger and the destination less finite. Third, proactivity: hunches at any moment, not only reactive answers. Reading frustration from the face or from far-field audio is, he says, a very, very hard problem in affective computing.

Is deep learning enough? Not for the leap he wants. Nuggets he likes include learning with less labeled data, zero-shot and one-shot, active learning, and transfer across domains and languages. Matching deep learning to those recipes is underway. The next wave needs reasoning. The good news: prediction-based methods have not been exhausted for accomplishing customer tasks. For the high-quality experience Prasad is passionate about, reasoning has to be solved to the same extent speech and intent already have. He calls it the hardest problem because the hypothesis space is huge — and it explodes further once memory leaves the session. The head of the request distribution is clustered; the long tail, and combinations among tens of thousands of skills, are not.

Fridman asks for Working Backwards press releases at five and forty years. Five years is as far as Prasad will think in this field; five years earlier, Alexa had already surpassed his imagination. His five-year claim: the line between goal-oriented dialogue and open-domain chat will close. Buying batteries is one shot; buying a camera is research. “Alexa, find me a camera” is not clearly a transaction or an exploration. Fridman admits he has never bought anything on Alexa he had not already bought on a desktop. Even a finite shopping goal has a huge space of attributes, on-site and off-site reviews, and forums. “What’s the weather in Cape Cod this weekend?” hides a purpose. Define success as completing goals in the fewest steps, and the distinction between task and chat goes away.

Forty years will not finish it. Even limited to human intelligence, he says, we are quite far: from sensing to neural processing to how the brain stores information, we still do not know how to represent knowledge. The first five-year transformation is already done: asking Alexa for weather, a reorder, or a favorite song feels more natural than picking up a phone. The next five years: plan a weekend, a meal, or a night out with seamless effort. They will work on these problems forever, and he will not pretend a roadmap will solve all of it in five years.

Fridman asks how it feels to sit at the center of a conversational agent in the home. Prasad says privilege. He spent years in the field in Cambridge and, five years before this conversation, almost left, because he could not find a killer application customers would love rather than merely find nice-to-have, and lab work was not landing. Seeing millions and billions of people use it is fulfilling, and it is still early — another 20 years of work he loves. Launching Alexa in 2014 was a first; launching Alexa Skills Kit was “democratizing AI,” the first real SDK for speech and language. Planning a night out with an agent is now in the realm of possibility, not impossible. Papers are no longer the only payoff; customers using the work is. He does not think there is a better place to be.