From Phonemes to CTC: How Deep Learning Rebuilt Speech Recognition

Adam Coates opens by saying deep learning has played an increasingly large role in speech recognition, and that the field has reached the point where it enables applications that reach real users. He lists captioning lectures and other video, hands-free interfaces in cars that keep eyes on the road, and easier mobile and home devices.

He cites a recent study with Stanford and UW on something as basic as texting by voice. Even with the mistakes a recognizer makes, people could enter text about three times faster by speaking than by typing. The engine used was powered by the deep learning methods he goes on to describe.

A complete speech application has many parts: transcription, word spotting or triggering (a phone that must listen continuously for a wake word), and speaker identification or verification. Coates narrows the talk to the bread and butter, an engine that turns raw audio into words. He calls this a consummate AI task, something people do almost effortlessly and machines have historically found very hard.

Coates spends about ten minutes on the traditional pipeline, because students will meet it in the wild and because it explains what deep learning changes. Raw audio is converted into features. An acoustic model links those features to the words being said, a language model encodes which words and spellings are likely, and a decoder searches for the word sequence that maximizes the product of the two.

Predicting characters directly is awkward because English spelling does not match sound; "night" could be a knight or an evening. So traditional systems predict phonemes, the perceptually distinct units of sound, such as the sounds in "hello". Phonemes simplify the acoustic model but add a lexicon to map sounds back to spellings, and the decoder's job gets more complicated. He mentions TIMIT as a popular dataset with annotated phonemes.

The advantage is that the system is tweakable: a new pronunciation can simply be dropped in. The cost is that, for a newcomer, it is confusing and hard to debug, since it is unclear which model causes an error. Accents, heavy noise and ambiguity make it worse, because working out how to adjust a pronunciation model for an unheard accent is a very hard engineering judgment.

The first impact of deep learning was to replace a single machine-learning component, the acoustic model. Coates describes work by George Dahl and co-authors that put a 2011-era deep belief network, with pre-training, into a state-of-the-art pipeline in place of a Gaussian mixture model. The result was a 10 to 20 percent relative improvement in accuracy, which he calls a giant leap compared with progress in previous years.

Old pipelines had a ceiling: more data and a bigger computer did not help, because the problems lay in how the pipeline was built, so performance petered out. Deep acoustic models raised that ceiling. The next question is whether deep learning can power the engine from audio input to transcription, so that more data, bigger computers and more experiments keep improving performance.

The goal of the tutorial is to get listeners to a point on that curve, after which the problem becomes one of scale. Starter code with a basic pipeline, including an open-source CTC implementation, accompanies the talk. The plan is preprocessing, CTC, training, decoding with language models, then scaling and production.

Audio is a one-dimensional signal, typically 8,000 or 16,000 samples per second, so it can be treated as a simple vector. For preprocessing Coates uses a plain spectrogram: cut a window of about 20 milliseconds, take an FFT, and take the log power at each frequency. He says elaborate features like MFCCs are unnecessary here and predicts preprocessing will eventually vanish as networks learn from the raw waveform.

The aim is a recurrent network, trained on pairs of audio and human transcripts, whose output neurons are read off as characters. The obstacle is that the input and the transcript differ in length, and the ratio changes with speaking speed. The older fix was to bootstrap by aligning phonemes to audio frames, which he calls horribly error-prone.

Connectionist temporal classification (CTC) gives each frame a softmax over the alphabet plus a special blank. A mapping rule then collapses repeated adjacent characters and drops blanks, which is why a blank is needed between the two L's of "hello". Because many symbol sequences collapse to the same transcript, its probability is the sum over all of them, in effect marginalizing over every feasible alignment.

Training maximizes the log probability of the correct transcript over all examples. Coates says a 2006 paper by Alex Graves and co-authors showed that, thanks to the independence assumption, a dynamic programming algorithm computes both the sum and its gradient with respect to the output neurons. This is now a mature technology: Baidu's warp-CTC runs on GPUs, a Stanford implementation exists, and TensorFlow includes a CTC loss.

Difficult data needs a few tricks. Very long utterances early in training can cause underflow, overflow and exploding gradients, so the first epoch sorts utterances by length and starts with short ones, a form of curriculum learning he calls SortaGrad. A model trained this way reaches a better optimum than one without it.

He also recommends batch normalization for recurrent and very deep networks, noting it is built into many frameworks. The demo network is simple: a spectrogram input, one or several convolutional layers, a recurrent layer (vanilla RNN, LSTM or GRU), and fully connected layers feeding the softmax outputs and CTC. It is trained with stochastic gradient descent with Nesterov momentum on Wall Street Journal read speech, with LibriSpeech as a free alternative.

The simplest decoding picks the most likely symbol at each frame and applies the collapse rule. Coates says this is not the truly most likely transcript, because it ignores the many alignments behind each one, and its output often does not look like English. But it is a useful diagnostic for telling whether the network is picking up signal or just producing gibberish.

He shows the outputs evolving. After 300 iterations the network outputs mostly blanks and spaces, the most common symbols because of silence, which at least shows the gradient is not broken. By about 1,500 iterations English-like fragments appear, and near convergence the output sounds out the right sentence phonetically.

The examples show both the strength and the limit of the approach. Proper names such as Cynthia get sounded out, and a French word meaning something like rubbernecking was transcribed phonetically by a network with no knowledge of French. Finding the exact best transcript has no efficient general algorithm, since the possible transcripts are exponentially many, so a generic search is needed.

Spelling errors arise because there isn't enough transcribed audio to learn every name. Coates calls one version the Tchaikovsky problem: a name you have never seen written can only be spelled correctly from text. Text is far more plentiful than transcribed audio, so a language model trained on huge corpora can supply context, for example that a Tchaikovsky concerto is plausible and "trike offski" is not.

He recommends traditional n-gram models, built with a package like KenLM, for production. They can be updated quickly in parallel on a cluster to track trending words, and probabilities can be looked up from a table inside the decoding loop. He hopes a neural network will replace them someday but calls n-grams best practice today.

Decoding uses beam search with an objective that combines the acoustic probability, the language model probability raised to a power, and a word-length term to offset the penalty on long transcripts. Candidate prefixes are extended character by character, and only the top few survive each step. With a language model, "Boston" with an O beats the misspelled version, and a neural language model can rescore an n-best list, for instance preferring "wine and port" over "wine and pork".

With the basics in place, Coates turns to scale. Transcription costs roughly 50 cents to a dollar a minute, and public benchmarks hold hundreds to thousands of hours. The key is to match data to the application: engines trained on quiet read speech do well on the Wall Street Journal but not in a crowded cafe, and conversational speech involves disfluency, stuttering, echo, microphone quality and accents.

Read speech is cheap, often under ten dollars an hour, so the team engineers it. Reading movie scripts or poetry makes people voice-act with more expression. To elicit the Lombard effect, where people raise their voices in noise, they played loud noise in speakers' headphones.

Most powerfully, they synthesize robustness. Clean read speech is mixed with cafe noise or reverb using off-the-shelf audio tools, and for Deep Speech about 10,000 hours of audio was combined with freely available noise tracks to produce hundreds of thousands of hours of unique audio. Coates's advice is that engineering the data pipeline is easier than engineering the engine, and performance improved steadily with data up to about 10,000 hours.

More data only helps if the model is big enough, which means more computation. Coates's estimate multiplies connections, unrolled frames, utterances, epochs and a factor of three for forward and backward passes. For Deep Speech this comes to about 1.2 times 10 to the 19 operations, roughly 10 exaflops, or about a month on a Titan X.

That is tolerable for a final run but not for exploring new models, so the answer is data parallelism. Speech allows large mini-batches, up to a thousand or two thousand utterances, and a server with eight Titan GPUs brings training to under a week. Baidu has used synchronous SGD with optimized all-reduce code and scaled to 64 GPUs with linear speedups.

He urges teams to compute the theoretical speed of light for their network and be suspicious if their code is far slower. In November the team found edge cases in cuBLAS at certain mini-batch sizes that cost a factor of two or three. He also advises sorting utterances by length so mini-batches contain similar lengths, avoiding zero-padding that can waste a factor of two or more, and says jobs can reach two to three teraflops on a five-teraflop GPU.

One of Coates's favorite results is from the largest model, trained on a large amount of labeled Mandarin. After error analysis with humans debating ground truth, the Deep Speech engine had a character error rate below 6 percent. A single person listening to the same audio did worse, at almost 10 percent.

Giving humans an advantage, a committee of native Mandarin speakers on a fresh test set scored about the same as the engine. Committee members sometimes looked at the engine's transcription and conceded it was right, since it knew an obscure word or place. He sees this as where all speech systems are heading.

In Q&A, Coates admits ground truth is inherently ambiguous and relies on teams of labelers working with the engine. He says CTC's independence assumption adds bias, especially for English where characters depend on each other, but more data and bigger models can compensate. The 20 ms spectrogram window comes from years of experience in the traditional speech community.

In production, users care about latency and cost as well as accuracy. Many research networks are bidirectional, with a backward pass through time, which means the model must wait for the end of the audio before responding. A forward-only model loses some accuracy, and CTC only partly learns to delay its outputs on its own.

The fix is model engineering: let the neuron that outputs a character see a few future frames without a full backward recurrence, bounding latency while keeping context. For cost, he notes that GPUs are inefficient if they process one audio stream at a time, because the work is matrix-vector multiplication.

Baidu's answer is to batch small packets, about 100 milliseconds each, from different users into shared matrix multiplications. A server can support perhaps 10 to 30 users in parallel, and as load rises the GPU naturally batches more, so efficiency improves, with batches rarely under four in practice. Coates concludes that end-to-end deep learning is mature enough for production and that data and compute will matter even more.