From Bag of Words to Attention: Quoc Le Teaches Sequence-to-Sequence Learning Through Email Replies

Le frames the talk around a personal problem: returning from vacation to 508 unread emails, many of which only need a yes or no. He proposes building a system that answers such emails automatically, starting with pairs like "are you visiting Vietnam for the New Year?" and "yes". The first step is tokenization and normalization, separating punctuation from words so the text is easier to process.

Next comes a feature representation: a vector the size of the vocabulary, roughly 20,000 words, in which each entry counts how often a word occurs, with a spare slot for unknown words. Each email becomes a fixed-length input paired with a yes or no output. Le notes an audience member's point that this ignores word order and promises to come back to it.

The classifier is logistic regression with a two-column weight matrix, trained by stochastic gradient descent. He samples an email, then nudges the weights to raise the probability of the right answer, using the log probability because it is monotonic and trains better in practice. At prediction time, a probability above 0.5 for yes means the system answers yes.

To fix the information loss, Le introduces the recurrent network. It updates a hidden state word by word, passing the previous state through one matrix and the new word's vector through another, then applying a hyperbolic tangent. The word vectors can come from word2vec or be randomly initialized, and a classifier on the last hidden state predicts the label.

Training now means updating the classifier, the recurrent matrices and the word vectors, which requires complicated partial derivatives. Le admits he gets them wrong when he derives them by hand and recommends automatic differentiation in TensorFlow, Torch or Theano. The matrices are shared across all time steps, and the sizes of the hidden layer are a model-selection choice balancing under-fitting, over-fitting and compute.

Asked about training time, he says a hidden state of about a hundred should take a couple of hours, depending on how many emails the model sees. He recaps the pieces so far: bag of words, recurrent networks for variable-size input, autodiff, and stochastic gradient descent.

A bare yes or no would lose Le friends, so he wants replies like "yes, see you soon", which means mapping a variable-length input to a variable-length output. The same shape covers translation, image captioning, summarization, speech transcription, conversation and question answering. Drawing on Andrej Karpathy's diagrams of recurrent configurations, he picks the many-to-many form where the input and output lengths can differ freely.

The model reads the input, then an end token, and predicts one output token at a time until it produces the end token again. With 20,000 possible words, logistic regression is expanded to many classes. His first version fed nothing back and did not work well, because the model never knew what it had said in the previous step.

The better solution feeds each predicted token back as the next input, which people call autoregressive. The recurrent network reading the input is the encoder and the one producing the output is the decoder. Training samples an email, a reply and a random position in the reply, then raises the probability of the correct token given the input and all earlier reply tokens.

Le says people usually truncate the unrolled network at around 400 steps because longer sequences are too expensive. Sequences of different length share the same parameters, so the model handles variable lengths without change.

For prediction, the simplest method is greedy decoding: pick the most likely word, feed it back and repeat until the end token or a length limit. A less greedy approach keeps several candidates, such as three, expands each one, and scores whole sequences by their joint probability. Le says the beam grows quickly, so low-probability sequences are pruned, and beam sizes of three to ten work fine in practice.

In answers to audience questions, he explains that out-of-vocabulary words map to an unknown token, that inputs are padded with zeros to make batching efficient for the GPU, and that personalization can be done by adding a user embedding at the start of the input. For the problem where one bad prediction derails the rest, he mentions scheduled sampling, which trains the model on its own sampled outputs so it learns to recover.

This is how Inbox's Smart Reply works in production, he says: for an invitation to a Thanksgiving dinner, the system proposes three answers, such as accepting, saying you will be there, or declining. The three come from the beams, with a heuristic to keep them diverse, and a separate model decides whether an email should get a reply at all. There is no guarantee that decoding terminates, but a well-trained model nearly always does, and a length cap handles the rest.

Le points out that the encoder's final state is a fixed-length vector that must summarize everything, whether the input is five words or a thousand. Attention, invented at the University of Montreal, fixes this. Before predicting each output word, the decoder looks again at all the encoder's hidden states.

Mechanically, the decoder produces a vector the same size as the hidden states and takes its dot product with each one. A softmax turns these scores into weights that sum to one, and the weighted average of the encoder states is fed in as extra information for the prediction. Everything is trained with stochastic gradient descent, and he notes it is implemented in TensorFlow.

In translation, the learned weights concentrate on the corresponding input word: "hola" attends strongly to "hi", and the next word attends to "how". Asked about languages with different word order, such as English to Chinese or Japanese, Le says he does not hand-code alignments, because the weights are learned. With a large data set, normalizing case and punctuation matters less because the model can learn it.

Le gives practical advice for strong results on big data sets. Stack recurrent layers, as the original paper used four layers and people were moving to six, and note that sequence-to-sequence with attention had been behind a couple of years earlier but had since reached state of the art on many translation tasks. For unknown words like "Barack Obama" he suggests word segments or splitting rare words into characters, as in work at Stanford.

Multiplying matrices many times makes gradients explode or vanish, so clip them, for example capping the magnitude at ten. He also recommends GRUs or LSTMs, which add new information to a cell state instead of repeatedly multiplying it, and notes they are already implemented in TensorFlow. Applications he lists include summarization, image captioning from a CNN representation, speech and question answering.

For speech recognition, audio frames are converted to features such as MFCCs or spectrograms and fed to the encoder, but thousands of steps make backpropagation difficult. A pyramid of layers can shorten the input by a factor of eight or sixteen, and the output can be characters. The decoder has an implicit language model, but it must wait for the whole input, which suits offline use better than voice search.

Le is candid that in published results the approach did not match CTC or hybrid HMM-DNN systems for speech, though it excels at translation. Online, block-by-block decoding is possible but, he says, still a work in progress.

Le says the Inbox model he described was trained on English but is language-agnostic, so a multilingual user would need a larger vocabulary, perhaps a hundred thousand words, and training examples in those languages. For translation, training data came from WMT corpora with tens of millions of sentence pairs, and sentences averaged twenty to thirty words. He thinks Google search autocomplete could use something similar.

Several questions point at improvements: adding a GAN loss, conditioning the beam on a user's first typed word, and combining sequence-to-sequence with CTC. On training the output globally, he mentions a Facebook paper on sequence-level training with reinforcement learning that improves translation metrics such as BLEU. Yet humans still prefer the plain next-step-prediction output, so the metrics may not capture what people like.

He closes the first half by calling sequence-to-sequence a form of end-to-end deep learning that needs a lot of data. With little data, he says to split the problem into components, or to combine related tasks in one model with an indicator for which task is wanted. Answering other questions, he suggests that new data can be added by raising the learning rate again, and that an emoji is just another token.

Le sets sequence-to-sequence in a broader context, starting with machine reading: read a Wikipedia page or a book and answer questions, with attention looking back across the text. Humans often answer from memory instead, such as knowing that Barack Obama is the US president, so it may be better to augment the recurrent network with memory. He lists memory networks from Facebook, neural Turing machines from DeepMind, dynamic memory networks and stack-augmented RNNs.

He explains attention as writing the encoder's hidden states to memory and reading them in the decoder, then generalizes this to a controller that can read and write at every step. Most such systems use soft attention over the whole memory, which is differentiable but expensive. Predicting where to read makes training harder, which leads to reinforcement learning approaches.

Another extension adds operations. To answer how long a building survived after being built in 2000 and destroyed in 2010, a network could learn subtraction from many examples, but it is better to give it addition and subtraction functions. Neural programmers assign probabilities to these functions, take a weighted average of their outputs and push results onto a stack, as in two Google Brain papers.

Le tells practitioners they can have impact now with plain sequence-to-sequence, while memory and operation augmentation remain exciting but still work in progress. He points to a blog post on attention and augmented recurrent networks, his own simple tutorials, and the TensorFlow implementation of attention-based translation that people can download and train.

Asked about ambiguity in reading comprehension, such as which Harry a question refers to, he suggests injecting user or context information as extra tokens. On sentence embeddings, he mentions skip-thought vectors, which predict neighboring sentences, and an autoencoder approach that repeats the sentence. On common sense, he says he is deeply interested but has no answer, guessing that multimodal data and unsupervised learning may help.

On how much data is needed, he says larger is better but English-German translation succeeded with only about three to five million sentence pairs. With less, he recommends pretraining word vectors or a language model, and using dropout on the input embeddings or on random input words to improve regularization.