From Word Vectors to Memory Networks: The Building Blocks of Deep NLP

Richard Socher defines natural language processing as the intersection of computer science, artificial intelligence, and linguistics. For him, the practical goal is for computers to process language in order to perform useful tasks such as question answering, while full understanding and representation of meaning remains elusive. He notes that perfect language understanding is in some sense AI-complete because it would require visual inputs, thought, and many other complex capabilities.

He describes the traditional levels of language analysis: speech, phonemes, morphology, syntax, semantics, and discourse. Deep learning has significantly improved speech recognition, syntax, and semantics, but it also skips some levels that traditional NLP treated as necessary. In many cases, deep models do not need explicit morphological analysis or syntactic parse trees to build useful semantic systems.

The difficulty of NLP comes from ambiguity and the need for situational and world knowledge. In the sentence Jane hit June and then she fell or ran, the referent of she depends on the verb that follows. The four-word sentence I made her duck can have at least four meanings, and disambiguating it requires knowledge beyond the words themselves.

Applications range from simple spell checking, keyword search, and synonym finding to medium-difficulty named entity recognition and sentiment analysis. Harder applications include machine translation, question answering over large domains, automated email replies, and spoken dialogue systems. Deep learning is making major progress on translation, question answering, and dialogue, but Richard emphasizes that these systems are still far from human accuracy.

Earlier approaches represented word meaning with resources such as WordNet, which defines words through hypernyms, synonym sets, and a large graph of relationships. Richard explains that such discrete resources are useful for people looking up synonyms, but they cannot capture every nuance of context. They also miss new words such as YOLO or ninja, require constant human labor, are subjective, and make accurate word similarity hard to compute.

The alternative is distributional similarity: represent a word by the neighbors that appear around it. A simple example builds a co-occurrence matrix over a symmetric context window and then uses each row as a sparse, high-dimensional representation. Applying SVD or PCA to that matrix gives a reasonable first approximation to dense word vectors.

Word2vec instead walks through windows in a large corpus and predicts context words from a center word. The objective maximizes the log probability of context words given the center word, using a simple inner product followed by a softmax over the vocabulary. Richard notes that this is not really deep learning because it lacks many nonlinear layers, and the full softmax denominator is expensive, so practical methods approximate it. Training uses stochastic gradient descent over mini-batches of a few hundred windows.

GloVe combines the efficiency of global co-occurrence statistics with prediction by fitting inner products to log co-occurrence counts. It trains very fast, scales to billions of tokens such as Common Crawl, and works well on small corpora because it uses co-occurrence statistics efficiently. Richard shows nearest neighbors and vector analogies, including king - man + woman ≈ queen, and notes that more data and around 300 dimensions gave the best analogy performance. A single vector per word still oversimplifies polysemy, but it is a useful first step for building NLP systems.

A recurrent neural network differs from a standard neural network by tying the same weights across time steps. At each step it combines the current word vector with the previous hidden state through matrix-vector products and a nonlinearity. In theory this conditions the model on all previous words, although in practice optimization tends to capture roughly the last 30 words or so.

The hidden state is then passed to a softmax classifier, where each class is often a word in the vocabulary. Richard uses language modeling to explain why this matters: in the phrase the price of wood, the context makes the noun wood more likely than the auxiliary would. Language modeling is useful for speech recognition, machine translation, and providing a general prior over language.

Language models are evaluated with perplexity, which is based on the average log likelihood assigned to the correct next word; lower perplexity is better. Richard says language modeling is almost NLP-complete in a silly sense, because perfect next-word prediction would disambiguate many tasks. A standard RNN has a problem, however: it modifies its hidden state at every time step, so a long plot description in a movie review can wash out an earlier sentiment cue such as wonderful or great.

GRUs address this by computing update and reset gates at each time step. The reset gate can ignore previous memory and store only the new word information, while the update gate can copy previous hidden states forward and let gradients flow across many steps. Because the gates are vectors, different dimensions can reset or persist at different frequencies. Richard calls the GRU the second most important building block after word vectors.

Most sequence models end with a softmax over classes seen during training. In language modeling, this means the model can only predict words that were in its training vocabulary. Richard gives the example of introducing a new name such as Srini in context: even if the model has never seen that word, a human reader should be able to reuse it. A standard softmax cannot do that.

The pointer-sentinel mixture model combines the standard softmax with a pointer component that can point to previous context words. In an article about Fed chair Janet Yellen, the model may never have seen Yellen in training, but it can copy the name from earlier in the text when predicting later. A sentinel captures the remaining probability mass and routes it to the standard softmax, producing a mixture of context words and vocabulary words.

Richard places this in the perplexity progress of language modeling. In 2010, strong syntactic neural models reached a perplexity of 107, and an ensemble of recurrent networks combined with count-based methods pushed it to 87. Later LSTM-style models reached 78, and the pointer-sentinel model reported around 70, meaning perplexity dropped by more than 10 points in roughly two years.

Richard asks whether all NLP tasks could be reduced to question answering over some input. He shows logical reasoning over facts about Mary, Sandra, and the milk; sentiment classification phrased as what is the sentiment; named entity recognition phrased as what are the named entities; and translation phrased as what is the French translation. A single joint model for general question answering would push the field toward new sequence models and reasoning capabilities.

Two obstacles stand in the way. First, no single architecture consistently gives state-of-the-art results across tasks: memory networks lead on bAbI question answering, tree LSTMs lead on sentiment, and bidirectional LSTM conditional random fields lead on part-of-speech tagging. Second, fully joint multitask learning is hard; NLP usually shares only word vectors, and unrelated tasks can hurt each other through catastrophic forgetting.

Dynamic memory networks tackle the first obstacle with a modular architecture for arbitrary question answering. The input module is a GRU over the facts or text, the question module is another GRU, the episodic memory module makes attention passes over the input, and the answer module produces the final output. Word vectors serve as a simple semantic memory, and the model can take multiple glances at the input when a question is complex.

The episodic memory module is a meta-gated GRU whose gate is computed from the question, the current sentence, and the previous memory state. A scalar attention gate can skip irrelevant sentences, and soft attention sums to one over the facts that seem most relevant. The model can stop iterating when it has enough information if supervised, or use a fixed number of passes otherwise. All four modules train end-to-end through the final softmax and cross-entropy loss.

On Facebook's bAbI dataset, the dynamic memory network reached state of the art for simple logical reasoning questions. The dataset includes yes/no questions, counting, negation, indefinite knowledge, coreference, and temporal reasoning. Richard cautions that bAbI is synthetic: it was generated from human-defined patterns, so high accuracy is necessary but not sufficient for real question answering. Real language will not provide a thousand similar examples for every pattern.

The same architecture also worked on sentiment analysis, a very different task. Richard says multiple passes over the input help for reasoning over three facts or counting, where the episodic memory module may go over the input around five times. For sentiment, however, performance hurt after more than two passes. This motivates work on a single model that can learn different tasks with the same weights.

Richard shows difficult sentiment examples where attention explains the improvement. In the sentence in its ragged cheap and unassuming way the movie works, one pass fails, but two passes learn to pay attention to the movie working rather than only to cheap and unassuming. In my response to the film is best described as lukewarm, the first pass focuses on the positive word best, while the second pass realizes that best describes lukewarm and the sentence is negative.

On a sentiment dataset that had existed for about eight years, standard NLP models had not exceeded 80 percent accuracy, while neural models reached the high 80s. Part-of-speech tagging is less exciting and more of an intermediate task, but the same kind of architecture still improved state of the art on a dataset that had been around for over 20 years. Richard uses these results to argue that deep learning is changing many areas of NLP at once.

A computer vision researcher in Richard's group replaced the DMN input module with a CNN that produces vectors for regions of an image. Those CNN vectors take the place of word vectors and are fed into a bidirectional GRU that scans regions in a snake-like order and backward. Concatenating the forward and backward hidden states gives a representation for each image region, and the rest of the DMN architecture remains largely unchanged. This model achieved state-of-the-art results on visual question answering.

Richard was initially skeptical, so the group inspected the attention scores over image regions. The model learned to look at the bus when asked about its main color, at green areas for trees in the background, and at a man-made structure for a yes/no question. It looked at a boy's arms and the surfboard when asked what he was holding, and it looked at bananas in the middle of an image and answered green, overriding the language prior toward yellow.

The model also handled questions about a cat's tail and stripes, and it correctly answered whether a tennis player hit the ball. Richard describes trying to break the system with questions about what the girl was holding, what she was playing, what she was wearing, and the color of the ground. It answered many of these correctly, including smaller objects such as the ball, but it said yes both to whether the girl was about to hit the ball and whether she had just hit it, revealing that it did not understand arm angles.

In a demo, the DMN correctly classified a negative sentiment sentence and answered questions about the same sentence in Chinese. Richard uses this to argue that deep learning reduces the need to be an expert in every domain or language. The model can also answer part-of-speech and named entity questions, and it answered why men were wearing helmets with safety. The broader claim is that once a model sees enough examples in a domain, it can capture many task-specific patterns.

Asked whether tree-structured RNNs would improve the DMN, Richard recalls working on tree structures during his PhD and notes recent state-of-the-art results for natural language inference. He thinks tree structures might give a slight improvement in the input module. However, the episodic memory module can track subphrases and combine them over multiple passes, so he does not think trees are strictly necessary.

For domain-specific question answering without much data, Richard's practical answer is to create data. If an application cannot generate more than a thousand examples, automating it may not be important enough. Past emails, search logs, and previously answered questions can bootstrap training data, and humans can collect answers for the toughest failed queries. Research on smaller datasets and complex reasoning may help, but a system still needs to have seen a reasoning type a few hundred times before it can learn it.

Asked whether QA systems are robust to false input, Richard says they are not, largely because training never exposes them to such adversarial questions. Adding many such examples would probably help, and this matters for real-world security. He warns that classifiers can be stolen through repeated queries and fooled into misclassifying inputs, so he would be careful about using these systems in security environments today.

Richard also discusses combining text with knowledge bases for reasoning. He mentions work on recursive tensor networks that score whether two entities stand in a given relationship, and he says reasoning over knowledge graphs and fuzzy text has been done separately but not yet combined in a principled way. On interpretability, he says the best current tool is to show attention scores, such as which words or facts the model used for sentiment or question answering, while broader neural network interpretability remains open.