Pause Tokens, Analogical Prompts and Ring Attention: A Tour of the Transformer's Next Upgrades

The first paper is "Think Before You Speak: Training Language Models with Pause Tokens", from Google with a Carnegie Mellon collaboration. Its observation is that a model predicting token k+1 only manipulates k hidden vectors per layer, so the researchers ask what happens if it can manipulate k+10 vectors instead. Unlike chain of thought, the model is not asked to produce any reasoning; it simply gets extra space to process information.

Labenz says he saw this coming after the earlier Stanford "backspace" work, where the model learns to retreat and try again when it is not confident about the next token. He is ambivalent, though: he likes that chain of thought can be read and corrected, whereas pause tokens leave you with improved performance but no explanation, though he notes that chain of thought is not always faithful to the answer either.

The techniques are not mutually exclusive, he argues; you could train a model to pause first and then reason, which might make the reasoning more relevant and faithful. In the paper, pause tokens work best when added in both pre-training and fine-tuning, and at inference they are simply appended to the prompt. The model has not really learned when to pause, so Labenz suggests that GPT-4 could help annotate data showing where thinking breaks belong.

The second paper, "Large Language Models as Analogical Reasoners", has the model first recall relevant examples to a problem and then solve the original question. The authors report that it beats few-shot chain of thought, and it is easier to do. Labenz found it surprising, since with few-shot prompting he feels he is doing his part by guiding the model, whereas here the model must generate the examples with no tool or database.

His explanation is what he calls heuristic recall: the model may load the problem into a high-level conceptual representation and then pick a more canonical, relevant example than a hard-coded few-shot set could provide. He thinks of it as self-RAG, with the model itself as the database, and says the gains appear across models up to frontier ones and are dramatic relative to zero-shot.

Labenz tells how he built a prompt coach for executive assistants using few-shot chain of thought and had considered a thousand-example RAG setup to improve it. This result suggests he might skip that complexity by letting the model recall examples, though he suspects meta-prompting may not yet work because models may not have seen enough prompt coaching in training. As a footnote, the paper uses markdown, and he mentions chatter that markdown suits OpenAI models while XML tags suit Claude.

StreamingLLM, from Meta, extends how much text a model can handle with a surprisingly small change that retrofits onto existing open-source models like Llama, Falcon and MPT. Its starting observation is that tokens attend heavily to their immediate predecessors, little to the middle, and a lot to the very first tokens. The authors hypothesize that because attention must sum to one, the first tokens act as an "attention sink" where spare attention is dumped.

A plain sliding window that forgets older tokens breaks down, but keeping the initial sink tokens and sliding the rest keeps the model coherent, with flat perplexity out to millions of tokens. Labenz stresses this does not enlarge the window's usable memory: the model still only sees its 8K or 32K look-back, so users might experience odd partial amnesia, remembering something and then forgetting it.

Why the sink works at the start but not as the window rolls is unclear. Labenz's best guess is that the model wants something that looks like a beginning, possibly a reflection of how it is pre-trained, and he would like to see a model trained on randomly chopped starting points. The code is on GitHub, so he expects people to apply it to their own models soon.

Ring attention is the deepest technical paper, sitting at the intersection of hardware and algorithms; Labenz relays Stability's Emad Mostaque saying such researchers could knock about 10% off global training compute needs. He cautions that he is not an expert, but explains that it restructures the computation, passing different data between devices, to work around the memory bottleneck. It is not a shortcut or approximation; it is still full attention at the same precision.

He quotes claims that context length scales linearly with device count and that for the GPU-rich, going from 4,000 to 10 million tokens on a 175-billion-parameter model costs about 150 times the training compute. That is a 250-fold context increase for far less than quadratic cost. For the GPU-poor, eight GPUs could expand context eightfold at only twice the cost.

Labenz thinks this matters most at training time, where longer windows allow learning connections between remote parts of a text, like a mystery novel's final reveal. A 10-million-token window could hold something like 100 Great Gatsby-sized books in full attention. He would not necessarily run that at inference, but for science such as DNA interactions it could drive lower perplexity.

He guesses that this could be where models tip into superhuman performance, learning things experts do not know, because humans cannot hold 10 million tokens in full attention. Kollmer adds that he would love to load an entire script and ask about logical inconsistencies, and wonders whether blocks really need to pass key-value pairs around the full ring, since a hierarchical, local scheme might see more at each layer.

Labenz sketches a combined system: much more capable thanks to long-context training, but typically run with a smaller inference window that can be adjusted dynamically, longer when needed and rolling otherwise. Most of the time an assistant does not need old context, but occasionally it does, and the attention-sink work suggests such on-the-fly adjustment is possible.

What he finds most interesting, and says is strongly suggested but not yet here, is a highly compressed historical record for retention. Combining attention sinks, pause tokens, backspacing and a rolling window, the model would carry high-level representations of what has left the window. Rather than every word, it would hold something like a few tokens standing for a whole slice of history that it could expand when needed, a "real self-RAG".

Prompted by a question from Kollmer about whether this is just memory in language or hidden state, Labenz says he would train the model to emit a memory token, by analogy with pause and backspace, representing the conceptual content of past context in latent space. It would be high in meaning but low in exact-wording resolution. He would be quite surprised if something like it had not appeared in six months, and thinks it could be done without architectural changes.

He closes by arguing that such compressed memories would resemble how humans recall things, and that connecting a machine's memory to the raw transcript would be far easier than for people, which is one more reason superhuman performance cannot be ruled out. Add planning and some more scale, and both speakers think the next couple of years could be very eventful.