267 Papers in Seven Months: Albert Gu Explains the State Space Model Revolution

Nathan Labenz opens *The Cognitive Revolution* by arguing that Albert Gu's Mamba paper, posted in December 2023 with co-author Tri Dao, was the first architecture to combine subquadratic scaling with wins over Transformers on key metrics. Other projects had already shown cheaper-than-quadratic attention alternatives, but Mamba was the result that made Labenz declare the end of a pure-Transformer era and the start of mixed-architecture designs. He had already recorded a two-hour solo "Mamba emergency" episode and a later literature review with Jason Moe, who tracks the field at statespace.info and joins this conversation.

Moe's spreadsheet, Labenz reports, already lists 267 papers and projects downstream of Mamba, including multiple ways of scanning images and other non-sequential data and experiments with more than one internal state. Gu, an assistant professor at Carnegie Mellon and co-founder of Cartesia AI, is on the show to walk through the intellectual history and the internals of Mamba and the newer Mamba 2. Labenz also flags a misconception he wants cleared up: he had thought GPU SRAM size capped Mamba's all-important state, when the real bottleneck was the speed of the original selective scan.

The conversation will run past 90 minutes, covering long-context training, more expressive mechanisms that might cost more at train time, and Labenz's favorite speculation: multi-state systems in which each state has a job. Gu says he is interested, because that path might improve both performance and interpretability. Commercial work at Cartesia is only touched at the end; Labenz points listeners who want that story to Gu's recent appearance on *No Priors*. He closes the intro with an aside for people still building taste: even Gu has stumbled onto results, reverse-engineered his own models, and still has as many questions as answers.

Asked for an intellectual history, Gu starts with RNNs. Before the fifth year of his PhD he interned with an RNN specialist who later took a faculty job in Switzerland, and the idea of stateful recurrence—a finite state that compresses the whole history—felt like a fundamental computational paradigm. That was conventional wisdom before Transformers. The pendulum then swung so far the other way that some younger researchers barely know how recurrent models work and treat attention as obviously correct. Gu thinks both families have merits; what dominates is often what is popular.

The first major paper in the line was HiPPO, with Tri Dao in 2020. It used older theory on orthogonal polynomials and function approximation to ask how a finite recurrent state can compress history as faithfully as possible. Gu says he did not yet know what a state space model was; HiPPO turned out to be one, derived from a different angle. The name is a backronym for high-order polynomial projection operators. The team wanted to call it HiPPO after the hippocampus, the brain's memory structure, and the paper even includes a small brain diagram with that region highlighted. Gu adds a disclaimer that he barely knows neuroscience; the analogy is loose intuition, which he still finds useful.

HiPPO led to S4, the structured state space sequence model, published across 2020–2022. Those papers made the models trainable by solving computational bottlenecks, and they turned out to be strong on perceptual, continuous signals such as audio and video. Language was a miss; people tried and dropped it. Gu already wanted an input-dependent, time-varying version in the HiPPO days—remnants of those scans still sit in public codebases—but training was intractable. Only after years of simplifying S4 and its follow-ups could the team implement selectivity efficiently. The idea itself is old: it is close to LSTM gating, and close to what makes attention work. Putting RNN ideas, SSM theory, state expansion, and stable training together just took a long time.

A lot of it was accidental. For S4 they had several HiPPO matrix initializations; one formula looked like nonsense in that model, Gu tried it because it was cheap, and it worked better. Two years later, preparing his CMU job talk, more theory showed the matrix had been principled all along. Labenz offers a chemistry parallel from his student days: a seemingly failed experiment, once they extended the trend line the other way, became a real finding. Gu's PhD adviser used to say take big bets and figure out what you want to bet on. Conviction and intuition, he argues, matter more than a tidy process of theory versus experiment.

Labenz's picture of the Mamba 1 breakthrough is a move from "the same computation every time" to a forking, input-dependent update. Gu agrees, and keeps the intuition non-notational. A recurrent model only has its state, so the quality of putting information in and taking it out is everything. Older SSMs applied the same linear transform at every step, which in the simplest view just multiplies the whole state by a constant, shrinking everything, then mixes in the new input. That is close to how LSTM gates worked. Always down-weighting by the same amount feels wrong on discrete, information-dense data. A speech transcript is full of ums that should not occupy a time step; a truly important token should be able to overwrite the rest.

The selection mechanism is the simplest fix that worked: whatever parameter up-weights or down-weights the state versus the input becomes a function of the input. The model can look at the next token and decide to keep its memory and ignore the input, or the reverse. Why this succeeded where LSTMs stalled is the rest of the SSM history. Gu's counterintuitive claim is that nonlinearity was part of why old RNNs failed. People added it because they thought it would make the model stronger; in practice it squashes the state in unpredictable ways and you lose control of information flow. Linearizing almost everything, then reintroducing a little nonlinearity as selection, gave them that control back, along with cleaner initialization and parameterization.

Labenz asks whether there is one shared state or one per layer. Gu's framing: a modern net is a stack of sequence-to-sequence maps. Attention, RNNs, and CNNs all mix along the sequence; LayerNorms, MLPs, and residuals run independently at each time step and have no state. Each Mamba block's inner SSM—the paper called it the S6 layer—has its own state. Any autoregressive model has a state in the precise sense of what must sit in memory between tokens. For an SSM that is a fixed-size recurrent vector that summarizes history. For a Transformer it is the KV cache: every past token, or a representation of it, stored so the new token can be compared pairwise to all of them.

That lens makes the tradeoff sharp. An SSM compresses history into a form the model can read from, the way human memory is fuzzy and powerful and cannot hold a dictionary. A Transformer is closer to an external database that can look everything up, at the cost of a cache that grows without bound. Gu's rough and, he thinks, accurate accounting: Transformer inference memory is token count times model width; SSM memory is expansion factor N times model width, with N a hyperparameter typically around 100 to 200. More compression is fuzzier and faster, and sometimes even helps by forcing the model to be intentional. Diminishing returns show up, but the lever is more natural than deciding how many tokens to keep.

Labenz had assumed original Mamba states were capped by SRAM, then divided across layers down to a megabyte or less. Gu says training is subtler. Classic LSTMs had to materialize the state on every forward pass, which limited size and scale. S4 and Mamba can be rewritten so that cost is hidden in training and the state is materialized again at inference. During training the whole sequence is visible at once, so there is no need to keep a state between tokens. Mamba's hardware-aware algorithm still builds the state, uses it to produce the output, and throws it away, rematerializing it on the backward pass. S4 went further and rewrote the recurrence as a mathematically equivalent convolution—a sequence-to-sequence map with no state in the compute graph at all.

The duality Labenz is reaching for, Gu says, is that a sequence layer is only the map from input sequence to output sequence. The state is an auxiliary that reconstructs history you no longer have. At inference you must keep it in main memory, typically moving weights and state into SRAM per layer like any other block, unless you try older RNN tricks such as parking a tiny state in SRAM to avoid transfers. Efficiency then comes from size: the same shuttle, a much smaller payload. Gains appear in a specific regime—long sequences, larger batches—when a KV cache actually explodes. Gu's faith is that if people spend as much optimization effort on SSMs as they have on attention, more of those tricks will show up, because the models are a different computational object.

Mamba 2's practical motivation was that Mamba 1's core SSM still did not use tensor cores. Gu invokes the hardware lottery: software, chips, and models co-evolve, so the architecture that won early gets specialized silicon, and leaving that loop is hard even if you are sitting in a local optimum. He does not think attention is going away, and he was reluctant to reshape the model just to please current GPUs, but without a multi-year bet on custom hardware there is no other game. Restricting expressivity in one specific way let them compute the same map with matrix multiplies, drawing on linear-attention ideas. Most of the algorithm and model were done in October 2023, before the Mamba 1 paper came out; they were too busy to release it sooner. Gu says the long Mamba 2 paper's theory may be the more interesting half, even if the obvious consequence is a faster algorithm.

Gu still does not know what they paid for the speed. In Mamba 1, if a layer's state is on the order of 100,000 numbers, each scalar decays by its own input-dependent factor. Mamba 2 splits that vector into chunks—say a thousand blocks of a hundred—and every number in a chunk shares a decay. That is strictly less flexible dynamics. On the downstream tasks they had time for, it did not seem to hurt. He can imagine the extra structure helping, the way convolutions' inductive bias helps vision. The filler-token example is the simple thought experiment: to ignore an "um," Mamba 1 must set every scalar decay to one; Mamba 2 only has to set a few chunk-level factors to one.

The task that most intrigues him is associative recall, studied by Simran Arora and Sabri Eyuboglu in Chris Ré's lab: read a dictionary, then look up a new word. Recurrent compressors, like human brains, are bad at it; attention is not. Mamba 1 was already strong among recurrent models. Mamba 2 blows Mamba 1 out of the water on that task, and Gu has only the empirical result. Maybe some "irrelevant" architectural bit is doing the work. Or maybe blocking the state so that a chunk is a concept that evolves together is the right bias for carrying information exactly. "Jury still out" is his phrase. William Merrill's formal-language analysis, as Gu reads it, says these SSMs are no more expressive than other linear models and would need a dense square state matrix instead of a diagonal one to match LSTMs; that speed cost looks unmanageable with current techniques. Gu tried dense input-dependent matrices after HiPPO and could not make them train.

Comparisons are messy because FLOPs, wall-clock, inference cost, and parameter count do not move together. Gu separates two buckets of Mamba 2 changes. The core layer shifts from S6 to SSD, the state space dual, which is the expressivity-versus-efficiency change and the reason larger states train more cheaply: the forward pass mostly avoids materializing state, or only does so between chunks, while each chunk looks like attention-style matrix multiplies. Surrounding architecture—where linear projections sit, extra norms, sequence and tensor parallelism—could mostly have been applied to Mamba 1; they shipped with Mamba 2 because the duality with attention made Transformer systems tricks easier to reuse. In the scaling plots they held parameter count roughly fixed. In that regime Mamba 2 dominates on perplexity and wall-clock. If you only cared about inference, you might still want smaller, more expressive states, because decode cost scales with state size and performance saturates. They optimized for training speed because that dominates research iteration.

Moe asks the Chinchilla-to-Llama question in reverse: pay a training tax for a more expressive inner mechanism if inference then gets cheaper. Gu thinks that is viable, especially after you know SSMs fit the application and you are squeezing on-device decode, and he was hesitant to claim Mamba 2 is strictly better than Mamba 1. He does not expect a big shift in that direction, but it is "not out of this world." Doubling Mamba 1's state roughly doubled wall-clock because the work is elementwise; Mamba 2 shoves more of the growth into matrix multiplies, which scale far better, which is what "larger states" actually means.

Labenz's community read is that hybrids almost always win, with a rough recipe of one attention layer every eight to twelve SSM layers, and that a small-scale search might push the attention fraction even lower. Gu has been saying for a while that the mechanisms are complementary. He is no longer sure a pure SSM can recover attention's skill at dictionary lookup, string reversal, and other exact scratch-pad tasks that also strain human working memory. Attention is a database of everything seen so far. He is not sure SSMs can fully close that gap.

The under-explored SSM advantage, he argues, is raw or lightly processed data. Transformer quality is contingent on the tokenizer, which is a handcrafted compression stage that throws away what the model should not have to store. Attention by definition does no compression, so without that crutch it struggles. Sasha Rush's group at Cornell showed in MambaByte that modeling raw bytes, Mamba beats attention by a wide margin. Humans do not look at characters; they chunk into words. Tokenization does that for Transformers. DNA modeling in the original Mamba paper is the same pattern: modalities that co-evolved less with Transformers, and that have sequential structure, are where SSMs shine. Gu's no-free-lunch point from the S4 era still stands. Attention's reputation as universal is, in his view, overstated.

Asked for a third core mechanism, Gu wants a differentiable, input-dependent chunking or pooling layer—an automatic tokenizer the model learns. He has not seen one, and thinks it would matter because current models waste work in ways that are hard to route around. He leaves the door open that throwing the world's compute at scaled attention might still be enough. On the follow-up literature, he guesses a few hundred papers; Moe's curated count is 267, with probably closer to 500 citations if you include passing mentions of SSMs. Name variants—Zamba, Jamba, and the rest—surprised him. He wanted a primitive as reusable as attention. 56.6% of Moe's entries are vision and image work, which fits Gu's advice: if the input has little order, use attention; scan order is why vision papers care so much about how they snake through an image.

Labenz's favorite research bet is a loss on the contents of the state itself: several internal memories with different jobs, loosely inspired by human memory types, aimed at the missing "middle memory" of an assistant that neither fits in a context window nor lives only in email search. Gu is interested. Once you leave the Transformer regime of caching what you saw and enter SSM compression, the state can be used as an object. It may disentangle concepts the way word embeddings do. At Cartesia they already treat state as an encoding of style. In ordinary systems a speaker ID is a separate vector; if you model speech, music, or video autoregressively, the fixed-size state of a prompt has to carry attributes of that prompt, and you can interpolate to blend voices or image styles.

You can also swap initial states the way people stash KV caches of old conversations, except the objects are fixed-size and much easier to put in a database. The RWKV project, a parallel recurrent effort, has fine-tuned downstream tasks by adapting only the initial state, not the weights. Moe's long-sequence experiments, and a collaboration with NVIDIA, found SSM models extrapolating to about three times trained context, and much farther with a little fine-tuning. Gu suspects training data is the real fix: if you need long dependencies, the model has to see them. Engineering has taken Gemini to million-token windows, which he respects, but those windows are still finite. Recurrent models at least update a state with the entire past in principle. He is not close to Google's Infini-attention paper; many "recurrent Transformer" variants still lack, in his view, head-to-head evidence at the scales where Mamba first matched attention on perplexity as a fully recurrent alternative.

Labenz asks why keep publishing, given that Adept's team going to Amazon showed a competitive training stack is itself an acquisition. Gu's answer is that he is an academic researcher at heart. He got into AI to understand the nature of intelligence, wants humanity to understand it better, and will keep sharing ideas even if some work is delayed. Cartesia and Together, where Dao is chief scientist, are not tightly coupled: Together is infrastructure and serving; Cartesia is betting on SSMs, training models from scratch. They collaborate when it is natural and otherwise operate independently. Labenz signs off after more than 90 minutes, still watching for papers as much as products.