Trained on a Thirtieth of GPT-4's Tokens, Mamba Beat Transformers on Home Turf

In December 2023 a two-author paper was enough to make Nathan Labenz, host of the usually interview-driven *Cognitive Revolution*, cut a solo emergency episode. He opens with the line that frames the next two and a half hours: neither the human brain nor the Transformer is the end of history. Since *Attention Is All You Need* in 2017, Transformers have dominated language, images, video, and niche tasks that Labenz first noticed while building multimodal products at Waymark. He sometimes calls the chance of a better architecture the hundred-trillion-dollar question — the size of the global economy.

Ray Kurzweil's 1990s exponential charts had human-level compute arriving around 2020 to 2025. Human-level AI showed up roughly on schedule, Labenz says, but it is human-level without being human-like: alien, and in most post-training domains already ahead of the average person. Almost all of that progress has been one mechanism, attention, running everything. A few 2023 challengers looked plausible — Microsoft and Tsinghua's RetNet, Meta's Lili Yu on the hierarchical Megabyte architecture — without actually flipping the table.

The paper is *Mamba: Linear-Time Sequence Modeling with Selective State Spaces*, by Albert Gu and Tri Dao. Gu is an assistant professor at Carnegie Mellon and chief scientist at Cartesia AI; his Twitter bio is simply that he is leading the state space model revolution. Dao is an incoming Princeton professor, founding chief scientist at Together AI (which had raised a $102.5 million Series A), and the author of FlashAttention — the work that sped up exact attention by rescheduling data movement rather than approximating it. Labenz's analogy is deliberate: *Attention Is All You Need* did not invent attention; it stripped away complicating detail and showed the mechanism was enough. Mamba, he argues, does the same for state space models.

Labenz starts not with equations but with a small experiment: what comes to mind when you hear the word *rainbow*? Some people see color. Others smell air after rain. Others load optics, mythology, Roy G. Biv, identity politics, or a personal memory. Seven letters, seven bytes, and the brain taps a high-dimensional associative space that is not encoded in the word itself. The same stimulus is also loaded differently under a mythological prompt than under an optics prompt. Humans treat an input according to context, then work thoughts back down into a couple of tokens of speech at a time.

Transformers, he argues, already run a similar loop. Embeddings climb through dozens of identical layers; the highest-order concepts live in the middle. Anthropic's influence-function work found that small models mostly match keywords, while models in the tens of billions of parameters form sophisticated relationships, located in those middle layers. ROME-style concept editing can rewrite a model's world so that Michael Jordan played baseball, while Larry Bird and LeBron James still played basketball. Dan Hendrycks and collaborators' representation engineering can classify and even steer concepts as vector directions in middle-layer activation space.

The weird part is the homogeneous stack. After the embedding, every block is multi-headed attention, a token-wise MLP, nonlinearity, and skip connections, repeated until the desired scale. Division of labor among layers *emerges* even though the form does not change. The power of attention, Labenz stresses, is a forking path: inputs both flow through the network *and* shape the attention matrix itself, so ChatGPT can explain a rainbow through mythology or optics and get two different answers. Older classifiers apply the same computation to every image. That second path is why Transformers work, and it is what Mamba is about to unlock for a completely different class of model.

The same all-to-all information passing is also slow. Dense attention is quadratic: each new token computes a relationship with every token that came before. Approximations exist and often work, Labenz says, but they still look a little worse than full dense attention. Context windows have grown anyway. A hundred thousand tokens fits *The Great Gatsby*; Claude 2.1 is on the market at 200,000. A few years earlier people were arguing about 2,000-token windows; BERT was 512. Two hundred thousand tokens is still one book, or roughly this three-hour transcript. It is far below a person's daily intake of audio, conversation, reading, and imagery.

Worse, a Transformer has nothing but that window and its weights. It always wakes, Labenz's phrase, with total amnesia. When the context runs out, the episode is discarded. OpenAI's system prompt and ChatGPT custom instructions exist because the model has no mechanism to turn one interaction into durable memory. Retrieval-augmented generation loads logs back in, but from the model's point of view that is still one episode. The Stanford AI Town paper's most interesting move, in his view, was having little bots periodically sweep memory logs and synthesize higher-order memories — still language compressing language, still lossy, still a hack.

That is why today's models make poor companions, shaky long-term assistants, and clumsy agents. They can hold explicitly declared preferences in a prompt or a database. They cannot develop an inbuilt intuition for you, or even for their own history. Labenz can see a Transformer-native path — extra behavioral tokens such as a backspace token, a pause/think token, and hypothetically a memory token, perhaps by adapting the same small-adapter trick that maps images into language space — but he has not seen that paper yet. In the meantime, someone invented a better way.

What state space models add, conceptually, is a third object besides weights and inputs: an internal state that evolves and propagates through time. Each step processes the last state plus a new input, emits an output token, and writes a new state. The old state is let go. That long-lived object is exactly what sounds like a memory. Even before Mamba, SSMs already dominated odd long-sequence tasks such as closing arbitrarily nested parentheses and brackets — one of the few categories where Transformers are not supreme, because the sequences can get arbitrarily long.

Labenz traces the math back to an earlier Gu and Dao collaboration, the HiPPO paper, which compresses long sequences into small states by projecting onto a polynomial basis. The representation is slightly lossy and extremely useful, with attractive scaling. The operational fact that matters is simpler: the state is a fixed size. It changes from step to step; it does not grow. Fixed size plus one new token means each inference step takes the same time regardless of history, and training scales linearly with sequence length. Transformers, even with KV caching, still pay a quadratic bill as context grows.

Historically the problem was that SSMs just were not as good at the language-modeling tasks people actually value. Prior models were linear time-invariant: inputs proceed down an assembly line and get transformed the same way every time. They lacked the Transformer's (and the brain's) multipath influence, so their expressivity stayed limited. They did have a dual computational form — a parallel convolutional form for training and long prompts, and a recurrent form for autoregressive generation — which is why they could scale on GPUs at all. Mamba's bet is that the missing ingredient was the ability to let the computation itself depend on the input.

The dual breakthrough is conceptually small and computationally brutal. Let the transformations applied to state and input depend on the input itself — selectivity. The change in the equations is modest. The cost is that the convolutional form disappears, leaving only the recurrent, one-token-at-a-time path. On GPU hardware whose value is parallelism, that sounds like a dead end. This is where Dao's hardware-aware algorithm design comes in, targeted at the NVIDIA A100: the chip GPT-4 was trained on, one generation behind the H100, 400 watts, thousands of dollars, "half an electric teapot" or about twenty light bulbs.

An A100 has 6,912 CUDA cores. Next to them sits SRAM, also called shared memory: 164 kilobytes, less than a leading LLM's context window if you count one letter per byte, and the fastest memory the cores can touch. Farther out is 40 GB of high-bandwidth memory, where parameters live. Forty gigabytes cannot even hold GPT-3's 175 billion weights. FlashAttention's advance was smarter movement of those parameters from HBM into SRAM. Mamba's advance is more extreme: the model state never leaves SRAM. Parameters stay in HBM and are touched as needed so the computation can change with the input. The state evolves in the tiny fast pool and is never exported unless you intervene.

Two supporting tricks keep HBM–SRAM traffic down. Kernel fusion collapses multiple transformations into one when the linear algebra allows it. Recomputation, used in training, recalculates activations and gradients on the backward pass rather than saving them, spending extra FLOPs to avoid the worse bottleneck of moving data. Naive implementations would be too slow to scale. The code is open source. With that stack, Gu and Dao can support Transformer-like multipath processing even though parallel convolution is gone — and they can finally train.

The language models in the paper are relatively small — 1.5 billion and 3 billion parameters — trained to 300 billion tokens. That is not trivial, Labenz notes, but GPT-4 is generally thought to have been trained to about 10 trillion tokens, thirty times more, and Llama 2 to about 2 trillion. It is far enough down the loss curve to compare architectures. Matched on FLOPs, at a 2,000-token sequence length, Mamba essentially ties the authors' Transformer++ recipe, which they describe as the best Transformer training recipe they know — and one author wrote FlashAttention, so Labenz is not worried about a cooked comparison. Past 8,000 tokens, space opens up: Mamba's loss is lower.

The curves go out to 10^20 FLOPs, still about five orders of magnitude from GPT-4. Scaling laws have been smooth enough that a similar shape holding from 10^18 through 10^20 makes it quite likely, in Labenz's view, that the advantage extends a few more orders of magnitude. They did not test language at million-token length; his guess is that ready-made ultra-long language corpora are scarce. They did test DNA. As sequence length goes to a million tokens, every other architecture they studied drops. Mamba's performance keeps improving. A million tokens is five times the then-state-of-the-art Transformer context window, and this is the first Mamba paper.

Inference throughput wins too. An A100 can run about 64 concurrent autoregressive Mamba-1.4B inferences before it saturates, at higher throughput than a similar-sized Transformer. How the SRAM-resident state will scale past seven billion parameters onto multiple devices is still open work. One more result sits in the paper like a seed: interleaving Mamba with multi-headed attention lowers the loss a bit more. They tried one naive interleaving and it was already visibly better. Labenz reads that as "a lot more where that came from."

A 2.8 billion parameter checkpoint at roughly 6 GB is already out, and people are fine-tuning it. Labenz spent the most time with Mamba Chat from Haven HQ's Justus Mattern: 16,000 chats from UltraChat 200k, runnable in a Google Colab notebook. The model chats, knows things, can be induced to hallucinate, and summarized 500 tokens and 1,500 tokens without trouble. Around 3,000 tokens it lost coherence. If the architecture is built for long-term memory, why does long text fall apart? He looked at the data. Average UltraChat length is 1,467 tokens. The state *can* carry information forward indefinitely. That does not mean it *will*, if training never asked it to.

Speed is a second caveat. The paper expects 58 tokens per second on a single A100 for one inference stream — faster than most people read. Colab, even on Labenz's paid account, gave him a previous-generation V100 with 96 KB of shared memory, a little more than half the A100's 164 KB, and runtime was inconsistently fast or slow. Recurrence is a real limit: at 20 billion parameters you might be waiting, and at Waymark a thousand-token video script could take the better part of a minute. Fine-tuning libraries and datasets mostly just worked. Long context and latency did not come for free.

Head-to-head scores may also be asking the wrong question. All the familiar benchmarks are things Transformers can already do; none of them test a long-lived state. "Is Mamba the Transformer killer? The answer there is almost certainly no." The comparisons so far are on the Transformer's home turf — short sequences, familiar evals — because that is where Transformers have traction and where datasets exist. The structural weakness on the SSM side is the other face of memory: the state is lossy compression. If important information is dropped at one step, it is gone. Load *The Great Gatsby* into Claude, bury one weird sentence, and attention still computes that sentence against every other token. An SSM that did not flag the anomaly in the moment may never retrieve it. Selective forgetting is not a free lunch.

Two agendas follow. The first is sculpting memory, and Labenz thinks it is more a data problem than ever. Intentionally designed long sequences have to teach what to retain and what to let go. If you want needle-in-a-haystack behavior, synthesize anomalies and require them to be recalled much later. If those diamonds never appear in training, the state will not keep them. Datasets, he notes, often outlast architectures: MMLU is years old and still relevant. Private companies may be especially well placed, because onboarding, customer-support logs, and clinical processes are already long-token investments that the public web does not contain.

The second agenda is remixing. We used to have one block so much better than everything else that we stacked it with itself, even knowing the layers were doing different jobs. Now there is an existence proof of a second block with similar, slightly better power, better scaling, and higher throughput, plus a different memory profile. Within two weeks of Mamba, Together AI — Dao's company — released Striped Hyena 7B, a globally competitive hybrid of attention and a (not yet selective) state space model, already on the Together API, with training planned out to 600 billion tokens. A Block State Transformer paper followed, feeding an evolving global state into attention via cross-attention. Neither hybrid even uses selectivity yet.

Labenz quotes Together's blog because it matched what he had already inferred: "Early in our scaling experiments we noticed a consistent trend: given a compute budget, architectures built out of mixtures of different key layers always outperform homogeneous architectures." The future is not an all-attention stack or an all-Mamba stack. It is mixtures. He also wants the SSM itself to evolve: multiple states the way Transformers have multi-headed attention; contrastive objectives that push states apart; dedicated classifier states; a few historical states paid for with extra HBM traffic; mixtures of experts; maybe even an approximate convolutional form that recovers some training parallelism. Human memory is already a hint — you remember this morning's breakfast, the best breakfast, and what you usually eat, not every breakfast — which he renders as *state decay*, analogous to the weight decay that makes grokking possible.

State size is a new dimension beside parameter count and context window, and the current paper barely explores it. Hardware is probably not optimized for it. V100, A100, and H100 have grown SRAM and HBM at roughly the same ratio. If models do not need to get much bigger but states do, Labenz would not be shocked to see chips shift toward more shared memory. Exporting every state would likely be prohibitively slow; exporting every thousandth state over a million-step inference is already a thousand snapshots whose cost is unknown. That scaling law — how much internal state you can save without tanking throughput — does not exist yet.

Applications would then shift from context management toward context selection, or from RAG toward *state search*: a library of precomputed states, each the residue of a textbook, an email history, or a thousand worked problems, swapped in instead of stuffing text into a prompt. He analogizes mixture of experts to mixture of states, including disposable parallel copies. An agent paging through gigabytes of email search results could fork twenty copies, scan for relevance, discard the copies, and return to the parent with only the hits — without clogging context or state with junk. Transformers handle that poorly even at 100,000-plus tokens.

A dedicated state could also be a runtime immune system: a log of known attacks, updated in a lightweight way that does not stuff the system prompt. Labenz notes that his spear-phishing demo kept working on ChatGPT; OpenAI could have added "do not spearphish" to the prompt and probably blunted it, but filling the prompt with that kind of patch is a bad trade unless the hole is serious. Shame on you for fooling me once, shame on me if you fool me twice, only works if there is memory. Transformers do not have that mechanism. SSMs do. If he had to guess the split, it is two-thirds datasets and one-third architecture. Jobs, not tasks, would need to be recorded, with reasoning steps and a higher-level narrative — who am I, what is my job, what am I doing — made explicit. When Labenz fine-tuned GPT-3.5 at Waymark to write video scripts, raw examples did little; spelling out strategy first helped a lot. That narrative data is not on the web.

Safety researchers may get a gift. Instead of hunting circuits across Transformer activations, they can compare last state, current state, and next state, and ask whether deceptive, helpful, or harmful intent is loaded. Representation engineering looks especially portable. The catch is the same SRAM trick that makes Mamba fast: states are not taken off-chip, and exporting every state would violate the premise of the hardware-aware algorithm. Interpretability in the lab is natural; production monitoring would need on-SRAM checks or new silicon. Pure SSMs will also be behaviorally weird, Labenz suspects, about as weird as Transformers were, and hybrids may only partly cancel that. The field has a decent map of what Transformers trip over. For state space models, that map has not been started.

The social stakes rise if long-running states actually come to know people. Replika listeners already fell in love with primitive chatbots; Character.AI users spend hours with systems that still lack long-term memory. Mutual evolution would make projection of value much easier, and losing a prized state could be a real, hard-to-recreate loss — logs of every input are not the default, and intermediate states are discarded. Labenz even floats a dedicated state whose job is to predict the state fifty steps ahead, which sits near one operational definition of consciousness, without claiming subjective experience. Robin Hanson's *Age of Em*, with pausable, cloneable digital minds, looks more apt in a world of durable states than in a world of Transformers. He would like to get Hanson on the show.

Both authors' startups are worth watching: Cartesia says it trains models with subquadratic scaling; Together is the infrastructure and metal-level programming shop one would expect from Dao. Labenz is genuinely puzzled that they published, including the CUDA. In a closing era, this is the kind of secret you could sell. A thought experiment: take it to Microsoft and they would have paid a lot. Unless he is wrong about what it enables, publication means acceleration. One skeptic told him maybe the leading labs already have something like this inside GPT-4. He doubts it — the public products still present a hard context window — but he would bet OpenAI and Anthropic already have people characterizing the paper. OpenAI's Assistants API, with arbitrary-length threads that currently fetch history back into a finite window and bill for the fetch, would be dramatically smoother with a propagating state, possibly billing only for new tokens. The API looks, honestly, a little more like it was built for this future than for a pure Transformer.

Labenz expected more agent progress in 2023 than he got. GPT-4V arriving late is one reason: the browser is visual, and vision only recently became usable. He still thinks agents can be brute-forced with Transformers — process reward on intermediate math steps already set a MATH-benchmark record — and he half-expects GPT-4.5 to be the same raw power as GPT-4 with much more reliable reasoning, the way 3.5 was 3 plus RLHF. Long memory is a qualitatively different reason to believe agents will work: coherence, robustness, and predictability that scaffolding and skill libraries cannot fully fake.

The control problem, in his view, just shifted qualitatively. DeepMind's FunSearch paper in *Nature* used a frozen language model, not even fine-tuned, to push the state of the art on mathematical problems open for decades. The pattern is P versus NP flavored: hard to find, easy to score. A million generations at GPT-4 retail prices is low tens of thousands of dollars. That paradigm extends extremely naturally to exploring architecture space now that there are two equally expressive blocks with complementary strengths — dense all-to-all attention versus long memory that must forget. Codey, a PaLM 2 descendant, was enough for FunSearch. GPT-4 remixing skip connections, interleaving patterns, and block sizes, trained at one-millionth of GPT-4 compute, might beat the frontier. If the explorer itself is already a state-space hybrid with long-term memory, it may not even need to be tortured into trying novel designs.

The caveat is large: maybe it will not scale past 300 billion tokens. He does not know why it would not, given how predictive scaling laws have been, but he might be wrong. If he is right, this is not the end of Transformers. It is the end of the Transformer *era* — the time when repeating the same block gave the best result — and the start of a multi-architecture era whose recombination will be hard to analyze, especially if models start exploring architecture space themselves. From 2017 to late 2023 was six years. He would guess this new architecture gets validated and elaborated in a third or a quarter of that time: more people, more hardware, existing datasets and benchmarks, and models that can help. The forecast he opened with is the one he closes on. We are going to see more effective agents, more compelling long-term assistants, more compelling long-term AI friends and companions, several times as fast as the Transformer era. If you thought the cognitive revolution was going to give you any rest, buckle up.