Mamba After 90 Days: What 30 Papers Reveal About State Space Models, Hybrids, and Mixture of Experts

Labenz, host of the Cognitive Revolution, introduces the format as a zoomed-out survey of a research subfield rather than a single-researcher interview. His guest Jason Meaux runs a site that tracks Mamba and other state space model work. Meaux is an engineer who spent a decade in large-scale energy infrastructure and came to state space models because he had use cases that needed sub-quadratic architectures.

Labenz recaps why the original Mamba paper stood out. It beat the Transformer on the core text-modeling loss with a simple, attention-free design, and its selective mechanism lets the input influence how it is processed, a kind of dynamic computation that earlier state space models with fixed weights lacked. Because the state has a fixed size rather than growing like an attention matrix, each token costs constant time, so inference and training scale linearly instead of quadratically.

He frames Mamba not as a successor to the Transformer but as a second powerful core mechanism next to attention, and says even in December there was evidence that hybrids would likely maximize performance. The 90-day look is meant to test how that prediction plays out.

Meaux reports that well over 30 papers and significant projects appeared in 90 days. About 60% concern vision or image processing, a surprise since the original paper focused on language, genomics and audio. Roughly 15 are vision papers, nine of them on image segmentation with biomedical applications, and about a quarter concern natural language, with the rest scattered across graph networks, audio and even stock picking.

Eighty percent of the papers modified the architecture instead of applying vanilla Mamba, and 73% reported state-of-the-art results, though neither host has verified them. Labenz stresses the caveats: the work was done fast, the papers feel quickly drafted, and baselines are often unclear, which gives him a new appreciation for peer review. He also notes the Mamba paper itself drew at least one confident reject in review.

One of the first projects was a blog post that swapped Mamba into nanoGPT, Andrej Karpathy's minimal GPT training code. Its author reported Mamba beating the Transformer on a speech-synthesis experiment, using less memory, and called a Mamba block a drop-in replacement for a self-attention block. Labenz highlights it because Colab notebooks make it easy for anyone to try.

Labenz worries that most papers use small models and small datasets, where hardware-aware optimization matters less, so results may be more forgiving than at scale. Meaux says his own experience was pleasant: Mamba works with the Hugging Face training library and with fully sharded training, and he has trained several small models. He has not touched the CUDA kernels and trusts the original authors' highly optimized code.

Labenz explains the state as a substitute for the attention matrix: a running encoding of everything seen so far, which in Mamba is learned and selective rather than fixed by design. This promises a long-lived memory that could be saved or forked, in contrast to the episodic way Transformers manage context. He also corrects a mistake from his December episode: the roughly 200 KB of SRAM quoted for an A100 is per compute core, and the aggregate is closer to tens of megabytes.

He then raises a basic question the papers do not clearly answer: in a model with many layers, does each selective state space block have its own state, or is there one shared state? Meaux explains that the Mamba parameters B, C and Delta depend on the input and that the state is updated at each step and kept in SRAM, but admits the specifics are hazy. Both agree that some low-level details are still being worked out collectively.

On learning theory, the first question is whether Mamba can do in-context learning. A layer-by-layer study from Italian and German institutions says yes, and shows representations refining across layers, as in Transformers, similar to logit-lens analysis. Early layers predict poorly, then predictions improve steadily, with some bends in the curves. Labenz also notes how global the authorship is, with a minority from the United States.

The standout result is Othello-Mamba, modeled on Othello-GPT. The models are tiny, 9 and 17 million parameters for Mamba versus 11 and 21 million for the Transformers, and they only see sequences of moves, never the board. A linear probe trained on each layer's activations reads out the board state at 67% and 71% accuracy for Mamba, compared with 55% and 57% for the Transformer version.

Labenz argues this is hard to explain for a model that merely associates earlier moves with later moves, because tracking captures requires an internal board representation. Accuracy peaks in the middle layers and drops near the output, as in Transformers, and it worsens as games approach 64 moves. Meaux suspects a bigger model or a bigger state would help and suggests freezing parameters while enlarging the state to test it.

Meaux mentions an 11-million-parameter Mamba chess model, trained on 18.8 million games, that still makes illegal moves but won 37% of games against Stockfish level zero, which a search suggested is around 1,200 Elo. A 50-million-parameter version is expected, which would allow apples-to-apples comparisons with existing Transformer chess work.

On efficiency, the Othello-Mamba authors found it more data-efficient but initially 7.6 times slower to train than an equally sized Transformer on an A100. Changing the batch size from 256 to 64 cut that to about three times slower. Labenz takes this as a sign that matching the original Mamba's speed is subtle and does not carry over automatically.

Labenz finds it philosophically odd that the state is never directly optimized: the probes read activations between layers, not the state itself, and training only targets the next token. He expects explicit objectives for the state, such as asking it to represent the true board, to become a loss function soon, possibly speeding training. Meaux has not seen such work but calls it very interesting.

Meaux walks through a paper from a South Korea and United States team that compares a Transformer, S4, Mamba and a variant without selection, and then proposes the hybrid MambaFormer. On sparse parity learning, where only a few features matter, a 24-layer, 768-dimension Transformer could not beat random guessing, while Mamba solved it with just two layers. On a noisy linear in-context regression task, with 90% of inputs as noise, Transformers managed to a degree but Mamba performed better at every step.

The reverse holds on multi-query associative recall, where the model must retrieve values for keys given earlier in the context. With very small recurrent state sizes, Mamba outperforms Transformers, but once attention can use its full capacity, Transformers solve the task perfectly, while Mamba plateaus near 90%. Meaux cautions that synthetic tasks may not transfer to real settings.

Labenz offers a theory for the split. In noisy settings, attention considers everything at once and may thrash by overfitting to noise, whereas Mamba's sequential processing lets it quickly notice which tokens carry signal. But Mamba gets one chance to encode each token into the state, so if a pattern seems unimportant when it passes, it cannot be recovered later, while attention can look back at tokens it previously passed over. Meaux agrees with this framing.

The authors' answer is MambaFormer: keep multi-head attention, add a Mamba layer after it, and replace the positional embedding with a Mamba layer. It solved every task in the suite, though the paper does not test training or inference efficiency. Labenz is struck that the feed-forward block is removed, and guesses a scaled-up version would bring the MLP back because it likely stores facts.

Both hosts find it striking that swapping a Mamba layer in for positional embeddings simply works, though no obvious principle explains why. Labenz says the field often tries things that work before anyone knows why. He summarizes that Mamba can learn in context, builds world representations, and has strengths that complement attention's.

Labenz briefly covers mixture of experts, noting that GPT-4, Mixtral and Gemini 1.5 have been described as using it. In place of one MLP block there are many, and only a subset is activated per token. Two papers, MoE-Mamba and BlackMamba, apply this to Mamba, either adding a switch-style MoE layer to Mamba or swapping the Transformer's attention for Mamba's selective state space, and both report positive results.

BlackMamba stands out because it comes from a very new stealth startup, one of whose co-founders recently left Conjecture, and it reported solid throughput. Labenz notes that MoE improves training and inference efficiency at the cost of many more total parameters, which must sit in high-bandwidth memory across arrays of GPUs. Even with GPT-4's weights, a model of two trillion parameters would need more than ten 80 GB GPUs, so he argues the approach favors incumbents like Google and makes open source harder to run.

He expects more intricate swaps ahead, especially replacing the selective state space portion with experts rather than only the MLP, or processing the sequence through an initial Mamba before swapping layers. Part two, he says, will cover vision applications, super-long context, the risk of rotting internal states, and biology uses of hybrid models.