Seeing Like a Mamba: Vision, Rotting States, and a Cell Model in 90 Days

This is the second half of Cognitive Revolution's Mamba-Palooza. Host Nathan Labenz, an entrepreneur and AI investor, is back with Jason Meaux, the AI scout behind statespace.info, to finish a paper-by-paper tour of the first 90 days after Mamba landed on arXiv. Part one covered the original architecture; this episode is the vision literature, the first serious attempts to use super-long context in language, and a biology project Labenz thinks may matter as much as anything they discuss.

They open with a bet they had made a few weeks earlier: how many Mamba papers would appear in a two-week window in February. Labenz set the line at five and a half. Meaux took the over. When that number looked hopeless, Labenz reset the over-under to 14, which Meaux thought would be a tough get. The final count was 13. "This is definitely an object lesson in exponentials," Meaux says.

Labenz then sets a new line for the next 90 days: 55 fresh papers, to be resolved on June 5. Meaux takes the over again, because 55 would be less than a doubling of the first 90 days, and he expects the second 90 days to produce more papers than the first. With the wager on the table, they turn to the surprising majority of the literature: computer vision.

Meaux had already noted that more than half of the follow-on papers deal with vision. That is striking because the original Mamba paper worked on language, DNA, and audio — modalities that already have a natural order. Language goes token to token, music proceeds through time, DNA is one base pair after another. An image does not. Pixels sit in a two-dimensional grid, and the literature quickly adds a time axis for video and a third spatial axis for 3D imaging.

Transformers have a version of this problem and mostly solve it by chopping images into patches and treating those patches as tokens. Because attention is not recurrent, every patch can talk to every other patch on the first pass, so a single flattening is usually enough. Mamba cannot do that. It processes a sequence in order and can only decide what to remember based on what it has already seen. If something important appears late, the model may have already forgotten the clue it needed.

That constraint is why so much of the vision literature is really a literature of scanning strategies. Researchers turn an image into one sequence, or several, run selective state-space models along those paths, and then combine the results. Labenz's working theory is that images attracted so much work precisely because they are *not* sequential in the way everything else was, which made them the hardest and most interesting test of the new block.

The most common vision task by far is biomedical image segmentation. It was kicked off in early January by researchers at the University of Toronto, who interleaved convolutional blocks with Mamba blocks inside a U-shaped network. The idea was straightforward: convolutions extract local features, the state-space layer tracks long-range dependencies, and the combination posted numbers at or above the reported state of the art on organs, CT scans, and MRIs. That paper set off a flurry of segmentation follow-ups, including Swin-UMamba, which already claims to beat the Toronto U-Mamba variant. Mamba papers are no longer only scoring themselves against Transformers; they are scoring themselves against other Mambas.

Meaux, who is newer to the clinical use case, sketches why the task matters. A doctor monitoring disease progression may face hundreds of grayscale images and has to separate organs from tumors from other tissue. Better automatic segmentation tightens that feedback loop. Labenz is more interested in a methodological contrast with the language papers. Language experiments usually compare a homemade Mamba to "the best Transformer recipe we know" on relatively small data, and he trusts the original authors' recipe more than he trusts every later baseline. In segmentation, the datasets are hard to create, the benchmarks are established, and the claims sound closer to actual state of the art.

That does not make the numbers sacred. Labenz still wants readers to ask how hard authors worked to optimize the other side of the comparison. But he is willing to treat this cluster as more than a toy bake-off. If the models can quantify what is happening in an organ with precision, he and Meaux agree, that is not just a leaderboard result. It is a better clinical loop.

Labenz spends several minutes unpacking what segmentation actually is, because the name hides the computation. In a classic U-Net, an image is driven down one side of the U toward fewer locations with greater depth of meaning, then back up the other side toward a new image. Skip connections pass the concrete early maps across to the matching stage on the way up, so the output stays grounded in the original pixels. What you see is usually a grayscale scan with colored shapes overlaid, each color a class. That is image-to-image generation, not classification.

The traditional engine for that U is convolution, and convolution is inherently local. Patch by patch, nearby pixels mix; distant parts of the image have no way to meet early. That is where Mamba earns its keep. A selective scan can take a full pass — or several directional passes — across the whole field of view and bring far-apart structure into the same state. Combining that global pass with local convolution is, in Labenz's telling, why these hybrids post their best numbers.

He also flags a data-scarcity trick that later papers, including Swin-UMamba, started to use: pre-train on ordinary images such as ImageNet, then fine-tune on scarce medical scans. The hope is that edge detectors and other visual primitives transfer, even if the medical labels do not. He analogizes to work by Tanishq Abraham on virtual staining and on reconstructing images from fMRI, both of which leaned on large vision models because the in-domain data were expensive. The reported gains were only a few points, and the papers often cite different benchmarks, so Labenz treats the improvement as real but not yet a revolution.

Mid-January brought Vision Mamba, or Vim, from groups in China including Horizon Robotics. Inside each Mamba block the image-as-sequence is forked: one selective state-space model scans forward, another scans the reversed order, and the two representations are added together. Bidirectionality is the whole point. A surprise that appears late in a raster order can still inform the backward scan. The authors report that Vim is 2.8 times faster than their Transformer comparison, saves over 85 percent of GPU memory at 1248-pixel resolution, and, in a line Labenz quotes, has "great potential to become the next-generation backbone for vision foundation models."

The memory plot is what makes the robotics affiliation click. Processing that same high-resolution image with a Transformer, they report, required more than 80 GB — more than a high-end GPU. The Vision Mamba version dropped the requirement to about 11 GB, which Labenz notes is under what a several-year-old MacBook Air already has. If you want a robot to see at the edge, that is the difference between a server and a laptop.

VMamba arrives next with a four-way "cross-scan": start in each corner, stripe the image four ways, run those representations, and superimpose them. The models are small, from about 20 million to 70 million parameters, so four parallel state-space copies are cheap. Labenz immediately asks what happens at 7 billion or 70 billion parameters. If the hidden state already fills on-chip SRAM, four copies force you onto distributed hardware and start to erode the hardware-aware design that made Mamba interesting. Most of these papers, he and Meaux agree, are simply too small to have to care yet.

Two follow-ons make the scanning idea more systematic. VM-UNet takes the four-way scan almost as published and applies it to medical segmentation. Mamba-ND, dated February 8, generalizes further: it treats an image as height and width, adds a temporal axis for video, and even looks at weather data. The paper's distinctive move is not another clever scan. It is a set of wiring diagrams that ask how the scans should talk to each other.

The diagrams walk from a single unidirectional scan, to two in parallel, to four in parallel, then into mixed series-parallel graphs. What Mamba-ND reports is that putting the scans in sequence — one selective state-space pass after another, each seeing a different flattening plus the output of the last — performed best. Labenz analogizes it to cloning yourself four ways versus looking, then looking again with the first view still in mind. Only the second option is available in a real brain, and only the second option lets later scans be input-dependent on what earlier scans found.

Meaux agrees: the whole point of Mamba's selection mechanism is that features depend on the input, so the only way to incorporate bits picked up on the first pass is to run the second pass afterward. A figure of effective receptive fields in the paper shows the multi-directional designs lighting up the most of the image, which is intuitive once you remember the constraint Labenz keeps repeating. Everything has to be a sequence. If one flattening would miss a relationship, represent the same object as several sequences. That, he says, is the core takeaway of the entire vision section.

With the theme in place, they move faster. SegMamba takes the idea into 3D. A 64×64×64 grid is only 64 on a side, but that is already more than 260,000 little cubes if you scan one at a time. The paper uses a three-way scan: forward and backward on slices, plus a cut through the stack. The whole thing ran on four A100s, which Meaux reads as a positive sign. Three-dimensional medical volumes could eat arbitrary compute; four GPUs is almost nothing.

Video is the same trick with time instead of depth: scan forward in time, backward in time, and across the frame. MambaMorph is the oddest vision task of the set. An MRI and a CT of the same body disagree about shape because the patient was not in the same position. The model outputs a deformation field that warps one scan onto the other so a clinician can read them together. Graph Mamba, two papers, flatten graphs into sequences with heuristics and sometimes a reverse pass. Attention had been the long-range tool among nodes; here the Mamba version handles up to 1,400 nodes at under 200 MB of GPU memory, while the Transformer baseline they used dies around 800 nodes.

Labenz is not a graph theorist, but he can see the pattern. A graph has no unique order: if B and C both point to A, nothing tells you which token comes first. Flattening is an expert judgment, then you scan, and for good measure you reverse it. He starts calling the whole era "seeing like a Mamba" — a riff on *Seeing Like a State* — because the art is inventing a sequence representation that the architecture can swallow. Most papers still wire those scans in parallel. One systematic study preferred series. His current read is that a multi-scan approach wired almost any way will probably work, and that best practice is not written yet.

The last deep dive is the reason they got excited about Mamba in the first place: linear-time, constant-cost-per-step processing of arbitrarily long context. The original paper had long DNA; it did not have super-long text. LongMamba trains on a subset of the same data, cut off a little over 16,000 tokens. Perplexity looks good, and it keeps generalizing past the training window. You do not see perplexity rise until past 40,000 tokens.

They also run a passkey version of needle-in-a-haystack, the retrieval eval Greg Kamradt popularized on Claude and GPT-4. Inside a wall of "the grass is green, the sky is blue" sits a number to memorize. At 16,000 tokens the result is nearly perfect, with a few misses at the *front* of the context — lost at the beginning, Labenz notes, rather than the Transformer's lost-in-the-middle. Push past the training window and retrieval generalizes a little, then dies. Beyond about 20,000 tokens the model struggles. The author also points to a Transformer-XL-style trick discussed on GitHub with Tri Dao: save the SSM hidden state after a batch and reload it, so you can keep training 16,000 tokens at a time on the same hardware.

Labenz reads two interventions. Tweaking the delta hyperparameter helps immediately at longer context, but not as much as actually training long. Train at 16,000 and aggregate scores keep improving to around 40,000 — a multiple of the training length — while exact retrieval is already gone. He analogizes to his own memory: he cannot remember yesterday's breakfast, but he remembers what he generally has. Perplexity falling means the early tokens are still useful; needle-in-a-haystack dying means the state no longer has precision.

Then the curve turns. Past that 40,000-token region, perplexity gets a little worse. Labenz cites Robin Hanson: we will keep struggling with rot in AI the way we do in biology and in software. The state is getting gummed up. His prediction for the next 90 days is regularization *of the state*, not just the weights — some penalty on the magnitude of hidden values, a decay over time or over tasks, a way to let go of what is not carrying its weight. He points to *Seeing Is Believing* from Max Tegmark's group, with Ziming Liu as lead author, where a penalty on weight magnitude crystallized toy networks down to the sparsest solution. Do that to the state, and you might keep a model running instead of rotting. Meaux calls it throwing down the gauntlet. Infinite-life companions will process too many tokens to skip some version of cleanup.

The biology pivot is Arc Institute's Evo, which Labenz says has "super big deal written all over it." He describes Arc as an attempt to let excellent scientists just be scientists: a salary, no grant treadmill, machines already there. The model is StripedHyena, a hybrid originally from Together AI, adapted to DNA sequences of up to 131,000 tokens, each token a base pair, trained mostly on prokaryotic genomes. Because the context is long, distant parts of generated genomes appear to relate to each other. If next-token prediction on language grows a world model, he argues, this may be the first time we have seen something similar for biology: a cell model emerging so the network can handle long-range dependencies that are often independent of how close two loci sit on the genome.

The generalization ratio caught his eye. Evo is trained at 131,000 tokens and is reported to work up to about 650,000 — roughly the same multiple as LongMamba's 16,000-to-40,000 stretch. Asked whether DNA is all you need, Labenz analogizes to the graph papers. The interactions that actually determine a phenotype are a gnarly causal graph of promotion, inhibition, and blocking. DNA is a naturally occurring flattened encoding of that graph. If the best machine-learning strategy we have is to flatten, scan, and reverse, it is not crazy that nature already did a version of that flattening for us.

Meaux asks whether this is an inflection point. Labenz is not sure the inflection has not already happened: AlphaFold is a huge deal, Alphabet has a vision for commercializing it, and drugs discovered with these techniques are in or very near clinical trials. The memorable phrase is "when is it worth the wet work." Chemical space is infinite and slow; these models will have false positives and false negatives; they still pay for themselves if they narrow the search by a couple of orders of magnitude. He compares historical drug search to Battleship: throwing compounds at diseased cells in the dark. Score everything on a supercomputer, take the top 1 percent into tissue culture, and a 100-times higher hit rate is a regime change — and a reason to do a hundred times more of the work, not less.

Someone had asked Labenz, after the original Mamba episode, for the odds that state-space models sit inside frontier systems within two years. He still thinks the odds are significant, if only because the world is hungry for compute and linear-time sequence models take the edge off that hunger. The scenario in which it does *not* happen is that attention really is all we need: enough optimizations, line of sight to 10 million tokens with ring attention, and Gemini 1.5 already showing very long context that, as far as Meaux can tell, the company is presenting as Transformer-based work.

Inside the attention paradigm he imagines memory tokens: compress a span of text into embedding space the way a picture is worth many words but costs only 85 tokens at GPT-4V's low-resolution setting. A thousand-to-one compression of history, full attention over the compressed tokens, and maybe a librarian-style retrieval that pulls the original span back if a compressed token gets a hit. Pause tokens and backspace tokens already showed that extra token types can change computation. If that hierarchical, native-RAG Transformer works well enough, it could take the air out of the state-space line.

The counter-data-point is Magic.dev, which had just announced an additional $100 million from people including Nat Friedman and Daniel Gross, and which states on its site that Transformers are not the final architecture and that it has a multi-million-token context window. Labenz would like to have them on the show. For now he and Meaux refuse to count either side out. Ninety days of literature show no shortage of people eager to play with Mamba; Gemini 1.5 shows Transformers still flexing. They adjourn until their own context windows overflow, and Meaux points listeners to the index at statespace.info.