Five Independent Teams Found the Same 10-to-1 Mix That Beats Transformers

Sarah Guo and Elad Gil open this June 2024 episode of No Priors by introducing Karan Goel and Albert Gu as co-founders of Cartesia and authors of S4 and Mamba—leaders, in the hosts' phrase, of "a rebellion against the dominant architecture of Transformers." Gu has just started as a professor at Carnegie Mellon, where his lab continues the academic side of state space models while Cartesia puts them into production. The company is still tiny: 15 people, plus a class of eight interns that Guo likes to needle Goel about.

Cartesia's first product is Sonic, a fast text-to-speech engine built for interactive, low-latency voice. The two hottest uses so far are games, where studios want NPCs that can talk back on the fly to millions of players, and voice agents, where delay is the product. Goel says Sonic already shaves about 150 milliseconds off the stacks people typically use, and the 2024 roadmap was to cut another 600.

That latency budget is not a rounding error. In conversational speech, a hundred milliseconds is something a listener can hear, so the difference shows up as feel rather than a leaderboard score. The rest of the conversation is about why the underlying architecture, not just product engineering, is what makes that speed possible—and why the same models might one day leave the data center entirely.

Gu did a long PhD at Stanford. Sequence modeling became his problem after a DeepMind internship in 2019, right as Transformers were taking off. He went the other way, toward recurrent models that felt more like how people take in information: one step at a time, updating a state. He also found them mathematically more elegant. A couple of years before this recording he and Goel published S4, which showed a state space model could be effective; later Gu proposed Mamba, which put SSMs on a more even footing with Transformers for language modeling.

Goel's path was messier. He grew up in India in a family of engineers, tried to become a doctor in high school, discovered his aptitude for biology was "very low," and went to an IIT and then Stanford instead. In 2017 and 2018 he worked on reinforcement learning. His adviser, Chris Ré, was skeptical of RL as a field, so Goel entered a stretch of not knowing what he was working on.

Their first project together left a very specific memory. Goel kept filling a Google Cloud disk and insisting on expanding it by only one terabyte at a time. By the end, logs filled faster than they could be cleared: Gu tracked experiments, and Goel deleted logs in real time so the runs would not crash. The real turn was S4. Two or three weeks before a NeurIPS deadline, Gu was stuck working alone and recruited Goel because he "wasn't doing anything" for that deadline. The paper landed, and Goel was in. For years before that, Goel says, Gu was "over in the corner scribbling away, talking to himself. We don't really know what's going on."

Gu traces SSMs to the RNNs he worked on as an intern in 2019. The premise is simple: if you want to model a sequence, process it one step at a time, encode what you know, and update that state when new information arrives. He found connections to dynamical systems that felt "fundamental" and "just right." At a high level the model is a fuzzy compressor. It packs everything it has seen into a state, strips what does not matter, and updates constantly as data arrives—a loose analogy to how a brain handles a stream of information.

Different data types play to different strengths. Text is already compressed and tokenized; every token is dense with meaning. Raw audio and video are highly compressible, continuous signals. Early S4-style models were good at raw waveforms and raw pixels and weak at text; Transformers were the reverse. Mamba, then a few months old, closed much of that text gap. Gu's warning is that there is no free lunch. People assume you can throw a Transformer at anything. "If you tried to throw it at the raw pixel level or the raw sample level in audio waveforms, it doesn't work nearly as well." Tokenization and the training pipeline evolved hand in hand with Transformers. Break those assumptions and other models start to win.

Efficiency is the other reason they care. Transformers scale quadratically with history; a recurrent SSM processes each new token in roughly constant time and scales linearly. That is a huge advantage at long context, but the extra work a Transformer does is also why it is better at some modeling jobs. Gu's synthesis: let SSMs do the bulk of the processing as fuzzy compressors, and keep a little exact retrieval—the thing a Transformer is, a memory of every token you are allowed to look back over. It is the same rough picture as a brain that is mostly real-time state, plus a scratch pad or lookup.

Combining them into hybrids "tends to work really, really well," better than either alone. The ratio that keeps turning up is mostly SSM layers with a little attention, "maybe a ratio of like 10 to 1." Gu knows of "at least probably like five groups that have independently verified" that mix. People are using it mostly on text. Collaborators had just told him they applied a Mamba-based model to DNA: pretrain on long sequences, then fine-tune for classification, including the downstream effects of point mutations—not protein folding, as Guo, a former biologist, first asked.

Cartesia picked speech to show that SSMs work outside papers. Audio is a natural fit for faster inference and streaming. Goel is blunt about why not DNA: it is interesting, but it does not personally motivate them. What does is multimodal data that is information-sparse, where compression is an advantage because you can stream data through the system, update memory quickly, and handle very large context by design. Commercially, voice agents and games both want people to talk instead of type. Longer term he imagines replacing graphics and rendering with models that output streams of data in real time.

Sonic is the first proof of that bet: general models inside the company for efficient audio generation, conditioned on text transcripts, plus a training stack good enough that people want to use it. Text-to-speech has been built for 30 or 40 years and is still not at the ceiling. Goel's test, which Guo immediately named Karan's audio Turing test, is simple: "Would I want to talk to this thing for more than 30 seconds?" If the answer is no, it is not solved. Most systems fail that test. Builders get lost in the weeds of one mispronounced word or a boring intonation, and users disengage after two sentences.

Two hard dimensions remain. Emotion: you need control over how a line is said. Social role: a nurse, a lawyer, a judge, and a venture capitalist do not speak the same way, and "that's the nuance that I don't think any models really capture well." Gu adds a deeper trap. Even "say this word" is often not a local problem. Hard cases need real language understanding. To get perfect TTS or speech-to-speech, "you actually really need to have a model that has more understanding." Doing one modality well forces you into multimodal models. Speech is Cartesia's entry point into that stack.

Goel's longer north star is pushing inference to the edge. The hope is to reproduce a capability that "today cost a million dollars in the data center" with "$10 on a commodity GPU or accelerator at the edge." Instead of batch jobs in the cloud, processing happens next to the sensor. A security camera is the example: most of what it captures is useless, so the model should sift the stream quickly, discard what is not useful, remember what is, and use that for prediction and generation.

Guo notes that Apple's then-new on-device models were about three billion parameters. Goel is unimpressed on capability: 3B is still small. The real question is how to make capabilities "very, very good" with a low on-device footprint. He splits the industry into two waves. The first wave was a bet that scaling data and compute would work; someone took it, and we got the current models. The second wave is efficiency, the same story that produced phones. The goal is "the smartest model ever but run it real cheap so you can run it repeatedly at scale"—a hundred times for the dollars you spend once today.

His diagnostic is concrete. Take a 7B language model, run it on a Mac, and open the profiler: tokens per second go down, memory goes up, and power is a problem. Transformers are a bad fit for that constraint, even as data centers have only recently started talking about power. Guo, speaking as an investor, says you get different applications once you assume the model can run on hardware people already own. Goel's household example is an on-device music model as a personal musician you can talk to, without a round trip to the cloud.

Next for Cartesia is to keep improving Sonic as a real-time signal generator, add the speech features customers actually ask for—many languages, more controllability—and bring Sonic on device so a MacBook runs it as well as the cloud. After the output side comes the input side: ingest audio natively, converse, and reason over context. The intention, Goel says, is a large-scale multimodal model, trained with techniques he will "leave for another podcast," that is cheap to run on device and excellent at audio. Fidelity from SSMs, he argues, is "just very different" from what people have been able to see.

Guo points at the pain of today's voice apps: speech to text, into a language model, back to speech, with a round-trip of extra latency. Goel says orchestrating multiple models is "quite painful" and turns "something that the model should do into an engineering problem." It "feels almost inelegant." Gu finishes the thought from the science side: "In the end all the systems go away and it's just one model and then we also go."

Asked how he picks research problems, Gu's answer is aesthetic. He cannot fully explain it. SSMs "just felt like there was something really nice about it, elegant," and he wants simple, unified solutions to hard problems. He cites Paul Erdős on proofs from "the Book"—the collection of most elegant proofs. Machine learning is rarely that clean, but the same instinct exists at a different level of abstraction. "Sometimes I tell people this is just the right way of doing it and I can't explain why." At Cartesia they still have to solve hairy engineering, but he wants everything as simple and unified as possible.

Goel demos Sonic on "a standard issue Mac," the same feature set as the cloud playground. He generates "It's great to be on the No Priors podcast today," then, when asked to prove it is not a canned clip, switches voice and types an Erdős line: "You don't have to believe in God, you have to believe in the Book." Audio streams out immediately. Guo jokes she would talk to Erdős on her laptop every morning.

The company is 15 people and eight interns. Guo always gives Goel a hard time about the intern class; he likes that they are excited and want to do cool things. They are hiring for modeling roles and across the engineering stack, "team SSM," under a slogan they used in the PhD years: overthrowing the Empire. Whether the rebellion works is an open question. The Transformer ecosystem of tokenization, training pipelines, and hardware is enormous. What they can put on the table is S4 and Mamba, a shipping voice product, a 10-to-1 hybrid that five groups have already checked, and a partnership that once survived by deleting logs in real time.