The Author of Attention Is All You Need Now Compiles mRNA Like Bytecode

Jakob Uszkoreit spent more than a decade at Google, co-authored Attention Is All You Need, and led research that reshaped Search, Translate, and Assistant. Elad Gil notes that other Transformer authors credit him with the original push to focus on attention. Now CEO of Inceptive, he is trying to compile RNA the way software compiles bytecode. Sarah Guo asks him how the Transformer actually started.

He refuses the lightning-bolt version. In deep learning, he says, you cannot make something on the theoretical or formal end really work without going deep on engineering. It has to be efficient. In his mind that is the one thing we know works if you want to push the field forward: make the model faster, more effective, and more efficient on the given piece of hardware.

The linguistic clue is hierarchy. Linguists draw trees that "aren't ever really true" and "definitely not always false," which means they capture real statistics in language—statistics language probably evolved to exploit human cognition. You do not need to read a sequence all the way through, then all the way back. You can understand pieces in parallel, then combine incomplete representations until the last ambiguity disappears. That process is a tree. Evaluating all possible trees, reasonably approximated, means repeatedly looking at all combinations of pieces. That quadratic step is attention, and it was a natural fit for accelerators that do lots of simple work in parallel rather than complicated work in series.

The biggest difference between the Transformer in the paper and ancestors such as decomposable attention was not the idea so much as the implementation. Co-authors including Noam Shazeer wrote it in a way that was an excellent fit for the accelerators of the time. Theory that does not land on hardware, in this telling, is a non-event.

Gil raises a common worry: Transformers get interesting at scale, other architectures might too, but nobody throws enough compute at them, and GPUs fit Transformers so well that rivals look slow. Uszkoreit’s reply is: even if that is true, does it matter? He would rather evaluate simpler architectures paired with hardware they fit like a glove. GPUs were not built for this. Why assume they are anywhere near optimal?

Fast, scalable matrix multipliers are generally useful in scientific computing, he grants. The rest is bells and whistles. On memory bandwidth, and on the trade-off between inherent parallelism and latency, he does not think GPUs sit at the sweet spot for large-scale deep learning. Try the combinations and you might quickly find something better. MLP-Mixer, a vision architecture that is just an MLP, was not much better than vision Transformers and not much worse. Changing architecture is not that hard, especially if you simplify on the way.

Software and hardware are a chicken-and-egg. The newest accelerator designs already take large-scale Transformer needs into account, increasingly so. The other contributor to the Transformer’s success, he says, is not efficiency at all. It is optimism. Suddenly a bunch of things people tried with this architecture started to work, then more started to work. Human cycles are fueled by "suspension of disbelief," also known as hope. The community energized, tried everything under the sun, and updated its prior to "this just works"—which is not actually true the first time you try something new.

Asked where that optimism should go next, Uszkoreit names a first-principles bug. The compute a language model spends on a problem scales with how long the prompt is and how long the response is. Many hard problems have short answers and can be stated in a sentence. Prime factorization is one: simply stated, huge impact, almost no compute. There is no knob for the user, and no knob for the architecture, to say this is hard and needs more work.

That is why training on generated data looks less crazy than information theory first suggests. Classical information theory says you cannot squeeze more information out of a model’s own output. It also ignores compute and the energy spent generating that data. Retraining on generated text amortizes compute that was spent at some earlier point, so the system slowly spends more computation on the same problems. It works. It is also clunky. He would rather the model decide at inference time.

Anytime algorithms are the picture he wants: if you have more time or money, let it run longer, but only on hard problems. Right now GPT-4 will expend a ton of compute adding 2+2 if you bury the question in a long prompt and demand a complicated answer. Related flavors of the same waste show up as a lack of elasticity. Scale a video up, interpolate frames with a trivial algorithm, and feed it in again. The task did not change. The model still burns far more compute. Models should consume different resolutions, sampling rates, and durations and adjust work to what you actually need. They do not.

Guo mentions Meta’s depth-adaptive Transformers and test-time search, especially in code generation where compilation can close a loop. Uszkoreit thinks test-time search is super effective and also clunky: you cannot easily train end-to-end through it. As an engineering hack for amortizing compute you already have, it is really good. Universal Transformer, which his group tried years ago, never caught on for a blunt reason: it does not work well enough. Compute is so scarce that if it really worked, we would see it everywhere.

Uszkoreit has always been interested in biology and, by his own account, knows nothing about it—a problem, because it is hard to learn once you leave school. He also does not have high hopes that humanity will develop a complete, predictive theory of how life works. We do not even have a full inventory of what is going on, let alone the mechanisms. Even for what we do know, the theories are often not predictive enough to use. Protein folding is the example: pretend there are no chaperones and no early kinetic effects of translation, and the theory still was not practical.

Deep learning at scale is an alternative that can work around both gaps if you treat the system as a black box and observe input-output pairs at sufficient scale. The personal push was the birth of his first child, which gave him a different appreciation for how fragile—and wonderful—life is. AlphaFold 2 had just solved a fundamental problem in structural biology. mRNA vaccines were saving the world. He wanted to take the tools from Google and apply them to RNA, "this neglected stepchild," starting with mRNA.

Inceptive calls the result biological software. mRNA is roughly bytecode, the actual substrate the software is made of. Models compile a biological program—something that might look like a bit of Python specifying what a medicine should do inside cells—into a description of an RNA molecule that will, they hope, do what you wrote. Today’s mRNA vaccines are a programming language with one statement: print this protein. Self-amplifying RNA and riboswitches, which change structure or self-destruct in the presence of a small molecule, give you conditionals and recursion. Squint and it looks Turing complete, with I/O, and much easier to manufacture at scale than protein biologics.

Moderna, BioNTech, and peers already have hundreds of RNA programs in the pipeline, heading for high triple digits by the end of the decade—potentially the second or third biggest drug modality by revenue and impact. Current mRNA vaccines were still suboptimal: side effects, limited access when they were needed. Protein biologics often die in manufacturing; medicating the world with them would take more factory land than the planet has. RNA infrastructure, he says, was heading for six to eight billion manufacturable, distributable doses within two years, and Inceptive’s lab can already print pretty much any given RNA sequence. That matters for machine learning too, because you have to generate training data that does not already exist. Guo restates the vision: one antigen, many expression, stability, and immune profiles, no cold chain, cell-type conditions, a space of 10^630 variants. Uszkoreit agrees on the goal and corrects the verb. It will not be a search. It has to be generation. Screening is mathematically finished, and personalized cancer vaccines, with many antigens per patient over time, make that even clearer.

Gil is excited to get to the right answer without discovering every mechanism and running massively expensive screens. Uszkoreit takes the thought further. We may be at a crossroads where the hope of discovering and really getting how this works is holding us back. Computational linguistics spent years trying to build a sufficiently accurate, complete theory of language so that machines could use it. It did not become the path that worked.

Guo hears a parallel in genomic sequencing: first you sequenced big chromosomal chunks contact by contact, then you smashed DNA into random tiny fragments and let a machine reassemble them. Uszkoreit says the parallels are striking and do not end there. The invariant across domains is that the formalisms we invent to communicate conceptual understanding are great for education and for some kinds of reasoning. Given limited cognitive capacity, they may not be the right tool for predicting what a given intervention will actually do.

Gil adds the history of drugs. Aspirin came out of willow bark in the 1800s with no mechanism and minimal side effects. Metformin binds multiple targets and we still are not sure exactly how it works. Modern regulation asks for a mechanism, a proven pathway, hurdles that do not necessarily help efficacy. Uszkoreit’s reply is blunt: it is a waste of time and money, it might not even be true, and we have no way of telling. The ground truth is empirical. Does it work, and does it do more good than harm.

The blasphemous part of an end-to-end black box is that you may never extract the theory you are testing. People are trying to read theories out of models; he thinks it is worth trying and is not optimistic except in simple cases. Climate and weather forecasting are his counterexample. We understand the Schrödinger equation and how, in theory, it could solve these things. That is not practical. A theory that is both predictive and usable might just not be something we can put in our heads. Gil notes this is a throwback to pre-molecular biology: mutagenize, look at output, map genes by function before you understood the molecules. Deep learning, in that light, is an older and fruitful style of biology with a new instrument.

On human augmentation, Uszkoreit is very bullish in the long term and does not see the path intuitively. Brains look physically organized around input and output. Why would evolution have left spare compute sitting around in case we boosted I/O by a few orders of magnitude? Bound the problem by a lifetime and it is not clear there was pressure to make capacity much larger than some multiple of I/O. The usual comparison—LLMs trained on huge token counts versus a toddler hearing hundreds of thousands or millions of words—confuses fine-tuning with pre-training. Pre-training is all of evolution. The compute was amortized over millennia. We come pre-wired for language, so it only takes on the order of a million tokens at the end. Reading that pre-wiring out of the actual wiring may be impossible, because language and brains co-evolved and the mapping is fuzzy.

Guo notes that children born without a hemisphere can rewire, which makes some regions look like reallocatable general-purpose machines. Uszkoreit is wary of the phrase. After massive trauma, can the brain rewire to do something very different? It might still be specific, just built for a flavor of redundancy. That is also why he finds AGI as a term particularly problematic: he does not know what "general" means.

Inside Inceptive the team calls itself anti-disciplinary, pioneers of a field that does not have a name yet and draws from deep learning and biology. Designing the assays that generate the data that train the models is the core craft, because today’s experiment uses yesterday’s model. The dream of one clean loop—experiment, in silico, next experiment—is a pipe dream. What they actually run is a giant mess of little cycles: query a network, take a readout, feed another model, get instrument parameters, run the instrument. The boundary between wet and dry is increasingly blurry. "Our work happens on the beach, because that’s where the wet and the dry meet in harmony."

New hires usually arrive from one side, deep learning or biology. It does not take them long to start speaking a weird Creole of those languages and thinking in mixed ways. Then, he says, magic happens. Biologists work on problems they would not have thought about two years earlier, with people they would otherwise never have met. Most of the results do not work at all. Sometimes they really are magic.