A DNA Model That Can Tell Which Genes You Cannot Touch

Nathan Labenz opens with a survey of three papers from Brian Hie, assistant professor at Stanford and innovation investigator at the ARC Institute. The middle one is the result he keeps citing on the show. Evo is a long-context hybrid language model trained solely on DNA. Its job is next-nucleotide prediction. Out of that narrow objective come two headline abilities: it can tell which genes are essential to an organism's survival, and it can generate novel CRISPR variants. The third paper is almost as striking. A model that had only ever seen single proteins was used to evolve antibodies up to 25 times better at binding their targets than the natural versions.

The episode is built as a trilogy on purpose. One paper is pure machine learning: how to assemble attention and state-space layers and test them. One is DNA at genome scale: what a model can learn from sequences humans did not annotate. One is molecular design: once you have a target, how do you lock onto it. Labenz wants the connections among them, plus the background questions of biosecurity, interpretability, and how much compute a biology lab can actually use. Hie, who still calls himself a computational biologist, is game. The conversation starts with the biggest picture in the field, then walks the papers in that order.

Labenz, who is not a biologist, floats two grand challenges and asks Hie to correct him. The first is causal structure: among the molecules in a living system, what interacts with what, what is upstream, and therefore what is worth targeting if the goal is better medicines. The second is intervention: once a target is known, design something effective and specific enough not to wreck the rest of the system. His sense is that the second problem has moved fast, with AI in the middle of it, while the first is only starting to show hints. Hie says the overview is basically right, and that for drug discovery the first question is both the most important and the most complicated.

The clinical failure mode is not a weak binder. Most drugs fail, Hie says, because the team went after the wrong target. Alzheimer's is still stuck in an argument over whether amyloid is the right thing to hit. Some diseases have no known cause, which makes binder design a non-starter. From a machine-learning view he treats this as a data problem. Observational data are easy to collect: whole genomes, single-cell RNA-seq, a snapshot of health and a snapshot of disease. Biology is so complicated that most of the differences between those snapshots are spurious correlations. The actual causal factor often is not measured, or the measurement looks unchanged.

The way out, he argues, is interventional data: perturb the system and watch what happens. Early in graduate school he worked on Scanorama, which aligns single-cell transcriptomes of the same tissue measured in different labs so that matching cell types can be integrated. That line of work was his first real look at the causal-versus-observational problem, and looking at the data convinced him there is a ceiling on what observation alone can teach. He took a hiatus from the area and is waiting for large perturbation datasets to mature before returning. The other path is bottom-up: a perfect mechanistic model from atoms to a cell, in silico. His group is interested in both, and he suspects they may meet in the middle.

Asked how he taxonomizes himself, given that the next hour will swing from a pure ML paper to wet-lab antibodies, he still says computational biologist. For a long time that label was a slight in biology: models were not rigorous, and experimentalists did not believe them. AlphaFold changed the reputation. There is a mushy-middle version of the field, knowing a little of both, and a harder version that demands real expertise on each side. He tries to practice the hard version, which is why the same lab publishes in ML venues and in biology journals.

The first paper, *Mechanistic Design and Scaling of Hybrid Architectures*, is almost pure ML, driven largely by Michael Poli. Transformers and attention have dominated for years; state-space models now work remarkably well too. The question is how to recombine the primitives. Poli's early code was even called something like Zoology. Hie reaches for a biological analogy: the fossil record shows long stasis and then bursts of diversification, the Cambrian explosion being the type specimen. Architecture design, he thinks, is in one of those bursts, and evaluation has not kept up. New papers often bring their own uncontrolled test suites. MAD tries to formalize the harness: flop-controlled comparisons, a shared set of primitives, and tasks Labenz likes to call micro-skills, among them in-context recall, fuzzy recall, noisy recall, selective copying, compression, and memorization.

Two results matter most. Mixing layers beats purity: all-attention or all-state-space loses to a hybrid, and in practice something like one attention layer in eight to ten, combined with Mamba or Hyena, looks strong. Evo itself uses three attention layers out of 32, about one in ten. The second result is that small-model scores on those micro-tasks predict architecture performance many orders of magnitude up. A full Chinchilla-style scaling sweep is too expensive for an academic lab. Cheap correlates of expensive readouts are the same idea that makes drug development possible: Petri dish, then mouse, then a decade-long trial, each layer supposed to correlate with the next. MAD is trying to build that ladder for architectures.

Labenz wonders whether Kolmogorov-Arnold Networks, from Ziming Liu and Max Tegmark's group, will be the next primitive in a MAD-2 shuffle, and whether ARC-AGI plus Stephen Wolfram's recent cellular-automata rewrite of machine learning point at missing logical operators. His own view is that language models can already reason, just not well, and that going from none to some a few years ago is a reason to expect more. Hie will not speculate far. How much of an architectural change survives scale is an empirical question, and he likes this era of machine learning for insisting on benchmarks over stories. Even a new stack that merely matches perplexity while fitting academic GPUs or raising MFU is useful.

One more nugget from the paper: within a compute budget, pushing data rather than parameters helps, the same overtraining trend visible in Llama 3 versus Chinchilla-optimal token counts. Hie's intuition is superposition. Models have more capacity than the parameter count suggests, because the same neuron can participate in many concepts, and a smaller model stuffed with tokens is cheaper at inference. That last point is not academic aesthetics. When they build models for biology they do the arithmetic: biologists want to run the thing, a trillion-parameter or even 100-billion-parameter release would sit unused, and 7B is already large for much of the biological community. Overtraining a smaller model is a way for the compute-rich to spend flops so the compute-poor can actually use the result.

Evo, paper title *Sequence modeling and design from molecular to genome scale with Evo*, uses the StripedHyena hybrid: mostly state space, attention sprinkled in, long context that the state-space layers make affordable, trained purely on DNA. Labenz's reading is that it answers a standard skepticism. First people said language models were only next-token parrots; that has mostly been put to bed. The next dodge is that any concept they learned, they learned from us, because we wrote it down. Evo was not handed those write-ups. If it has higher-order biological concepts, it got them from the raw sequences.

Hie had not framed it that way, and he likes the dimension. Biology 101 is nature versus nurture; Evo does not model the environment explicitly. The environment still leaves an imprint on the distribution of DNA the way the world leaves an imprint on internet text. Sequencing is cheap, high-throughput, and coupled to almost every experiment, and DNA is the fundamental inherited modality, especially in microbes. Another advantage is blunt: human-level performance in biology is already abysmal, so beating it is not hard, and nature has produced things far beyond human imagination, from deep-sea vents to pathways in our own cells that remain unknown. The aim is a model that learns higher-order concepts in a fairly unbiased way across the diversity of life, from a data-generating process called evolution. The hard remaining problem is extracting those concepts and turning them into discoveries.

The gene-essentiality experiment is the concrete proof of usefulness. Some genes, removed, kill the organism; others can be deleted and the organism looks fine. That map matters for basic biology and for drugs: you do not antibiotic-target a nonessential gene. Evo's context window is 131 kilobases, long enough to hold some bacteriophage genomes whole. For each gene the team inserts a premature stop, a three-letter knockout near the start that would prevent translation, then scores the full sequence's likelihood with and without the insertion. In nonessential genes the likelihood barely moves, as if the model were saying the mutation is fine. In essential genes it drops sharply. Those likelihood changes predict experimental knockout databases the model never saw. Adding genomic context around a gene improves the call, so the model is using more than the gene in isolation. For one phage, a CRISPR knockout of every gene and a handful of Evo forward passes yield essentially the same essentiality map.

Labenz immediately sketches a broad-spectrum antibiotic: run the procedure across pathogenic bacteria, collect genes that are consistently essential, then take those to the wet lab. Hie agrees that is one application. He thinks of Evo as a conservation score on steroids, a density model over tens of thousands of bacterial genomes and millions of phage genomes, with a sense of which genes evolution does not allow you to touch. The parrot objection still comes from biologists: maybe it is only memorizing co-occurrence. The test his lab is running looks at regions that are *not* conserved in sequence but *are* conserved in function, especially bacterial immune systems, which diversify so viruses cannot lock on. If the model can generate genes that look like nothing in nature or in the training set and still do immunity, it has the concept, not the string. The data look promising; he says the public, knockdown evidence is not there yet, and the bar should stay high.

Interpretability is the other extraction path. Labenz imagines sparse autoencoders on a future Evo trained on trillions of tokens. Hie wants that work for three reasons: evidence that the model learned higher-level concepts; classifiers for free, by reading a concept neuron instead of fine-tuning; and sample-efficient discovery. Known CRISPR-Cas systems number something like a handful to about twenty. If you could find a gene-editing neuron, clamp its activation high, and mine the outputs, you might pull systems nature has not shown us yet. The paper is already adjacent to that hope. The other headline is that Evo can generate CRISPR variants.

Why more CRISPRs? Hie's troll answer is patents: some filings draw a circle at 80 percent sequence identity, and a generative model is a way to step outside it. The serious answer is that CRISPR is used in highly unnatural contexts, including human cells, where a chance variant from nature, or from a model, may work better than the canonical one. The old method was to sequence more bacteria from vents and sewers. Sampling a generative model is a way to get diversity without that field trip. Throughput, not imagination, is the bottleneck. His lab can seriously test maybe 10 to 15 CRISPR systems in reasonable time and money. A human expert mutating 400 amino acids to bust a patent would succeed at an astronomically low rate; a generative model might get 10 or 20 percent of designs to work. In bio-ML that is called low-n design: test ten things, have one or two work, and you have already won relative to brute-force screens. The lab remains the final check, because the model will hallucinate.

On safety they excluded eukaryotic-infecting viruses from the training data entirely, a significant omission Hie treats as prudent at this stage. Future versions, he says, should have evals on common viruses and toxins, and if the model understands those sequences too well, post-training should make it bad at them. Labenz puts the dual-use worry on the table: the medicine generator and the weapon generator may be the same object. Hie's reply is that it is much easier to make biology worse than to restore it. A motivated state actor can already build a bioweapon without generative AI; he would not use a generative model for that job now. What we cannot do well is design cures and vaccines, return a system to homeostasis, reduce entropy. Hitting an organism with a bus kills it. Nuclear radiation causes cancer. The bar for harm was cleared long ago. The bar for repair is still high, and that is where he thinks the capability jump lives, provided the guardrails stay up.

The third paper, *Unsupervised evolution of protein and antibody complexes with a structure-informed language model*, completes the trilogy by locking onto a known target. The model is ESM-IF1, an encoder-decoder that Labenz compares to machine translation: the encoder reads three-dimensional backbone coordinates, the decoder predicts amino-acid identities compatible with that backbone. It was trained only on single chains, and not even on large multi-domain proteins, mostly on small globular domains. Labenz wants to know whether using it on complexes was a fight or a gift. Hie says it was somewhat unexpected, an emergent property.

The method is almost rude in its simplicity. They treat a multi-chain complex as one huge protein whose coordinates have large discontinuities, and feed the whole thing in. The model still appears to know which mutations stabilize or destabilize the complex, which in practice means raising or lowering binding affinity among the parts. Antibodies have a heavy chain and a light chain and typically bind an antigen of one or more chains, so at least three chains sit in the structure. They give the model the full set of backbone coordinates of antibody bound to antigen, ask which antibody-side mutations stabilize the pose, and many of those mutations improve affinity. That is the slot in the molecular-development cycle where you have a weakly binding antibody from a patient or a library and want it ten or a hundred times better.

The scoring loop does not retrain anything. Antigen sequence is held fixed. At each antibody position they enumerate single amino-acid changes, twenty possibilities times sequence length, and rank by how the full-complex likelihood moves relative to wild type. Round one tests the top 20 or 30 point mutants individually. Round two takes the five or six that actually improved binding, scores combinations with the same frozen model, and picks the best. The readout is a concentration: the equilibrium at which half the molecules are bound and half are dissociated. A 25-fold improvement means you need one twenty-fifth as much antibody to reach the same bound/unbound balance, which can translate into dose. Hie also mentions improvements on the order of 100-fold depending on the complex. Labenz flags it as an even starker out-of-distribution generalization than he had realized.

With time almost gone, Labenz offers two last questions and lets Hie take both. The first is overfitting in a world where the target evolves too. Fit an antibody too tightly and a small change on the other side could throw it off. Hie says that is exactly the direction he wants to go. Therapeutic design usually assumes a fixed target. In viral disease, bacteria, and cancer the target is moving, constantly evolving around the therapy. Spend $2 billion to develop a drug, administer it, and six months later you have resistance. Tree search, an RL policy, or something more basic from the opening causal questions all look like open algorithmic space. Static-target design has come a long way. Moving-target design has not.

The second question is the ARC Institute, which keeps turning up, including a recently announced CRISPR successor. Hie is on site as they talk. He has been a professor for only eight months and does not think the work would have moved this fast without ARC's support and talent density. What strikes him most is collegiality: investigators are friendly, and he has ongoing projects with the others in some form. He notes, smiling, that ARC funds some of his science, so the praise is technically sponsored, and that the ad was not. Labenz's sign-off is that if they cure cancer they will deserve the fame. Hie's closing register matches the safety argument from earlier. Harm's technological bar was cleared a long time ago. Repair's bar is still the one worth raising.