AI Beats Humans at Reading Papers. It Still Cannot Count Atoms.

Nathan Labenz has been pestering Andrew White for months. This is White’s first podcast. He is a professor of chemical engineering at the University of Rochester and, more recently, co-founder and Head of Science at Future House, an Eric Schmidt–backed focused research organization building semi-autonomous AI for scientific discovery. The two men first met on OpenAI’s GPT-4 red team in August 2022, which is how Labenz knew White had caught the wave earlier than a typical chemical engineer’s LinkedIn would suggest.

White’s training is statistical mechanics: simulations of gases, liquids, proteins, and quantum systems with enormous degrees of freedom. A 2019 sabbatical at UCLA’s Institute for Pure and Applied Mathematics put him in a room with Yann LeCun, a visiting Yoshua Bengio, Pat Riley from Google Accelerated Science, and chemists already building state-of-the-art energy predictors for small molecules. A Tokyo materials meeting with Sergei Kalinin and Lee Cronin followed. He came home and wrote a textbook, *Deep Learning for Molecules and Materials*, as executable Jupyter Book course notes.

The paper that rewired him was from Gerbrand Ceder’s group: word2vec embeddings of materials science literature, so a material could be a vector in language space instead of a hand-built feature set. With Glen Hocky at NYU he started driving VMD, a molecular-dynamics visualization tool written in the obscure language Tcl, using OpenAI’s Codex. Nobody on the team could write those scripts; the old workflow was Google, copy-paste, and emailing friends. They published a short position paper in *Digital Discovery*. OpenAI, hunting for people who knew language models *and* chemistry for CBRN safety work, found a very short list: White and Hocky.

On the red team, White asked GPT-4 how to synthesize a nerve gas. The steps looked coherent and alarming. Then he drew the intermediate structures and traced the atoms. It was all hallucinated. The model had no conception of how to move atoms between molecules; it was really good at bullshitting. The first month he thought the system was crazy. October and early November, he decided it was a nice incremental improvement. Late November, after he hooked tools to the model—LangChain-style ReAct loops, still often on DaVinci—he changed his mind again. Mitigations fell apart once tool-calling started. Put the model on rails of *only* using tools, and it could do respectable chemistry. That path became ChemCrow.

What still surprises White is the shape of the remaining errors. Ask a frontier model to reason through reaction rules or a molecule’s properties and it is, in general, quite good. Ask it to count electrons, count atoms, or get the stoichiometry right and it still misses. Labenz had the same experience using GPT-4 as a chemistry tutor: the two of them would get lost together in a mess of wrong geometry. The models have since become passable tutors, but the pattern remains. They struggle with the robotic steps and do well with the free-form ones—the opposite of how people expected machines to learn STEM.

White’s prediction is a grokking effect. Once a model can count atoms and electrons reliably, chemistry unlocks, because the extra reasoning was never the bottleneck. That is why he is optimistic about o1 and the rumored models after it: they might remove the need to keep molecules on tool rails. He was on the o1 technical report. This round of red-teaming was more systematic; Future House’s own LabBench, covering biological and laboratory protocols, was already in the evals, so the team spent its time on wilder exploratory ideas. The models got very far and then hung on one key step.

They did not get to try consensus—run k=32 and take a majority vote—which, on other evals, smooths exactly those little missteps. His read is that the model available at recording time is not there yet, but the next release should support synthesis planning and protocol planning. OpenAI’s report already showed o1 above human level on ProtocolQA, Future House’s assessment of laboratory plans in biology. That is strange on its face. Biological sequences are long and often do not fit in a context window. The ancillary knowledge—how you clone, what the steps in a process are—is what has crossed the line first. Chemistry, where the atom-counting still bites, has not.

Future House picked biology for reasons that track Dario Amodei’s account in *Machines of Loving Grace*: a small number of platform technologies, used over and over. If you want a protein, you clone it, you do cell-free synthesis, or you print it on a machine. There is no step where the field shrugs and asks whether *this* protein can be made. Chemistry is the opposite. Every molecule is bespoke, and scale-up is never trivial. Sequencing is almost free. Synthesis is cheap enough that a hundred-amino-acid protein starts with a $25 gene order. Hypotheses are clear to test and cheap to test.

The other reason is that the intellectual work does not run out. There is always another genome to annotate, another protein with an unusual function, more metagenomic dark matter whose genetic material nobody can name. Physics is expensive to instrument and, at heart, reductionist: you are hunting for the law. Biology already has its universal law, evolution. What remains is to understand how complex systems fit together. Labenz asks how much “dark conceptual matter” might still be sitting out there—Michael Levin’s bioelectric tissue-level interface is one candidate that still looks niche. White’s answer, via Ed Boyden’s story of inventing optogenetics, is a tiling tree: write the goal, enumerate every way to send a signal into tissue, and cut the idea-space into paths. The field will not run out of those paths.

People like to say that a crystal structure means you know a protein. Then you need the statistical ensemble, then post-translational modifications, then a water wire that changes how protons diffuse to the active site. “You will never be able to reduce biology to these cartoon diagrams,” White says. Complexity at every level always plays a role. That is why he treats biology as a domain of observations and empirical measurements, not a virtual model you can derive. It also, in his view, stops the AGI story in which a model wakes up and knows how to cure cancer by thinking. Superintelligence may not scale as much as a large set of good hypotheses you can actually run.

The organization itself is Sam Rodriques’s brainchild. Rodriques had already written down focused research organizations with Tom Kalil and Adam Marblestone at Convergent Research: five-year nonprofits, funded at $20 million to $50 million, aimed at one question too big for academia and not commercial enough for industry—new model organisms, connectome technology, Lean scaffolding for mathematics. Future House keeps that scale and stretches the horizon into a moonshot you may not finish in five years. White argued they should wrap the whole thing around automating science rather than a new microscope. They pitched Eric Schmidt, who had been excited about AI, and built a mission-driven nonprofit on purpose: without a profit motive, a mission is what tells you whom to hire. The ten-year mission on the website is to build semi-autonomous AIs that scale scientific research and open world-class scientific expertise.

Amodei’s ten-year picture has AIs as the PIs and humans as helpers. Future House’s “semi-autonomous” is a deliberate downgrade, and White is bearish on laboratory robotics. Too many biotech and AI companies swallowed “automate the physical world” as a mission and died of it. He is careful not to say Emerald Cloud Lab or Ginkgo Bioworks failed—they have made real progress—but full automation is a grand challenge Future House refused to take on. They use the ordinary kit: robot arms, liquid handlers, acoustic dispensers. If someone solves lab robotics tomorrow, they will buy it or sign a partnership. It is not in the mission.

The semi-open secret of cloud labs is that the programmable interface sits on a mix of robots and people. The industry name is biorobot, or wet robot: a human doing the two-minute task that is not worth automating. White is not a purist. Lights-out automation—turn off every light in the building and the system still runs—is the clean test, and almost nobody is there. Getting from 98 percent to 100 percent is a lot of work for very little throughput. Companies such as Medra are trying a different path, human-gripper arms that roam an ordinary lab instead of bespoke instruments. That last step, he thinks, will stay unautomated for a long time.

The other half of “semi” is the scientific loop itself. White does not expect a system you can tell to explore the metagenome and collect in a year. More likely: a patient population, a question about mechanism, then an iteration about which mechanisms a biologic can hit and which a small molecule can. Math can, in principle, be sent off to dump theorems once a quarter. Biology will not let you order 36 mouse measurements from a CRO API. Labenz floats a Science Dojo of tools, including “run a lab experiment,” dispatched to a person. White likes it. He also wants the control experiment nobody has run: wrap drug discovery and literature in about 25 programmatic tools, then put a PhD student or a Mechanical Turk worker on the other end instead of a language model, and see whether humans are any good at science-as-API.

Labenz’s mental model had been that better AlphaFolds and ESM models raise hit rates by 10 or 100 times, so existing wet-lab throughput suddenly produces far more value. White has not settled the question, but he has a bill. Accurate free-energy calculations of whether a small molecule binds a protein cost about $10 to $15 on the cloud. Making the molecule and testing it against the protein is about $20, and can be pushed toward $5 depending on the protein and the catalog. Chemistry, biology, and computation have been getting cheaper in parallel. You can bet on Moore’s law and better models for a future crossover. You are not clearly past it.

Zoom out and the cautionary tale is D.E. Shaw Research. Fifteen or twenty years ago they packed hardware engineers, programmers, chemists, and biologists into a Times Square building and built chips and a data center from the atom up, chasing milliseconds of protein-folding simulation. Extrapolate the charts and by 2030 you simulate an organelle, by 2050 a whole cell, an afternoon per division cycle, a virtual cell you never have to wet-lab. Molecular dynamics does not do chemical reactions. A lot of biology is acid-base chemistry, ATP to ADP, NAD+/NADH. Reactive force fields, including machine-learning ones, still miss that a proton moving through water is a quantum effect: you need electron-density calculations and the Born–Oppenheimer approximation, and biology actually behaves differently without it. You never reach the floor of complexity.

AlphaFold solved a great deal; large stretches of proteins are still intrinsically disordered. Coarse-grained force fields can capture another slice and then fail on the fraction where the coarse model is not useful. Sometimes empirical machine learning works. Sometimes first-principles methods look as if they will. The only thing that works 100 percent of the time, White says, is measuring it in the lab. Everything else is an approximation that raises your hit rate or shortens the experimental loop. Synthesis, meanwhile, is closer to solved than a chemistry undergrad would believe: virtual catalogs of some 50 to 100 billion makeable molecules, an 80 percent fill-rate, molecule transformers from Philippe Schwaller’s group, commercial synthesis planners from IBM and PostEra. A chemist in China or India, fully loaded at about $100,000 a year, can make around 20 molecules a week. Then PROTACs, molecular glue degraders, and induced-proximity drugs pack two or three jobs into one fat molecule, and you are back to the complexity of thirty years ago.

The industry also changed its starting materials. Old phenotypic discovery ground dirt and frog skin, found something active, and then faced a natural-product synthesis from enzymatic chemistry. Much of medicinal chemistry moved to petroleum-derived building blocks and a toolkit of perhaps 75 routine reactions, which is why catalogs work and why some people think chemical diversity was lost. Efforts such as the COCONUT natural-product database try to pull generative models back toward that older space; Octant does phenotypic screening in cell models with the newer, easier compounds. On multi-omic foundation models—Recursion, Insitro, Calico’s big matrices of drugs, transcriptomics, and cell images, or genetic models such as Evo—White is colder. They will be good at metagenomics, transcription factors, and regulatory networks. As a silver bullet for disease, every beautiful idea in this lineage has failed, from the Human Genome Project through GWAS. Cynical is the default. He points listeners to Derek Lowe’s blog.

Labenz objects that machine learning also looked like a graveyard of 1980s ideas until scale arrived. White’s deeper reply is latency. In ML you find out fast, and scale itself filters clever ideas that die at 10 billion parameters or a trillion tokens. Biology’s feedback lengthens at every rung: a protein in a well, then a cell, then tissue, then a mouse that has to live long enough to age, then seven years from mechanism to Phase 2. In seven years the team has turned over and forgotten why the project existed. Insitro took about seven years to get a first platform asset to the clinic. Recursion’s clinical assets, at the time of recording, had been bought, not generated by the platform. Atomwise had a good idea and good people and could still die from the wrong target. If there is a biological bitter lesson, we may not know for thirty years. The bottleneck White would fund instead is unglamorous: time to enroll the first patient in a trial, about 60 days, which is amenable to better ads for trial sites. And if a regulator wanted to help, it could release IND packets for every already-approved drug—expensive toxicity, pharmacology, and PK data, no longer competitively useful, sitting at the FDA for no good reason.

Future House’s lineage starts with ChemCrow: GPT-4 plus retrosynthesis predictors, literature search, and code execution, asked to design a novel two-step dye that absorbs at a specified wavelength. The agent fit a model to a paper’s supporting information, searched, planned a procedure, and ran it on IBM’s RoboRXN cloud lab. The product landed about 15 nanometers off. A person had to finish the last step because the robot ran out of time; other examples in the paper were fully robotic. What White took from it was not the robot. Scientific literature is a centuries-old networked artifact, and in his accounting 99 percent of doing science is knowing what is in it. PaperQA came first, written in a rainy Copenhagen hotel before Future House existed.

Ordinary RAG retrieves chunks and stuffs them into a final model. PaperQA pulls every possibly relevant chunk, runs a language model on each one to summarize and rerank, then answers. That extra map-reduce step exists because models are extremely sensitive to distracting context. Ask for the size of the placebo arm in a named trial, retrieve the methods section of a different trial, and the system blows up. The same step also washes out a lot of chunk-size and parsing pain. After a long grind on what counts as a correct answer—documented in an engineering blog of A/B decisions—they beat humans on literature Q&A, contradiction detection, and Wikipedia-style cited summaries. WikiCrow then wrote about 18,000 articles on the function of every gene in the human genome, except 1,500 to 2,000 with no papers at all. Wikipedia had covered roughly 2,500. The rest of that knowledge did not exist in one place until they built it.

At 75 questions a minute they can check an arbitrary claim against a corpus White puts at 250 million papers, scan every new arXiv paper each day for disagreements, and rewrite the Wikipedia article for every disease every three weeks. The philosophy, which Labenz wants more builders to steal, is to ignore cost and latency until the quality is real. Google amortizes a huge index into cheap queries; PaperQA does the processing just in time. A question costs about 15 cents to a dollar and takes a few minutes. Full-text search is the other unglamorous unlock: most paper search engines, and therefore most benchmarks and competitors, only see titles and abstracts. Future House built its own full-text index, open-sourced a Postgres or Elasticsearch version in the PaperQA2 repo, and still leans on Google Scholar and Semantic Scholar. Considering 75 sources and cutting to 10 beats considering 25 and cutting to 15. Query expansion into several differently phrased keyword searches—the trick behind Perplexity’s Pro Search—matters more than a clever agent prompt. Complexity lives in tool docstrings; the control flow looks like a first draft.

Figures are catching up. On Future House’s FigQA benchmark, the October Claude Sonnet already beats humans. PaperQA2 version 6 attaches page renders to chunks rather than fine-tuning a PDF parser: text is for search, the picture of the page is for answering. White would rather spend money and time at low technical sophistication than maintain Detectron-style layout models that fail on a 1992 PDF. The mindset shift underneath is recall versus precision. Ask a model to dump a table to JSON or list every number in a figure and it is mediocre. Ask whether treatment A improved response, and it is superhuman. The same pattern holds for a 100,000-token document: “list the 50 main conclusions” is messy; “what is conclusion one, then two, then three” works. hasanyone.com is a public sliver of the internal stack—“has anyone done X”—and the GitHub repo’s 6,400 stars sit behind a deliberately painful install, because the team only wants to support programmers. Commercial inbound is real (diligence, patents, PhDs). They have not decided whether API revenue produces more novel discoveries, which is the metric they actually optimize.

The next move was to split PaperQA in two. The environment is the untrained part: tools such as citation lookup, paper summarization, Google Scholar, keyword search over a corpus. The agent is the trained part: the core model plus memory, the thing that decides to search again or inspect who cited a paper. Anything you want to train goes in the agent; anything you will not train goes in the environment; they talk in language, observations in and actions out. Memory sits with the agent on purpose. Every memory design has hyperparameters—how you truncate, how you compress, whether prior images stay in context—and those knobs want to be learned. Environments now go beyond literature: ChemCrow-style chemistry, protein design, molecular cloning. An environment plus an agent is a “crow,” because crows talk and use tools. Aviary is the gym of those crows, aimed at an API for doing science.

They represent agents as stochastic computation graphs, a framing from a John Schulman paper. There is no recursion and no hidden state machine. The graph runs left to right once; the agent’s previous output state is the next input, along with a new observation, and the outputs are a new state and an action. That lets them backpropagate, execute efficiently, and serialize without nursing a program’s internal state. Almost every agent people ship today is zero-shot: tweak the prompt, maybe the temperature. In that regime the framing does not matter, and you do not need a framework. Aviary is built for training, including online PPO. White says DSPy is the closest cousin, but without a real environment or RL.

Closed models still sit on key nodes. Sid and Albert—Albert is first author on PyTorch RL and previously did RL at Generate—extended Schulman’s graphs with black-box gradient estimates. You cannot backprop through Anthropic or OpenAI, so they fit a small MLP as a local surrogate: several calls at temperature 1 to estimate aleatoric noise, then small input nudges to see how outputs move, then backprop through the MLP. It costs on the order of 25 calls per input. In the paper they show it can tune temperature or a punctuation hyperparameter in the prompt. White calls it a technical demonstration more than a working optimizer. Upstream knobs are weakly correlated; once you know temperature 0.2 is best, there is little left. The strategy that actually works is hybrid Q-learning: the closed model generates about eight candidate actions, an open-source Q model trained for the task picks one, and you never step into the environment until that choice is made. Tree search with a branching factor of two, duplicating the environment, is how they harvest rare positive-reward trajectories to train on.

The gains follow how formulaic the task is. Literature question-answering, where items can be “consider these five papers and tell me which one is different,” improves about 5 to 10 points. Molecular cloning, a narrower action space, goes from roughly 50 to 70. Before Aviary, White says, they felt helpless: drop in GPT-4o, change the prompt, add in-context examples, and if it failed there was no way to gather more data. Training still does not yield a universal agent. A Q model tuned on multiple-choice literature questions will not write a literature review. You can write down 150 or 500 tasks and get good at those; tasks off the list are unclear. He also doubts that a single huge test-time model will be runnable on every lab task at GPT-4o prices. The more likely pattern is an expensive model generating ground-truth rollouts or plans, and a cheaper model executing them—the same spend-what-it-takes instinct as PaperQA, now applied to a thousand four-minute, fifty-cent jobs that a human scientist would never do, such as a combinatorial sweep of every antibody against every reported disease in a given tissue.

Asked what needle he is watching, White does not start with Mustafa Suleyman’s long-term memory pitch. He starts with diversity. On a benchmark associated with Aiden, you ask a model for a hundred explanations of what caused World War II and, after about four, you are reading restatements with different punctuation. Beam search and top-k sampling do not yield a hundred hypotheses; they yield one or two ideas in many costumes. That caps how many scientific ideas you can explore at once. The other needle is whether bots can still touch the internet. Reddit and Stack Overflow, once training gold, are hostile. X closed. Bluesky is already overwhelmed. Open-access PDFs are often no longer accessible programmatically, even when the license says they should be, because anti-bot pages sit in the way. White would happily put a quarter in every site he needs to scrape if that paid for egress. He is not sure the next generation of AI systems should keep treating “the web” as their world model.

What Future House wants next is the playbook Aviary unlocked: environments and agents interacting in open-ended science demos. They already work with outside groups on the PaperQA API and want more of that. They have not solved the nonprofit distribution problem. Semantic Scholar, out of the Allen Institute, and Crossref both struggle with how to give a free intellectual service a sustainable shape; Future House is still exploring. Success is novel discoveries, whether those come from the in-house team or from every scientist on earth using the tools to speed search, peer review, or cloning protocols. They are hiring people who want to revolutionize a field, and they want academic groups willing to try the stack on their own questions.

Labenz comes back to the thing that rattled around after the conversation: because conceptual frameworks and automation platforms are making real experiments cheaper, in-silico models may prove less transformative than a generation of ML people expected. White’s closing picture is not ChatGPT-with-lower-latency. It is high-throughput, relatively shallow intelligence distributed at scale, plus a little very expensive intelligence on a few nodes, stitched with map-reduce over tens of thousands of papers. That is a different branch of science than the one a PI does by hand. It is also why a system that is already superhuman at reading the literature, and still unreliable at counting atoms, can be exactly on schedule.