The experts argued through the coffee break — and never agreed whether AI can build AI

This is part two of a marathon Cognitive Revolution live show that Nathan Labenz co-hosted with Prakash, known on Twitter as @8teAPi, on February 11. The first half, already in the feed, had gone to Stanford’s James Zou on AI for science, Sam Hammond on U.S. competition policy, and Shoshana Tikovski on agents in the wild. This half stacks three more guests who do not share a beat: Abhi Mahajan on AI for biology, Helen Toner on recursive self-improvement, and Jeremie Harris on U.S.–China competition and loss of control.

Mahajan, who writes as Owl Posting, works on AI for cancer at Noetik AI. Labenz introduces him, only half-joking, as the Zvi Mowshowitz of the biology intersection. The company collects real human tumor samples and trains foundation models to predict which patients will respond to which treatments. Mahajan is skeptical of a lot of published AI-for-biology results and still expects the trend lines to keep moving until the field is actually transformed.

The number that organizes his pitch is ugly. Ninety-seven percent of oncology trials fail. You can read that as a drug-design problem and try to invent better candidates. Mahajan reads it as an evaluation problem: even failed trials usually contain a pocket of responders, and nobody has a usable way to find them in advance. That is the wall the first conversation keeps hitting. The later two conversations hit different walls — disagreement among insiders, then a physical stack built on someone else’s components — but the pattern is the same. The grand AI story keeps running into a concrete constraint.

Labs are raising serious capital on a bet that AI scientists will design experiments, run them through automation, and learn from the results. Mahajan has written that biology does not offer much verifiable ground truth. He admits the slogan is a little hyperbolic. What he will defend is the narrower claim: there is not a lot of verifiable ground truth for the most clinically valuable problems. You can verify whether a protein is in solution, or whether a sequencing variant is real. You will not, he thinks, see the same intelligence explosion biology that math and code got, because those fields have cheap rewards and biology does not.

His analogy is training a model to pick bestsellers. Sales exist, the author’s country exists, the data is public, but you wait eighteen months for one point and then cannot attribute it to any single variable. Toxicology is the biological version, and it is what sinks most Phase 1 drugs. A compound can be toxic in seconds, like snake venom, or over months and years. It may not kill the animal at all. It may cause cognitive deficits or heart damage, and it is often dose-dependent and species-dependent. Those measures have no real path except observing them in vivo.

An San Francisco startup he names, Axiom, is trying to predict a small molecule’s toxicity on hepatocytes in a dish. That is a clean problem and could save months in preclinical work. It still does not answer how the drug behaves in an animal. Prakash points to Isomorphic Labs’ new model, which roughly doubled AlphaFold 3 on binding affinity, pocket identification, and structure prediction. Mahajan calls the leftmost benchmark extremely hard and the result impressive. Then he cuts: the field is already awash with good preclinical assets, and the bottleneck is how they work in patients. The story that better design automatically means better human outcomes has been told for ten years, he says, without strong evidence. A chemistry paper from a few days earlier, *The Affinity Advantage*, is one of the strongest bull cases that optimizing every preclinical facet has superlinear benefits. He does not share the view. He is sympathetic to it.

After the break, Labenz asks Mahajan to double-click on Noetik. The economic pitch starts from those failed trials. Look at the papers published afterward and you usually find some patients who did respond. Researchers then hunt for the biological archetype of that group and come back with something complicated and heterogeneous — a cytokine signature, high granzyme expression — that never turns into anything useful. One argument, Mahajan says, is that the biomarker defining response for that drug is non-human-legible. You need a black-box biomarker to encapsulate whatever that information is. Noetik is built around that thesis.

The company profiles human tumors on four modalities at once: pathology slides, the blue-chip almost everyone has; a 16-plex spatial proteomics panel to identify cell types; whole-plex spatial transcriptomics covering about 19,000 genes across the tumor surface; and exome sequencing for alterations such as KRAS-positive status or an STK knockout. The machine-learning bet is a self-supervised masked model. Place any incoming tumor in the universe of cancers the model has seen, and ask whether responders and non-responders fall in different regions of embedding space. If they do, Noetik may have a biomarker no human understands.

The more interesting move is generative. Knock out a transcript or a gene and watch how expression in the tumor microenvironment changes. Cancer literature has started talking about “nudge drugs,” agents that do not hit the immune system or the tumor directly but push the site into a state other drugs can exploit. In the thought experiment, you knock something out, then hallucinate adding Keytruda — a PD-1 checkpoint inhibitor — and the model tells you the tumor is hot, inflamed, likely to melt. Labenz asks about interpretability, pointing to Goodfire’s work on Alzheimer’s biomarkers. Noetik has an internal mechanistic-interpretability group. Mahajan is not sure it should be the priority.

A year ago, he says, you might have cared because the FDA gets upset at black boxes. Around August or September 2025, a pathology company called Artera AI brought a companion diagnostic that takes a prostate-tumor slide and predicts response to androgen-deprivation therapy. They have no idea why the model works. They retrospectively validated it on thousands of patients from prior Phase 3 trials, and the FDA was fine with that. Everyday mechanistic interpretability in biology, as he describes it, is staring at semantic-segmentation plots and asking whether the model found a real signal or a spurious correlation. That time, he says, feels better spent elsewhere. He is optimistic that human-simulation companies can help a few trials in the next few years from capabilities that already exist. He is much less sure they will discover brand-new targets, which is what people actually care about, unless someone builds a genuine human biology simulator — which is Noetik’s bet.

Zoomed out, Mahajan sees three ideological camps. The first, Noetik’s, indexes heavily on human data: you cannot bootstrap from in-vitro settings to the real thing, so you start with the most complicated object. The second models single biomolecules and their interactions, hoping to lift success rates from maybe 5% to 20%. That camp, he says, is most of today’s ML-bio startups; some have clinical candidates in play, and there has not been a massive step-change yet. The third is barely a for-profit thesis at all: improve the clinical-trial process itself. That, he argues, is China’s main advantage.

China can run trials far more cheaply, partly from labor costs and partly because the system is set up so that getting started is less of a regulatory and financial headache. The downside is that drugs are treated as innocent until proven guilty, the reverse of the FDA. The benefit is that you are betting on neither AI-plus-human-data nor AI-plus-in-vitro data getting better. You are betting that a slightly more financially efficient version of ordinary drug design will improve outcomes. He refuses to assign unequal weights. All three camps feel important.

Prakash asks what is driving the flood of new INDs from China — AI, more hands, regulation, cheaper U.S. costs. Mahajan says it is not his expertise. The story he finds most interesting is path-dependence: China started as a strong generics manufacturer, extended into a mature CRO ecosystem around WuXi, incubated enough talent, and then asked why not develop its own drugs. Having the person who designs the molecule sit close to the person running wet-lab assays matters more than people think. In the U.S., the loop is long: assemble a slide deck, find VCs, buy a lab. In China the ecosystem is already there, even if the VCs are more risk-averse. He heard a similar colocation argument from Soham at Popfax, one of the few people doing novel biotech in India, who expects compounding in India and Egypt for the same reason: intellectual capital plus hands.

As for a DeepSeek moment in Chinese bio-AI models, Mahajan cannot see one. There is interesting work. It is not clearly radically new, and a fair amount looks like scaling up ideas that started in the U.K. or the U.S. American visibility into those labs is much lower, so he could be wrong. Dario Amodei’s *Machines of Loving Grace* vision of scientists in a data center churning out ideas does not move him much. There are already tens of thousands of PhD students churning out good ideas, he says. Most of them cannot be validated because validation is too expensive.

Labenz wants a way to calibrate. Headlines say AI discovered an antibiotic, or designed nanobodies in a virtual lab; the other side says most of it will not pan out. Even programming, a more legible domain, produced a METR study showing slowdowns that still confuse him. How should a careful outsider climb the curve in biology? Mahajan points to an old post of his, “Five things to keep in mind when reading biology ML papers.” The short version: evaluations in biology are very difficult, and papers often hide confounders that would be obvious only to a domain expert.

Wet-lab biology has the classic “we cured cancer in a mouse” problem. Biology ML has a cousin. His favorite example is small-molecule binding-affinity papers. You train a model to separate binders from non-binders, it looks great, you publish in *Nature*. People at Leash Bio found the result can be confounded by which chemist made the molecule. Some chemists are attached to particular targets, they are good, and the molecules they make often bind — and, crucially, look very similar to one another. The model may be learning that human, vibe-based similarity rather than a real binding rule. The same class of hidden variable, he says, shows up over and over in in-vitro biomolecule generation.

Can an LLM catch those holes? He wrote another piece: could o1-preview find mistakes in 56 MLSB papers, the structural-biology workshop at NeurIPS? It was not very good. That was a last-generation model, so maybe current ones are better. Recurring hygiene issues — train set too small, test set not stratified — an LLM can flag, and then you pick your battles. More fundamental problems, in his experience, the models miss entirely. The fun pattern is that in almost every article he writes, the LLM tells him something about a subfield that domain experts completely reject. They say that is not how you should think about the domain, and that is not the problem they were worried about. Biology, he thinks, is a science whose human taste LLMs have not quite captured.

Helen Toner, who runs work at CSET at Georgetown and is a former OpenAI board member, comes on next. The report is *When AI Builds AI*, built from a day-and-a-half closed-door workshop in July 2025. Labenz notes that this fight is as old as the field — he was reading Eliezer Yudkowsky on intelligence explosions in 2007 — and that even as “long timelines” have gotten short, disagreement on recursive self-improvement still seems impervious to new evidence. Toner calls the workshop a highlight of her year. Frontier-lab people, policy people, and researchers were in the room.

The first session was how AI is already being used to automate AI R&D. Before the first break, Ryan Greenblatt of Redwood Research, Nicholas Carlini of Anthropic, Sayash Kapoor of Princeton — of the “AI as normal technology” line — and Thomas Larsen, one of the *AI 2027* authors, were arguing so fiercely, in a friendly way, that everyone else stood up for coffee and snacks and they just kept going. Toner tells the anecdote because they later posted about the part they agreed on. Carlini kept asking where the first empirical disagreement actually was. Looking at 2026 and 2027, they agreed a lot about what you would see before the recursive point. That is nice, and also a bummer: it will be hard to tell in advance whether you are heading into a recursive loop or not.

Two core disagreements came out for her. First, does AI truly replace all of what humans can do? Without that, you do not get the scary recursion. You can have much more productive researchers — Alec Radfords and Ilya Sutskevers managing fleets of AI scientists — but if it all has to come back to them to digest, you do not get the loop. The report uses an Amdahl’s-law diagram: speed up coding and the bottleneck bites somewhere else. Skeptics also bring an expanding-pie model. We already stopped punching cards and writing assembly; AI writing more code is another inner band, and humans remain on the outer ring. The *AI 2027* view is the opposite: first automate some of what humans do, then all of it, then run until some other bottleneck hits. The second question is how soon those bottlenecks bite.

Prakash brings up Sholto Douglas’s “software-only singularity”: getting extremely good at coding and digital work, including AI research, without matching that in power, copper, and physical substrate. Toner hears two versions. One, closer to Tom Davidson at the Forethought Institute, is that software improvements alone produce crazy recursive loops and massive real-world impact. The other is a jagged software-only explosion: the system gets much more capable in some ways, but its effects stay limited because it is only software. The under-discussed question, she says, is what extreme skill at AI R&D implies for everything else. Some people think that skill means you can train a model for any task after a week of data collection. Others think even a superb AI-R&D system will still struggle to design new biomolecules or think through geopolitical strategy, because you have to go look at how countries and decision-makers actually react.

Labenz offers a position he rarely hears represented: maybe there is a plateau, and the plateau is already superhuman. Zoomed out, humans look like the entry to a steep part of an intelligence S-curve, not the end of history. Being somewhat better than what came before was enough to take over the world. It is hard for him to imagine that the remaining gap from today’s models to “somewhat better than us” is not transformative, even without a godlike singularity. Toner says that sounds close to her default expectation. A frame she uses, and did not put in the report, is three S-curve parameters: how long the lead-up is, how steep the middle is, and how high the ceiling is.

Most people pick the same pole on all three — short lead-up, steep curve, high ceiling, or the reverse. She wants the mixed cases: steep but low ceiling, or gradual but very high. Another neglected combination is a high ceiling delayed by real-world adoption. Sleep data is the running example from earlier in the marathon: lots of headroom in interpreting signals humans throw off, but you still have to collect the data and deploy the sensors. Superhuman ability and superhuman influence are separated by friction. Jaggedness, for her, is fractal. Zoom into “the task of AI R&D” and it is many tasks; some will accelerate first. On the public-private gap, her sense is it is not huge right now, and she has no inside information. Prakash has heard the gap is mostly the same model at lower batch size, about three times faster, plus tooling. Toner notes that AI researchers are unusually well placed to benefit: they know the failure modes, they shape the models, and they build the harness.

Prakash asks whether policymakers are naive on privacy. Ban facial recognition and you still have gait, voice, handwriting, network analysis, and WhatsApp metadata. Toner says the United States has done a worse job on this than almost every other country. She would not write rules at the level of “no facial recognition.” She would write them at the level of no data brokers: you can collect data, but the user needs notice and consent. She is not a privacy lawyer and does not pretend to have the finished proposal. She thinks there are better designs than the U.S. status quo that still leave underlying flexibility.

On insight into the labs, she thinks transparency is better than it was. California’s SB 53 and New York’s RAISE Act are starting points. OpenAI and Anthropic, and to a somewhat lesser extent Google, already put out a fair amount of information, and she wants them credited for that. She does not love that disclosure is still almost entirely at their discretion, though that should shift as the state laws start to be enforced. She also wants evaluation to move off a model-release schedule toward something more continuous, partly because internal deployment can carry risk that never shows up on a launch day.

Independent third-party audits keep appearing in proposals and keep getting stripped out by industry lobbying. That, she says, is a new frontier. Beyond the labs, the report’s “harden the world” bucket is societal resilience: cyber defense, bio defense, disease monitoring rather than people-surveillance, and epistemic security — tagging real content, telling real from fake. Assume capabilities get much better and that automated R&D speeds up the pace of change, then make those investments anyway. A mindset shift also needs revisiting. Over the past couple of years people started assuming open models would stay close behind closed ones, which relaxed worries about access gaps and concentrated power. If you take seriously the possibility that automating AI R&D significantly speeds up the closed labs, those assumptions have to be checked again.

Jeremie Harris of Gladstone AI wrote the first U.S. government AI threat assessment for the State Department. About ten months earlier, one conclusion was that every American AI data center was compromised. Labenz asks what has changed. Less than one might have hoped, Harris says, for interesting reasons. People in the risk community spend a lot of time on model-reconstruction attacks and whether models can even be secured if an API exists. More fundamentally, the entire AI industrial base is being built on components made in China, often with Chinese-national personnel. Forget the Manhattan Project, he says. We are so far behind that. His newer lens is the board itself, not the pieces. Are we playing on something that does not allow a winnable outcome? Closing your eyes does not fix it.

Prakash walks the supply chain: talent, Taiwan, South Korea, ASML in the Netherlands, thousands of suppliers, neon from Ukraine. TSMC has been upfront that it only works in a safe, globalized economy. Harris’s Taiwan scenario is stark. If Taiwan is invaded, TSMC is gone — either China takes it, or, as he would expect and hope, it is booby-trapped to blow. Think of it as a giant box of 500 dials, each of which has to be perfectly tuned to hold yields. Missing the equipment or the people, you do not replicate it. It may be the most fragile production process primates perform. The runner-up complex he takes seriously is SMIC plus Huawei. Huawei has emphasized networking large numbers of GPUs. The chips do not have to be as efficient as Nvidia’s if they scale together.

Energy is another bottleneck, possibly by the end of the year, and underneath it the grid. There have been components in Chinese transformers, he says, sneaked in as Trojans that can take gear down. A scenario people on the intelligence side take seriously: a Taiwan invasion begins, and one of the first things China considers is shutting down the U.S. grid. If that is on the table, you do not even get to play Samsung versus SMIC. You do not have an economy. On export controls, he is not moved by the argument that letting Nvidia sell into China will make Beijing ease off a national priority it has already identified and funded. A quarter-trillion dollars in PPP terms into national AI chip capacity does not get abandoned because H200s can ship. “Why not both?” is the obvious CCP answer, and so far it seems to be the one they are giving.

He thinks the controls have worked. The public evidence he wants remembered is from before DeepSeek was famous. The CEO was saying they really thought they could do the AGI thing, and the one problem was chips; export controls were killing them. Then R1 dropped, they got dragged in for a debrief, and the talk changed. Look at the size of H200 orders, he says, if you want a measure of pent-up demand. His biased take is that we should listen to Chinese companies when they tell us the policy is working.

Labenz puts the bind on the table. Harris has little hope for true U.S.–China collaboration, and little optimism that we can control a superintelligence. Meanwhile Anthropic and OpenAI have both, in the last week, said they could not run evals as intended — eval awareness on one side, autonomy tasks that are not long-horizon enough on the other — and shipped anyway. Labenz does not see China forcing that. He sees Western labs racing each other, down to Super Bowl ads. Harris agrees it is a race to the bottom. Domestic regulation of the kind lab leaders have talked about for a decade will not happen, he argues, unless you deal with the outermost loop: international competition. Pretending other countries do not exist is the same target fixation as ignoring infrastructure.

If the next model can design custom bioweapons and catastrophic malware, and jailbreak defenses are not 100% effective against the people you worry about, you need someone who can call a tactical halt. You cannot do that with a shot clock set by a Chinese program six or twelve months behind. Any serious solution therefore involves China. Path one is a kumbaya consensus. He does not think it works. International treaties, in his reading, reflect realpolitik, not Star Trek. Nuclear drawdowns happen when everyone still keeps an arsenal that can destroy the planet three times over, so the marginal warhead is pointless. Chemical and biological treaties stuck where they did not beat artillery and gunshots. AI is not like that. The marginal return is huge.

Path two is offensive optionality. You are not going to build a perfect Fort Knox. The question is how you induce consequence on the other side. It is not a pretty theory, and it rhymes with mutually assured destruction, including frames Dan Hendrycks has used. Harris is not calling for a war with China. He is saying that if an adversary can turn off your grid and tens of millions of Americans die of starvation or exposure, you need to be able to say the same thing happens in Beijing — and that you can turn it back on. De-escalation has to be in the design. A tractable AI treaty, he adds, would need inspections of compute stockpiles and precise overwatch of algorithms and eval schemes. He does not see the trust.

Before he will give a p(doom), Harris wants a category that does not depend on timelines: pure optionality. Some choices, if you get them wrong on day one of a frontier cluster, mean that by day 360 the site is compromisable and there is no going back. Gladstone calls those one-way doors. A concrete example he can share: the people you bring in to review site plans and details that would be useful to an adversary. If those people are Chinese nationals, he says, you are done. You are never going to unbake that. The tragedy is that these doors tend to be surprisingly cheap — a fraction of capex and opex for pure optionality — and nobody has systematically listed them or set standards.

Putting offensive options on the table is the same kind of play. You do not have to exercise them. You need them present. Asked for a hit list as if he were replacing David Sacks, he bundles a lot into that first bucket of personnel and insider threat. Zoom out to the grid and ask what redundancies you can add quickly. Look at data-center builders who offer to go suspiciously fast, and at who owns them. He points to a letter from the House select committee on the CCP naming May Day Data Centers as a somewhat suspect entity. If China is a command economy with civil-military fusion, and the CCP wants you to have a rare, backlogged component for a U.S. site, that should tell you something about how much faith to put in the part.

The labs, he thinks, want to do the right thing. Nobody wants to find out the building is compromised. Incentives are aligned; attention is not. Other low-hanging fruit includes using AI to hunt malware and vulnerabilities in old, load-bearing software. Two years ago his own blind spot, he later says, was this blue-collar layer: leasing dynamics, construction delays, procurement schedules. When xAI’s Colossus looked like it would hit a gigawatt sooner than Anthropic’s cluster, the driving variable was not a paper. It was a build. Nation-state physical security is siloed by design, so you cannot even rank teams without an Elo-style bake-off you will never get, and cyber gets overweighted because it couples to AI and feels nerdy. Russians do cyber, he notes, and they will also arson a transformer.

On probabilities, Harris would rather talk about loss of control to superintelligence than generic doom, though multiplied out he is not sure the numbers differ much. His generic answer for years has been that any figure between 10 and 90 percent is reasonable. Below 10%, he thinks you have homework left, because a lot of the crazy-sounding scenarios are already halfway unfolding once you get into the nitty-gritty. Above 90%, living as if you are in that band makes you less effective. He likes the happy-warrior posture: assume you might be wrong, keep agency, do not sit in a deterministic hole. Richard Feynman walking around 1970s New York, sad that the skyscrapers would be wiped out by a U.S.–Soviet nuclear war in a few years, is the cautionary tale. That was a reasonable read of the dynamics. The world adapted.

Uncertainty about uncertainty is what buys him that remaining 10%. “The eleventh chapter of the book will often involve a new character that comes out of nowhere.” He has been wrong at 100% often enough not to push it. Timelines: 2027 does not sound insane, 2030 does not sound insane, 2035 sounds a bit far. When GPT-3 dropped he had two-year timelines because he did not understand infrastructure buildout. He still does not know what the next bottleneck is, and it does not change his to-do list, because there is unused alpha on the table right now. He wants people acting as if 2027 is plausible. It would be a shame, he says, if it happened and we had had analyses pointing there and did nothing.

How he keeps up has flipped since 2021, when you could not use GPT-3 to understand a paper. He now spends about 30 to 40 percent of his time actually reading, and the rest interacting with a model — implications, alternative experiments, “is this a stupid idea?” Reading is SFT; talking to the model is on-policy. It feels like rotating a shape instead of staring at it. The fresh scar: Gemini helped him through gradient flow in residual connections until he switched to Claude and realized he had hallucinated his way into a confident wrong picture. Double-check, especially in a rabbit hole. He does not pretend to have taste in every domain. Frontier-lab papers, and authors he already respects, get a hard look; Twitter supplies the rest. He reads for concepts, not results, because next week’s paper will obsolete this week’s outcome and the furniture in the room stays put.

After Harris leaves, Labenz and Prakash try to metabolize the whole marathon. The disagreement is fractal: Turing Award winners, people inside frontier labs, and AI-for-science specialists still do not see the same phenomenon. Prakash’s update is that things are moving faster because of market pull, not because researchers suddenly understand demand. A 1% improvement that clears a hurdle is a discontinuity, and he thinks software started clearing it in the last month. Jason Calacanis running All-In on OpenClaw machines plus an “Ultron” consolidator was a two-years-from-now picture that arrived now. Labenz’s true north is still being hands-on. His watch item for the year is spending less time at the desk — walking the neighborhood, talking, still moving work forward — and teaching models when to read the document instead of groping with a regex. Consensus on the foundational points never arrived. The homework did not get lighter.