For under $10, Google's AI independently recovered a lab result nobody had published

Nathan Labenz opens with a claim he repeats often: if people actually understood what AI can already do, many would rewrite their plans. The evidence this week is two Google DeepMind systems, AMIE (Articulate Medical Intelligence Explorer) and Co-Scientist, presented by Vivek Natarajan, on his fourth visit, and Anil Palepu. A year earlier AMIE had already beaten general practitioners on diagnostic accuracy in text chat with standardized patients. New papers, Labenz says, show it also beating primary-care physicians on analysis and treatment recommendations, and surpassing medical fellows in cardiology and oncology while still falling a bit short of attendings.

The part he wants builders to hear first is that none of this required Google-only continued pre-training or heavy custom post-training. The approaches, he argues, could be developed and replicated by API customers using commercially available models, advanced prompting, and thoughtful agent design. After reading the papers he came away thinking it might be time to call it: the AIs might now be clearly smarter than he is. Matching what a single scaffolded model did here would be years of work for him.

Palepu puts the AMIE leap in clinical terms. Earlier medical-LLM work was question answering. Real practice is not a board exam. Doctors gather information themselves, and the first visit often ends with the right test and a direction, not a final diagnosis. The new management-reasoning study asks whether AMIE can carry a patient across visits: work up, tests, treatment, grounded in clinical-practice guidelines and medication labels. Labenz wants that on billboards. Palepu will not quite let him. These are still simulated consultations with patient actors, not real patients, who will not stick to a script. The team is optimistic the results will transfer to some degree. They have not shown it yet.

Labenz asks the obvious product question: why is this paper still text-only when the same group has already published multimodal medical work? Natarajan's answer is methods, not missing parts. Research wants one variable at a time. Images would add confounders. The Med-PaLM and Med-Gemini pieces already exist, and a multimodal AMIE is coming soon. The study as written is still chat, still multi-visit, still a comparison in which the AI doctor outperforms humans on diagnosis and on landing at accepted standard of care, with the simulation caveat attached.

The specialty papers take AMIE into cardiology and oncology. Labenz's headline is that it surpasses fellows and closes in on, but does not yet hit, attending-level performance. Natarajan agrees there is room before the system is as consistent as the most experienced attendings. The finding he cares about is in the cardiology paper: the types of errors AMIE makes and the types general cardiologists make are different, and they look complementary. Head to head, some metrics go one way and some the other. Give the cardiologist AMIE's assessments, and it is a landslide. In almost every aspect the assisted physician is judged superior.

Access is the reason they bother. Natarajan notes that a neurology consult in the United States can wait 12 to 18 months, which he does not consider sustainable for a serious problem. Specialization itself, in his telling, is a human cognitive limit: you cannot cram neurology, cardiology, and internal medicine into one brain, so medicine grew silos with primary care as the door. AI systems do not have that limit. The question hanging over the work is whether AI-powered care still needs those walls, or whether a pocket PCP plus a pocket neurologist is the better picture. Palepu adds a model-generation caveat: the cardiology study ran on Gemini 1.5 Flash, and the stack has since moved through 2.0 and 2.5. Attendings still won on 1.5. The guests are not willing to bet that holds.

Co-Scientist is a multi-agent scaffold that can take human feedback at any step or run fully autonomously. Labenz ranks three tests by difficulty. Drug repurposing is bounded and combinatorial, the kind of grind a careful person might finish. Therapeutic target identification is harder: go into a diseased cell and find a causal node you can drug, on a causal graph he guesses humans understand perhaps 10 percent of. The third is open-ended: how bacteria achieve drug resistance, with only a conserved observation across a few species as a seed. The system succeeded on all three. On resistance, its top hypothesis was the mechanism independent academic collaborators had just found experimentally and had not published.

Natarajan tells the Thanksgiving story. José and Thiago at Imperial College London were not, in his telling, believers. After months of pestering they sent a prompt built from lab results that were nowhere in the literature. The system ran a couple of days. A Google technical fellow forwarded the output. Within ten minutes José emailed that he needed to talk immediately, then asked whether Google was reading his email, then whether ChatGPT had leaked the result. Both no. The AI had produced the same findings his group had hypothesized and confirmed in experiments. The next four candidates also made sense. After the break he put postdocs on them. For Natarajan, a pragmatic scientist with decades in the field reacting that way was the signal they were onto something. The system is, he keeps saying, a relatively simple one, throwing compute at synthesis until original facts show up reliably.

Labenz treats the episode as an answer to Dwarkesh Patel's complaint that models with all of humanity's knowledge still do not make enough cross-domain connections. Human lab and AI data center reached the same unpublished mechanism with no shared pipeline. It was not the first such hint. After Med-PaLM, Stanford's Gary Peltz called to say the models might help millions of rare-disease patients. On a mouse hearing-loss grant, Med-PaLM proposed a biogenic cause Peltz had not considered. CRISPR knock-in experiments reversed the cause; the write-up is still under review. Later Med-Gemini work on human variants of unknown significance looked promising on retrospective data. The ugly part: in crude single-shot use, one useful hypothesis rode along with thousands of pieces of utter garbage. Peltz had the eye to discard nonsense and the patience to read all of it. A less experienced scientist would have quit after five. Co-Scientist's job, over nearly 18 months, was to stop sampling thousands of times for one hit.

Labenz asks whether post-training that suppresses hallucinations also suppresses the crazy associations that become big ideas. Palepu's intuition is that hallucination does foster creativity: hypothesis generation has to deviate from the script a bit. Natarajan used to think they were two sides of the same coin. Older models were actually more useful for this task *without* post-training, because they produced crazier ideas. Structure has made the process more reliable. They do not know what wild original thoughts they are now sacrificing. The system's answer is diversity: plug in more than one model, generate a mixed pool, and let the tournament rerank so high-quality hypotheses bubble up. Labenz's synthesis is that a rich literature wants a disciplined reasoner and a thin literature wants a whimsical hallucinator, and a mixed pool can hold both.

The high-level design, Labenz suggests, is interviewing experts on how they think, turning each step into a prompted sub-agent, handing them literature search and tools such as AlphaFold, then turning the hyperparameters up: more rounds, more generations, more evaluation. Natarajan says the narrative has changed since his earlier visits for Med-PaLM and Med-Gemini. Those systems specialized a general model with medical fine-tuning. AMIE and Co-Scientist drop that step. Some of the old specialty data has been upstreamed into Gemini, but the better setup, in his view, is specialized prompts chained as agents. Palepu agrees post-training is not gone, but in these papers it has taken a back seat to inference-time system design. Nothing is hyper-optimized. They wanted a functional prototype fast.

On AMIE, Palepu's team leaned on auto-evals and vibe checks. The management-reasoning agent jumped when they stopped generating one plan and started drafting four concurrent plans, then refining them together. In a self-consistency style, the model combined the good parts and left the bad. Internal auto-eval made the lift obvious. They picked four; they did not sweep two versus eight. Co-Scientist was closer to an 18-month project. Instruction following got good enough that a precise prompt on a general model replaced fine-tuning. They decomposed tasks, evaled, patched weak spots with specialized agents, then ran a simplification pass because a newer model could do with less scaffolding. Labenz has lived that loop on smaller projects: a new model is a license to delete.

Natarajan puts long context next to raw model progress as a reason the 18-month grind worked. Co-Scientist has no explicit memory. Gemini can take up to two million tokens, so the system generates ideas, reviews, and debates, then dumps the walls of text back into the next round's context and lets the model use the feedback implicitly. Without reliable million-token reasoning they would have had to engineer RAG and maybe train it end to end. He thinks long context is underappreciated because it does not produce Studio Ghibli-style viral moments, even though it makes these applications simple. Palepu's management agent is the same bet: stuff a bundle of clinical guidelines into context rather than praying retrieval picked the right one. At 256K they catch something relevant either way.

They did not blindly max the window. Palepu's teammate Valentine set up auto-evals. The original plan was to go all the way to a million tokens of guidelines. Performance looked better when they dropped to 256K, and both beat zero-shot or a single retrieved guideline. On Co-Scientist the long-context story is less scientific: they rely on redundancy. Many ideas, a tournament, a pile of feedback, then generate again, hoping at least one sample catches the key elements. They did not build evals that measure long-context command. They also refused cozy benchmarks. The philosophy was to sprint to a domain expert, show the idea, and try to get it validated as a real discovery. Hill-climbing every component would have added a year.

Labenz remembers GPT-4 maxing out around three to six rounds of self-critique and then getting worse. These systems run for days. Natarajan's explanation is that the loop is not closed. Every round can pull new information from web search, specialized tools, and AlphaFold, which raises entropy and blocks mode collapse. Strip the tools and you fall back on the quality of published literature and the complexity of the problem. Some unknowns no amount of test-time compute will invent if the information is not reachable. Some problems are trivial on the first try. The sweet spot in between is, he suspects, a good chunk of the problems humanity actually cares about. Palepu adds that humans can inject ideas too, another entropy source, and that pairwise comparison of every hypothesis combination is a very different variance structure from polishing one idea forever.

Labenz guesses the microbe project burned 10 billion tokens. Palepu says it should fit in current context limits, likely less than 10 million inference tokens from question to answer, a little more if you count every tournament call. Back-of-envelope at then-current GCP prices: most queries, tournament included, a few dollars, less than $10. Prices should keep falling. This is, he says, probably the dumbest and most inefficient version of the system. Intelligence per token and dollars spent should rise as they clean it up. Labenz asks about specialist models as a source of entropy that is not already in the papers. Natarajan agrees they have mostly scratched peer-reviewed text. Negative results are dark matter, unpublished because of incentives. Supplementary files hide giant experimental datasets. Pair Co-Scientist with a data-science agent and those tables become feedback. The ARC Institute's virtual cell atlas, gene perturbations for 300 million cells, would take human teams years. Agents generating hypotheses, analyzing, and coming back is the experiment he wants.

Labenz's high-concept question is whether specialist scientific models will follow speech, vision, and language into one architecture, so a future reasoner thinks in protein space instead of making a lossy AlphaFold API call, the way DALL·E used to sit behind a language bottleneck. Palepu is not sure deeper integration is easy in every specialist domain. Natarajan locates the real gap in incentives, not architecture or compute. Frontier training data is still mostly unspecialized public internet. When they have added medical images or genomic information, the new-modality benchmarks go up and general language and vision benchmarks go down. Labs do not want to sacrifice LM-harness scores for a capability that is a small slice of users. He thinks benchmark scores matter less than practical utility, but without a clear utility metric the trade is hard to sell to decision-makers.

Labenz hears the old vision-versus-language split: separate teams, then late fusion, then mixed pretraining once people believed it would work. Natarajan says their cheaper experiments were continued pre-training and SFT, not even late fusion, and they still saw original-benchmark regressions. The honest next step is putting the esoteric data into pretraining from the start, which itself needs a proof of benefit on benchmarks everyone else already cares about. So they sit in a local optimum: train their own agents, wire them to specialist tools, and ship the pudding. Architecture is not the unknown. Incentives are.

For people building their own systems, Labenz wants two transferable pieces: tournament-style pairwise ranking, and structured reasoning instead of free-form chain of thought. Natarajan traces the tournament to AlphaStar. He does not call it an industry standard, because it is inefficient, an n-squared hint that models still cannot independently score and verify ideas. They cluster hypotheses to avoid a thousand-by-thousand comparison. He expects the ranking idea to survive by sinking into latent space rather than remaining an explicit World Cup. Palepu's case for structured reasoning is practical and cognitive. Structured outputs make agent-to-agent plumbing easier. More important, they force a path: long analysis, then high-level management goals, then a plan. Left free-form, the model starts writing the plan before the higher-level steps are done.

Labenz analogizes it to an airline checklist versus a random walk around the plane that ends with "looks good to me." Palepu warns not to over-slice. When they got too granular, first summarize the chief complaint, then summarize the history, quality dropped. Higher-level slots, with unbounded lists of analysis items and goals, produced better plans on their auto-evals and vibe checks. Dynamic length, fixed skeleton. Labenz's takeaway for listeners: use structured outputs, do not overdo them. On the ranking future, Natarajan hopes head-to-head gives way to "here are ten things, pick the best," with the tree search happening in latent reasoning so you spend fewer tokens. Explicit write-out is interpretable. It is also wasteful.

Labenz's last design question is whether they could prepend a question-finding agent that wanders the internet, poses its own problem, and then solves it. Natarajan calls generating novel solutions a root-node problem at Google DeepMind, and asking the right question an even more upstream one. The day AI systems reach that is the day he would say there are geniuses in data centers. He already thinks current models have decent ability to surf and notice good questions; they will try it. All of the published work ran on Gemini 2.0. 2.5 results should follow soon, a nearly free step-change. For the next couple of years he sees mostly an engineering challenge, not fundamental unknowns, plus the social problem of getting scientists to pay attention. Co-Scientist is in a trusted-tester program with close to 100 world-leading scientists, rolling out in waves. If he is optimistic, millions of scientists have access by year-end. AMIE is harder. A clinical study at Beth Israel Deaconess Medical Center, a Harvard Medical School teaching hospital, will put an LLM system in front of real patients with enough on-site clinicians to take over. If little goes wrong, they dial down oversight, add centers, and add patient-facing capabilities. The research, he says, has gone far enough. It is time to take it for a drive.