Two and a Half Years From 33.3% to 86.5%: Google Sent Foundation Models to Medical School

Nathan Labenz opens the last episode in Cognitive Revolution’s medicine series with a timeline that still sounds implausible. Two and a half years earlier, language models were just better than chance on medical licensing-exam questions. By May 2023, Google was beginning to commercialize systems it described as expert-level. The paper graph he returns to later makes the climb concrete: GPT-Neo at 33.3% in December 2020, Med-PaLM 2 at 86.5% on the same kind of USMLE-style items, a score in the top quartile of human test-takers.

The more arresting number is not the exam. Google now says Med-PaLM 2 is preferred to human clinicians on eight of nine evaluation dimensions, with a tie on the ninth. Labenz will later put that gap next to his usual reference class: GPT-4 is preferred to GPT-3.5 only about 70–30 in OpenAI’s technical report, and GPT-4 versus Claude v1.3 is closer to 60–40. Preference ratios this fat are not what the field usually produces.

The guest is Vivek Natarajan, a research scientist at Google and a lead author on the Med-PaLM and Med-PaLM 2 papers. Labenz wants the two-year history, the data-efficient techniques, and especially the validation work, which he treats as a lesson for anyone building AI in a high-stakes setting. For Natarajan, who grew up in rural India, the point is not only to build the systems but to deploy them so that medical expertise is less of a geographic lottery.

Natarajan traces the present to 2017 and the Transformer: a general-purpose architecture optimized for GPUs, able to guzzle data at scale. In the six years since, he says, the architecture itself has barely moved. The work has been data, scale, compute, and applications. That stasis, in a field that usually turns over every season, is his tribute to the original paper and to authors such as Ashish Vaswani.

The second break was OpenAI’s GPT-3: decoder-only Transformers trained on internet-scale text with next-token prediction, and few-shot learning that actually worked. PaLM, at 540 billion parameters, followed a similar recipe with different details. The third was alignment and RLHF. Natarajan reminds Labenz of Microsoft’s Tay, the Twitter chatbot that went off the rails in a day or two. Today’s GPT-4-class models mostly produce what he calls delight, because human feedback can steer what comes out.

Labenz has always wanted to ask about the 540 billion. From the outside it looked as if Google saw GPT-3’s 175 billion and decided to go bigger, while DeepMind’s Chinchilla work was arguing for fewer parameters and more data, both landing around the same time. Natarajan was on neither study; his personal read is that both were useful data points now shaping the next generation. Labenz also flags a buried line from the PaLM paper, around page 44: PaLM outperforms the average human on BIG-bench. At the time, almost nobody noticed.

Inside Google, a finished base model is not a tidy API. Software and hardware keep moving; three months later there is often a better substrate, and teams have to decide whether to migrate work they have already done. Natarajan says he is sometimes surprised the infrastructure works at all. Google has legendary engineers, but it can feel taped together, as if the system could crash at any time. The internal advantage is seeing the raw model rather than an API, which matters in medicine, where you need to specialize.

Flan-PaLM is instruction-tuned PaLM, example-based, no RLHF, and it set a then-state-of-the-art on USMLE-style questions. The exam is what you take if you want to practice as a doctor in the United States: vignettes with symptoms and metadata, then knowledge retrieval, reasoning, and often process-of-elimination across multiple choice. Passing is typically around 60%; expert test-takers sit near 85%. Natarajan now thinks the exam may already be too easy a measure for today’s models, and that the field needs new yardsticks.

The team has been careful never to say the model “passed the USMLE.” They evaluate on a representative set of USMLE-style questions, not an exam taken from the board. The interest is not a public-relations badge. It is usefulness in real clinical settings. Natarajan’s personal view is that the exam was useful for showing systems could clear a bar, and useful for press, but it is not a good indication of real-world utility in medicine.

That gap showed up as soon as they moved off the benchmark. Flan-PaLM looked great on USMLE-style items; on consumer medical questions, scored by physicians and later by non-experts, it sagged on factuality, bias, reasoning, knowledge recall, and helpfulness. Natarajan’s lesson, which Labenz wants on a poster, is that narrow systems with a clear use case are easy to evaluate, and general-purpose systems are extremely hard. You have to go deep into the actual use. Med-PaLM 2 later showed the same pattern in reverse: a general PaLM 2 is strong, a medically specialized model is much stronger, including nearly a 9x lift in evidence of good versus incorrect medical reasoning.

Labenz singles out a figure from the first Med-PaLM paper: clinicians, Flan-PaLM, and Med-PaLM scored side by side on ways of being right, such as recall and understanding the question, and ways of being wrong, such as bad recall or omitted information. The bars add up to more than 100%, because a single answer can be both. That is true of doctors too. He credits Shekoofeh Azizi for the illustrations and Alan Karthikesalingam for the evaluation rubric. In a high-stakes domain, drawing the failure modes this finely is a stance, not a decoration.

The working assumption for Med-PaLM was that Flan-PaLM, trained on internet-scale data, already had the medical knowledge in its weights. The job was to elicit the right knowledge, teach better deduction, and, separately, teach the way clinicians talk. Doctors rarely say you 100% have a disease or 100% do not. They carry uncertainty and ask for follow-up tests. A student can memorize every textbook and still be useless if they cannot apply it. Knowledge and manner are different problems.

The method was instruction prompt tuning, “soft prompts.” The team froze Flan-PaLM and trained a prefix of roughly a million parameters, not corresponding to real words, that conditions and anchors every token that follows. Natarajan’s image: the model’s knowledge is a huge library; the soft prompt hangs a spotlight over the medicine stacks and teaches tone. The data were shockingly small: low hundreds of expert-written demonstrations, trained in a few hours, which made iteration fast. That data-efficiency, he adds, is itself an emergent property of scale. Small models do not store enough knowledge for prompt tuning to help. Large ones only need guidance.

Labenz wants the linear algebra in plain language. The big model stays frozen. What you learn is a prefix in the same embedding space as language, sitting in a region you cannot type. Soft tokens are abstract conditioning, a general idea of how to answer; the hard instruction on the specific question is a constraint. Together they produce the reply. Natarajan would like to know whether the vectors cluster in human-interpretable ways; they have not had time to look. A curriculum of multiple prefixes, bedside manner versus clinical detail, is something they are exploring in a different form on Med-PaLM 2, as knobs for factuality, safety, and style, because different users will want different outputs.

After expert raters, the team circled back to laypeople. Language models are platform technologies with a wide spectrum of users: physicians, administrators, life-science researchers, and non-experts who already search the web for medical information. Natarajan’s personal stake is sharper. He grew up in parts of India where seeing a doctor meant walking 30 miles in extreme heat, giving up a day’s wages, or going without food. For many people it was not an option. They went a lifetime without a doctor, and small problems compounded into lower life expectancy.

His 2013 undergraduate thesis was an app called Ask the Doctor, anytime, anywhere, built with pre-deep-learning methods that did not work. Now, he says, it is realistic to think about putting a world-class general practitioner into the pockets of millions of people. That is the subtext of the layperson evaluation. The team has also seen that the more you know, the more you get out of these systems, and the less you know, the more they can harm you. Expert scores on factuality and reasoning are not enough. You have to ask non-experts whether the interaction addressed their intent.

The first evaluation was deliberately simple: single-turn, English question answering. PaLM was not optimized for multilingual use, and multi-turn dialogue was more a LaMDA strength. They wanted something scalable, without a feature creep that would sink the paper. As the models open up to academic and medical-research collaborators, those studies can follow. Labenz, who has plenty of worries about the general AI future, keeps coming back to this one: the people who do not have access to doctors will want a word with anyone who says we should just forget about the technology.

The timeline is compressed even by 2023 standards: PaLM paper in April 2022, Flan-PaLM around August or September, Med-PaLM in December, PaLM 2 and Med-PaLM 2 by May 2023. A little over a year. PaLM 2 was more compute-optimal, which made full-weight fine-tuning practical. They still did not use much data: the same datasets plus on the order of 200 expert demonstrations, about a million tokens. Updating all the weights, rather than a soft prefix, was the key change. Accuracy on the USMLE-style set went to about 85%, in the top quartile of doctors who take those tests.

Memorization was the obvious objection. The team measured overlap instead of hand-waving: for most datasets, less than 10%, and the score gap between overlapped and clean items was not statistically notable. This looks like emergence, not cramming. Labenz asks how much you gain from prompt engineering versus soft prompts versus full fine-tuning. Natarajan’s stance, including in debates with people around OpenAI and Microsoft, is that any gradient update is specialization, and those methods sit in the same category. Simple prompting looks great on benchmarks. On multi-axis real-world evals the gaps appear. A general model may hold the world’s knowledge and still not know how to act in a real setting. That is why they fine-tune. The project is a Brain moonshot, bottom-up across Healthcare, Brain, DeepMind, and other Alphabet labs, under the tagline of sending foundation models to medical school.

The new human-eval chart is the one Labenz cannot stop looking at. On consensus, reading comprehension, knowledge recall, and reasoning, Med-PaLM 2 is preferred about 70% of the time, tied 10 to 20%, physicians preferred about 10%. On omitted information, possible harm, and likelihood of harm, the model wins cleanly. Demographic bias is a tie. The one axis physicians still win is inaccurate or irrelevant information, which Med-PaLM 2 is more likely to include. Labenz’s eye test is that seven of nine are not close, a much fatter ratio than GPT-4 versus GPT-3.5. The first paper’s line that Med-PaLM “remains inferior to clinicians” no longer sits easily on this chart, at least in this question-answering setup.

Natarajan will not let the chart become a deployment license. These models should not go out autonomously; experts still need to be in the loop. Physicians are biased toward succinct answers because they are short on time, which may have hurt them in the comparison. Follow-up studies will ask doctors for their best work and tell them how the answers will be used. What the results do show is augmentation value right now: in e-consults, a doctor writes the gist and the model completes it in language a patient can follow; documentation that took ten minutes can take two. Anywhere an expert can verify quickly, the model can shine. His picture for the next few years is a co-doctor that listens to the visit, reads the record more holistically, interprets genomics most doctors do not really understand, and surfaces information while the physician decides. In regions where the standard of care is nothing, the upside is enormous, and still not a reason to ship without validation.

Medicine is inherently multimodal: lab records, charts, scans, genomics. Natarajan’s larger claim is that biology and medicine are the largest data-generating flywheel on Earth, bigger than almost any other domain, and no human team can digest it. AI is the only way to make sense of it. He expects two consequences: faster fundamental discovery on disease mechanisms and biomarkers, and precision medicine scaled to billions of people. The horizon he gives is 15 to 20 years. Labenz, looking at the 2020-to-2023 exam graph, wonders if that is conservative.

Are the conceptual pieces already on the table? Natarajan splits the question. If you want human-style intelligence, we do not understand human intelligence well enough. If you want a different, complementary intelligence that handles information at scale, most building blocks exist. Language models are already very strong; vision models, despite Segment Anything and the rest, are not there yet, and medical scans make that obvious. Interpretation is finding a needle in a haystack, so representations have to be exact. EHR and genomics are different beasts again, and encoding choices combinatorially explode. There is also an unsettled architecture bet: one giant model versus compressed specialist models you route between, in the spirit of HuggingGPT. Nobody has run a serious comparison.

Google I/O brought a Med-PaLM API, for now a limited trusted-tester and academic program, still optimized for the question-answering settings they built for. Documentation and co-pilot workflows with low risk will move fast. Expert-in-the-loop use needs more validation but has no shortage of interest. A consumer product is the least clear path. People are already using GPT-4 for medical information. “The genie is kind of already out of the bottle,” Natarajan says. He does not know who builds the consumer product, where, or how, or whether regulation clamps down.

Labenz raises a live debate: startups training only on licensed “trusted” data, implying internet pretraining teaches the model a lot of junk. Natarajan stays with internet scale. Next-token prediction at this scale is a complex multi-task objective; syntax, semantics, medicine, physics, and common sense come out of it, including the kind of context a previous guest had flagged, that cereal usually means milk and Burning Man usually means dust. Rare diseases and odd symptoms have usually been posted on some forum. Four years of his own work across imaging, records, and now LLMs keep showing that more diverse data improves reliability, calibration, and out-of-distribution performance. Small datasets are biased by definition. Alignment is the other leg; you need both. Asked to zoom out, he comes back to the line that opened the show: this feels like the opportunity of a lifetime, to use AI safely to improve the health of billions of people and help them reach their potential.