Among 100 Random Doctors, Zak Kohane Would Be Surprised If One Beat GPT-4

Nathan Labenz introduces Isaac Kohane — Zak to colleagues — as chair of the Department of Biomedical Informatics at Harvard Medical School and co-author of *The AI Revolution in Medicine*, with a foreword by OpenAI CEO Sam Altman. Kohane was among a small group given preview access to GPT-4 in fall 2022, before ChatGPT existed in public. Labenz, who jokes that people call him "AI hype boy," calls the book the best practical study of modern AI application he has seen from academia, and quotes Altman: every sphere the technology hits will need this kind of effort.

Kohane grew up in Switzerland, came to the United States for college, and learned computers as a biology undergraduate. Medical school terrified him in a specific way. He had no doctors in the family, and he learned firsthand that medicine was a noble profession but not a great application of science. He bailed for a few years to take a computer-science PhD during the 1980s AI heyday, then watched the field drop into winter after overstated promises. He finished clinical training as a pediatric endocrinologist, became a Harvard professor first in pediatrics, and later created and chaired the biomedical informatics department.

Along the way he has worked on health IT infrastructure with a bias toward getting data flowing for patients rather than third parties. He also launched *NEJM AI*, a *New England Journal of Medicine* spin-off, to ask how society actually gets clinical-grade validation of AI artifacts in care. Labenz's bottom line: as credentialed as you can get at the intersection of AI and medicine, and someone who has already lived through a cycle that did not pan out.

Peter Lee, head of Microsoft Research and a former Carnegie Mellon computer-science chair, called with a condition: Kohane could tell nobody, and Lee had needed top-level company clearance to have the conversation. This was about GPT-4, before ChatGPT. Lee showed him what it could do, then gave him direct early access. That is when Kohane, in his word, became dumbfounded. His first reaction was that his language failed him, which is unusual for him. He was not surprised that this would happen. He could not believe it was happening now. He had expected maybe five years.

He gave the model very difficult cases. One was a child called to the nursery with a hole at the base of the phallus and no palpable testes. He asked GPT-4 what it would do next. It went step after step — imaging, then a hormonal workup — and as he fed back each result it reached a molecular diagnosis of 11-hydroxylase deficiency. If he ran into 100 random doctors, he would be surprised if one of them could do that. He had already seen the model hallucinate and make things up. That it could hold a high-level medical dialogue, on a general language model not particularly tuned for medicine, and then turn around and offer expert commentary on Talmudic questions, was mind-boggling, unsettling, and exciting at once.

Within a few days two consequences were obvious. First, it would transform the back end of healthcare: how money is paid, how bills move, how procedures are permitted — the administrative load clinicians hate. Second, because it would be released to the public, it would change the expertise of patients who have been increasingly bereft of primary care and who already use "Dr. Google" out of desperation, even when the sites are bogus or wrong-headed. As long as there is a human in the loop and they eventually talk to a doctor, he says, it is much better than what exists today. People were already using GPT-3.5 for their own healthcare, in the United States and internationally.

A transformative technology hitting a conservative discipline will rewrite power relationships. Some pushback will be genuine concern about accuracy and bias. Some will be sacred cows: people making a lot of money from a choke hold on how medical information flows. If a model lists five alternatives to the surgery just proposed, and the patient takes that list to another doctor, the first surgeon will not be happy. Kohane says that is one example of millions of such conversations, and that medicine will make the issues personal in a way other verticals will not.

Labenz had his own version of the night. As an OpenAI customer in the GPT-4 preview program, he stayed up until morning trying thing after thing. Once the shock wore off, how did Kohane actually study it, and what should people in other disciplines copy? Get your hands on it, Kohane says, and put it through its paces on real-world examples of what you do. He ran literally hundreds of clinical scenarios — only a few made the book — of doctors engaging the model at different points from initial screening through management to generating billing codes. He asked friends what they had recently asked their own doctors, and ran those questions too.

He did the same for research: wrote down, on paper, the parts of clinical and basic biomedical research, and walked the model through the questions. Some areas were not ready for prime time. Others it did wonderfully. He sees no alternative, for a citizen, a consumer, or someone running a business, to doing that assessment directly. That is also why *NEJM AI* exists. Society still does not know where clinical-grade validation of these tools will live; he at least wanted one journal that would be a source of it. Decision makers who skip firsthand use, across any vertical, are where the danger is.

The line Labenz keeps quoting from the book is that to understand GPT-4 you have to use it and live with it; no amount of reading tells you what a roller coaster feels like. Kohane adds that this is a pre-paradigmatic moment: we do not have good theories for what these systems can do now, let alone later. On the one hand the model does some things much better than you could imagine. On the other it is incredibly limited. You have to learn that firsthand. Peter Lee's phrasing, which Kohane credits, is that it is simultaneously smarter and stupider than any person you have ever met.

Labenz puts two of Kohane's own sentences in tension. How well does the AI perform clinically? Kohane is stunned to say better than many doctors he has observed. For the foreseeable future, GPT-4 cannot be used in medical settings without direct human supervision. The impressiveness is real, and so is the speed: GPT-3 did very poorly on the national medical boards a year earlier; GPT-4 is now top-notch. It is still not a human being. Kohane says he does not mean that in a human-first bigoted sense. The common-sense things people share with one another cannot be assumed to be true of the model. It can engage at expert level on complicated diagnostic and therapeutic problems, and then, not often but with huge mistakes, go off the rails — arguing a point where it is wrong, or treating as unimportant something that matters to you.

There is an old joke: what do you call the person who graduated at the bottom of the medical-school class? Doctor. On average, half of doctors are the bottom half. Kohane is convinced that in those hands GPT-4 can make them much closer to the top, which would be a huge improvement in healthcare even before anyone talks about torchbearer, House-like feats. Just getting medicine practiced the way it should be practiced by better practitioners would be a tremendous advance. But because we cannot assume the model understands a given person's values, skipping the human loop will, he guarantees, create lots of problems. The more you know, the more you get out of GPT: more intelligent questions get more intelligent answers.

The book sketches four ways to plug a system like this into medicine — the trial, the trainee, the partner, the torchbearer — and Kohane thinks the first two, which fit existing regulation, are the ones that fail. Convolutional nets for a focused task, retinopathy on a fundus photo or pneumonia on a chest x-ray, can define a population, a success metric, a follow-up, even an FDA clearance. GPT-4 cannot copy that recipe. A conversation that starts with a cough can slide, because the model contains all of medicine, into any diagnostic or therapeutic neighborhood. A real trial would have to cover all diseases, thousands or hundreds of thousands of patients. Results will not travel: Boston with a lot of health technology is not China with almost no primary care, and in 2020, COVID mortality models a few hundred miles apart performed totally differently because obesity, race mix, and other patient factors differed.

The model itself is not static. The GPT-4 Kohane was using in April 2023 was not the one he had in October 2022. Drugs are static, which is why a trial can stand behind them; a trillion-parameter general solver that also interacts with different hospital cultures is not. You could narrow it to screening, but then purpose-built models might outperform it. Its lift is generality and linguistic supremacy. Labenz cites an old RAND experiment that gave one group free care and watched outcomes for years. Kohane compliments the skepticism and then says you still could not bank the result. Give one health system GPT-4 and another "just good doctors," and any difference is confounded by the systems themselves — for-profit, nonprofit, academic, community, different regions of the United States. Patients and some doctors will use the tool anyway. Double-blinding is close to impossible. Medicine is not cookie-cutter, which is both a source of error and a reason the unit of study is unclear.

The trainee paradigm — pass the same battery we give a doctor, then treat the model as licensed — collides with the same alien interior. Labenz and Kohane both use that word. Benchmarks rest on a pile of untested assumptions about how humans will act. A cancer patient who has a wedding in six months may prefer a better chance of arriving healthy over a better chance of an extra year of life. That trade-off is a conversation. Today the models do not have the common sense or the human grounding to have it. You only internalize how alien the misses feel by hitting them yourself.

The landing place is the partner paradigm, what Kohane also calls symbiotic medicine: a three-party encounter of doctor, patient, and AI. Could not consulting a GPT-4-class system become a violation of the standard of care? He remembers, from training, something that financial pressure has since killed. After clinic, the group sat together, went over every patient, and someone would say, did you think about this? Then you actually called the patient and changed the plan. That extra reflection improved care. The analogy now is an always-on, increasingly multimodal model listening to the visit, reading the notes, and asking whether you thought of that, whether Mrs. Jones already had the test, whether a new drug is twice as effective with lower toxicity.

You cannot compel doctors to keep up with the literature, or even to Google the latest finding for this patient. Genomics is expanding fast enough that patients sometimes know more than their physicians. An agent that is listening and looking at the record is how you give the patient an extra level of scrutiny, remember what you should have remembered, avoid errors, and notice who else on the panel needs more attention. Done well, Kohane says, absolutely no doctor should be without it. They should have a sidekick that is meticulous, completely up to date, ever vigilant, and sometimes wrong — and they should be there to evaluate it. If he were a patient, he would want his doctor to have that. Patients who can reach the same expertise will, he thinks, force the system to raise its game.

The demand side is lonelier than the supply side likes to admit. Labenz was glad to see Replika in the book; its CEO had told him that even an unsophisticated companion meant a lot to isolated people, which now terrifies him a little as the systems accelerate. Kohane goes back further, to Joseph Weizenbaum's ELIZA in the 1970s, a shallow grammatical parser playing a Rogerian therapist. People used it as an actual therapist. Weizenbaum's own secretary would lock her office for a session. Heartwarming and pretty terrifying. Younger listeners, he says, should not be deluded: people with real health problems have very few people they can talk to. A primary-care visit, if you are lucky enough to have one, is ten to fifteen minutes; most questions go unanswered; the next slot can be six months, by which time things are worse. The need is huge. The interaction around healthcare is not optional.

The fourth paradigm is the torchbearer: can the model independently develop testable hypotheses that imply specific therapies with a high chance of being supported by trials? Kohane's book says currently no, and Labenz reached the same conclusion. The barrier, as Kohane sees it, is that the system learns, sometimes superhumanly, about the things humans talk about. Where people talk a lot, it can do some theory formation. Where humans have not talked much, where a new paradigm would need data no one has described, it is at a loss. This generation is linguistically supreme and not scientifically supreme.

At the margins it can still surprise you, because it maximally exploits what we know collectively but not individually. Kohane is principal investigator of the coordinating center for the Undiagnosed Diseases Network, spanning Stanford, Harvard, Baylor, Duke, and others. Ask the right questions, do the right studies, sequence the genome, and in maybe about 34 percent of cases you can figure out the problem and change the patient's life. He ran GPT-4 on solved cases and on unsolved ones. In one unsolved case the team had mutations in five genes; he asked which gene was responsible. The model's top-ranked pick was the gene they had independently validated, including with fruit-fly modeling, against other biologically plausible candidates. That is not quite theory formation. It is working at the very edge of what is known by bringing knowledge to a focus.

Without waiting for a 100x larger model, he is most excited by a few increments. One is hospital-level customization: not fine-tuning GPT, which is still hard, but an accessory set of embeddings — he mentions a Pinecone-style framework — so the general model can draw on the multidimensional co-occurrence of findings in *this* patient population and *this* hospital's practice. That is the direct answer to the problem that Boston is not every other system. Another is speech. Omnipresent, multi-speaker recognition plus the prior record can draft the clinic note, let the doctor edit, and send appropriate versions to the patient, the referring physician, and the insurer. Physicians have been turned into documentation clerks; this is the burnout problem, and he does not have to stretch to say it is next.

Discovery is the stretch. Full de novo theory will stay hard, but constrained questions — which of these drugs looks most effective given X, Y, and Z — may work if the data are right, and it is not clear today's Transformer will be enough. AlphaFold is already a kind of language model whose language is amino-acid sequence, plus a little physics, and it has solved the structure of almost all proteins and begun to inform small-molecule docking. DeepMind is moving toward large-protein interactions. Kohane thinks simulating a cell becomes possible on a five-to-ten-year frame. Before that, in about five years, a company should be able to look at a pile of leads for cancer or Alzheimer's and say which are most likely to carry efficacy and tolerable side effects through phase one, two, and three, by linking structural models to hospital-scale disease-progression data. The science-fiction end state — a cancer patient with this germline and tumor mutation list, asking for the right drug, or for the right drug to develop — he puts at ten to fifteen years, maybe earlier. It is not just scaling.

Labenz closes on a line from the book that surprised him: Kohane believes in obtaining patient data from patients rather than from systems. Who, then, does the AI work for? Hospital, doctor, patient, or some shared brain? Nobody knows how it settles in the long term. In the midterm, Kohane expects AIs aligned to different organizations — an insurance-company AI, a hospital AI, a patient AI — and different markets for each. As a human being he hopes, and thinks there should be the most public support, for alignment with the consumer.

The logic is the same as a second opinion. If he can donate data to a model aligned with him, he is much more comfortable than handing it to an AI that is not. He is much more likely to tell his doctor it is okay to share a record with a colleague than to feed a misaligned institution. Years of work with Apple Health, which he was involved in, now connect some 800 hospitals, growing by hundreds a month, so a person can download medications, labs, diagnoses, procedures, demographics, and soon clinical notes, the way they already repurpose photos from a phone camera. Tens of millions of consumers could contribute. Medical alignment is not the end-of-civilization kind, but healthcare is full of competing alignments. Asking patients for consent, he thinks, will get the biggest buy-in, because patients want their data used for the right reasons. That is an autonomy question. In an era when more of the record has been liberated to the consumer, AI assistants should get it from us rather than from entities that may not be as well aligned.