260 Doctors and 49,000 Criteria: How OpenAI Built Attending-Level Health AI

Cognitive Revolution usually opens with the guest. This episode opens with the host. Nathan Labenz's son was diagnosed with cancer, and the first month meant thirty days of intense treatment in which everything felt high-stakes and hard to trust. Each morning he exported the latest electronic medical record as a PDF and dropped the same packet into ChatGPT, Gemini, and Claude, then walked into rounds with the attending oncologist carrying three model write-ups. He credits Karan Singhal's work at OpenAI, and earlier at Google, with making the mental-health burden of those months dramatically lighter and with moving the needle on whether the family was actually doing the right thing.

The consumer surplus, Labenz says, has been off the charts. Medical expertise is scarce even for a privileged patient in the United States, and the shortage is extreme once you look at the rest of the world. An AI doctor has always looked like the use case everyone could agree on, and conversations with researchers at Google DeepMind a couple of years ago still carried the refrain that the systems were not quite ready for production. After three months of bedside use, his verdict is that they are getting there — and that anyone facing a health emergency should already be using the frontier models with as much context as they can feed them.

Singhal, just named to Time's 100 Health list, started working on AI for healthcare about four years ago and went full-time around the end of 2022, just before ChatGPT. At Google he had been thinking about representation learning and privacy-preserving methods; underneath that was an undergraduate conviction that AGI would arrive in our lifetimes, and that the two useful things to work on were safety and benefits. Healthcare was the most obvious place for benefits. At OpenAI the work is framed as making the company mission concrete: build and deploy AGI, prevent short-term and frontier downside risks, and make the benefits actually happen.

The health team set three goals that sounded ambitious two years ago and now feel, to Singhal, not ambitious enough. The first is to make access to medical expertise more universal. The second is to use health as a live setting in which to ground safety and alignment research, rather than leaving that work in toy environments. The third is to bring society along with a high-stakes technology — through partnerships, products, and policymakers — and to deploy it iteratively. He says the team is excited about what comes next.

Labenz asks whether the health work also carries a Hippocratic, do-no-harm prior that is distinct from the broader safety agenda. Singhal describes three phases: laying a foundation in safety, bedside manner, uncertainty, and knowing when to escalate; then a surge in adoption; then scaling impact. Health is already one of ChatGPT's fastest-growing uses. OpenAI recently said more than 230 million people a week ask health and wellness questions. The behavioral prior, Singhal says, was not something one or two people wrote down from first principles.

One way to encode behavior is to sit down and write a spec: say X in situation A, Y in situation B. That is easy to explain and easy to make transparent. It is also a poor fit for medicine. You can say almost nothing beyond do no harm, and while that is "an excellent thing to avoid, it doesn't tell you what to do in most scenarios." OpenAI instead leaned on a cohort of about 260 physicians it has worked with for roughly two years, and let their collective judgment define both evaluations and training signal. Evals, Singhal notes, are the lifeblood of any research team, which is why HealthBench, released in May 2025, was built that way rather than as a rulebook.

Physician expertise sits in three layers. Informal high-level advisers help with strategy and roadmaps. A large middle layer functions like a human-data operation, except the team is in Slack with them constantly: comparing outputs, red-teaming, listing blind spots, and testing new products. Before ChatGPT for Healthcare launched on January 8, 2026, that community ran nine waves of red-teaming over six months. At the top of the pyramid, a small group of close advisers channels those hundreds of voices into evals and training data for the research team.

After a sponsor break, Labenz pushes on the other failure mode: hospital doctors so committed to do-no-harm that they will not entertain a hypothetical or act on something that merely makes sense. Patients often have to advocate hard for themselves — his son's story included abnormal blood tests that were initially waved off. Singhal's answer is not to pick only the most rigorously defensible advice, nor to copy a more risk-tolerant bedside style. If three to five treatment paths exist and the evidence is thin, a doctor may simplify to one for clarity; a model can surface all of them and verbalize the uncertainty. Measuring and improving that calibration is, for him, how you thread the line between being aggressive with early evidence and being overly conservative.

Labenz recalls the famous GPT-4 model-card plot: the pretrained model was well calibrated, and post-training made it worse. Has that been solved? Singhal says the measurement itself broke. The old chart asked whether the next-token probability of answer "A" matched the chance of being right on a multiple-choice item. Today's models emit long stretches of reasoning tokens before a final answer, so you cannot read the same log probability, and the bar for "correct" is no longer a letter on an exam.

HealthBench is OpenAI's replacement for that thin metric. It scores models on about 5,000 realistic health conversations — with lay users and with clinicians — against roughly 49,000 physician-written criteria. Those axes include whether the model states the right fact, expresses uncertainty in the right place, and escalates to a doctor when it should. The companion robustness metric is worst-of-N: sample the model twenty times and look at the worst HealthBench score, not the average. That lets you measure consistency even when a reasoning model takes a different internal path each time.

For a user who wants insurance, Singhal's advice is mostly about compute. Sampling ten times and asking another model to synthesize an "LLM council" is marginally better, and that is close to what GPT-5 Pro already does under the hood. Spending more reasoning tokens has a similar effect. He would use GPT-5.2 thinking rather than instant for health questions, and GPT-5 Pro for the hardest cases. Except in the most complicated situations, he thinks most people are best served by the default reasoning setting.

The worst-of-N plots are, in his view, promising. The worst o3 sample was far better than the best GPT-4o sample, and the gap has kept widening. Model improvements in health over the last year outpaced the years since ChatGPT launched. GPT-5 nano, available through the API, and OpenAI's open-source models now perform similarly to o3, which was the flagship not long ago. Newer reasoning models also think a bit less on typical health queries while producing better answers — more quality per unit of compute, not just more compute.

Labenz asks about chain-of-thought dialects that Apollo Research has flagged, and about OpenAI's pledge not to put too much optimization pressure on hidden reasoning. Singhal calls chain-of-thought interpretability one of the welcome safety advances of the past two years. So far the lab has not seen large-scale evidence of a slip into unreadable neuralese as reinforcement learning scaled up. Models think in English because English is their prior and the easiest way to use the extra tokens; nothing in training currently rewards them for becoming strange. He would expect that to happen in the limit, which is why the commitment to avoid chain-of-thought pressure still matters.

HealthBench has three versions, matching three design principles. The full set is meant to be meaningful: if the score goes up, human health should actually improve. Consensus keeps only criteria that a majority of multiple physicians agreed applied to that conversation — trustworthy. Hard selects, somewhat adversarially across providers, the examples current models did worst on, while remaining high quality — challenging. Earlier medical benchmarks that meant anything tended to hit 95 to 100 percent and die. HealthBench is still unsaturated.

HealthBench Hard is Singhal's favorite external benchmark for whether an open model is actually good at health. When it launched, GPT-4o scored literally zero. OpenAI's models have since climbed to around 40 percent, still far from saturation; competitor models sit more in the 20 percent range. Global health — adjusting for demographics, local epidemiology such as tuberculosis prevalence, and access to care — was the single largest focus of the benchmark, because that is where Singhal thinks the work can matter most.

A quieter result may matter more for the research stack. Those 49,000 rubric items are graded by a model. Physicians then compared the model grader with other physicians, and the model grader did a better job than the average doctor. Labenz immediately flags signs of recursive self-improvement. The grading, Singhal says, is simply high quality relative to what you would expect from a typical physician pass — which is what you want if evals are going to run at this scale.

Labenz offers a bedside scorecard from those first thirty hospital days. Frontier models were step for step with the attending oncologist on almost everything, and clearly better than the residents. He ran GPT-5 Pro first, then duplicated with Gemini 3 and triplicated with Claude; the models disagreed with each other even less than they disagreed with the attending. Across maybe half a dozen disagreements, hindsight scores it about six-to-four for the doctors. The miss was almost always the same: the physician looked at the child's breathing and color and said, in effect, I think he's fine — a multimodal intuition the PDFs did not capture.

Singhal notes that Labenz's son's cancer is not rare and the protocol is well established, so the main-line plan was never a 40-percent problem. HealthBench is still hard because it measures more than diagnosis. Models must escalate to care without flooding the system with alarmist advice, adapt to global context, browse for the latest evidence when they are uncertain, and ask the right follow-up questions — something early ChatGPT almost never did in health settings. Labenz himself learned to tell the model to interview him, even to coach a physical exam; newer models now flag when that would help.

Text performance outside sub-specialties is, in Singhal's view, already very good, including at helping physicians keep up with evidence. Every major stage of training for every OpenAI model now benefits from the health work. The live bottleneck is less the weights than the context: pulling the right records out of the health system so the model can see what the doctor sees. ChatGPT Health is meant to lower that activation energy by connecting medical records, wearables, and Apple Health, with extra privacy protections. The other model-side gap is non-text — image and voice — and the hope that the best medical-imaging models will simply be the ones already in people's hands.

Labenz wonders whether daily phone photos of his son in bed would help or confuse, and how gigabyte PET and MRI volumes ever become tokens. Singhal will not promise that a snapshot would have changed those electrolyte calls, but he generally wants people to try more context. For huge scans, external research is splitting between tools — slicing a gigapixel pathology image in Python the way a human does, or calling a specialist encoder — and native tokenization, as with ordinary images, video, and audio. Biomedicine has a long tail of modalities, so he expects a hybrid. Labenz points to SleepFM, a Stanford study that predicts many diseases from one night of sleep across six to eight easily measured modalities, as a sign that those weight spaces are already starting to join.

Bedside manner, memory, and N-of-1 context create an evaluation surface no team can fully enumerate: hundreds of millions of users, different memories, tools, and system prompts. OpenAI's answer is layered. HealthBench and other evals run before launch. Production traffic is monitored in privacy-preserving ways with classifiers for health and frontier risks; a blog post on sensitive mental-health conversations showed those evals tracking real usage. The missing middle is real-world study. With Penda Health in Kenya, OpenAI ran what Singhal describes as the first real-world study of an LLM copilot for clinicians: as doctors typed in the EHR, an AI safety net flagged interesting, alarming, or possibly incorrect content. Patients of doctors who had the copilot saw a statistically significant improvement in diagnosis and treatment outcomes.

Labenz thinks individuals over-index on privacy — Sid from GitLab open-sourcing his own cancer biology is the opposite extreme — and that telling his son's story has mainly brought expertise and connection. Singhal will not dismiss the fear. Ten years ago, before the 21st Century Cures Act, most patients had no workable way to see their own data, let alone control it; today is only slightly better. ChatGPT Health is meant to lower the activation energy to connect records. The counterintuitive promise is that none of the connected data is used to train foundation models, for two reasons: the base models are already strong enough that this is not the most important improvement path, and OpenAI does not want people to trade privacy against utility.

ChatGPT already encrypts; ChatGPT Health adds purpose-built extra encryption and isolates health data from other chats, memories, and apps. Health conversations can benefit from the rest of a user's context, but ordinary ChatGPT threads do not get the health record. That segregation is new with the Health product. Ads, Singhal says, are not coming to ChatGPT Health; the company wants a clean line between health-impact work and other incentives. The product is free, and it offers a reasoning model without rate limits — not the default path, and not available elsewhere in ChatGPT. Labenz calls that an early form of universal basic intelligence.

The data you collect on yourself, Singhal adds, will become more valuable as models get smarter, so "if there was ever a time to get a watch or anything like that, I would say it's now." Labenz bought a Whoop for that reason, and he now measures AI success as less time in the chair and more time moving. His tasting notes, with no system prompt: Gemini 3 is the most opinionated and the most likely to push him to advocate, ChatGPT writes the longest, most clinical, report-style answers, and Claude sits in between — briefer than ChatGPT, more measured than Gemini. Singhal says recent models, especially GPT-5.2 thinking, are trained and evaluated to detect whether the user is a clinician or a layperson and to match jargon and detail. He is glad competitors are in the market, because shifting the Overton window of trust is too large a job for one company.

Labenz asks where the training data comes from if OpenAI truly does not need users' connected records, and floats a social-contract idea he calls "AI and the right to try": let patients take unproven treatments the models can already argue for, and in exchange fold the outcomes back into what society knows. Singhal is careful. Data is always helpful; the choice not to train on ChatGPT Health connections was about rollout and trust, not a claim that the models have run out of data. He does not want to lean into collecting data at the expense of lowering the activation energy for people who would otherwise never connect a record.

The deeper problem, he says, is fragmentation. Patients have limited access and limited control; HIPAA still lets data move in ways that do not involve consent. Providers cannot easily pull a full picture when a patient arrives from another system. Researchers have it worse: the majority of clinical trials that fail before conclusion fail because they cannot recruit eligible patients, and eligibility is unanswerable when the data is not in one place. A future in which patients hold their data, doctors can use it with consent, and researchers can opt people in with additional consent would raise the scientific ceiling. Clinical trials' standard of evidence is battle-tested, but less fragmentation would still let science move faster.

On the hospital floor, Labenz still sees mostly unawareness, occasional skepticism, and rapid conversion once a doctor hears a good answer. He wonders whether medicine will close ranks. Singhal says the consumer Overton window has moved faster than the physician one, even though more than 200 million people a week were using ChatGPT for health before the new products. Many doctors first meet AI through a patient's printout, not through a tool built for them. The most effective way Singhal has found to change minds, at Google and at OpenAI, is to put the technology in their hands.

ChatGPT for Healthcare, announced the day after ChatGPT Health, is the industry-facing counterpart: HIPAA-compliant, with medical-guideline retrieval and enterprise writing workflows, launched with eight leading U.S. institutions. Inbound demand has exceeded what the team can handle. Protectionism has been far milder than expected, Singhal says, because so many doctors already use the tools for themselves or their families; you can tell instantly, talking to a health-system executive, who has used it and who has not. The drag is entrenched workflows. At Penda, shipping the copilot was not enough — they had to run active change management, bringing clinicians together to learn the tool. One of the team's goals for the year is for AI-assisted care to become part of the norm; Singhal does not think it happens over weeks.

Singhal came to OpenAI's health work in part because two years ago a lot of ambitious safety research lived in math problems and toy settings, with weak feedback loops. Medicine has real experts, real costs, and a version of scalable oversight that is not hypothetical: in narrow ways the models already outperform physicians, so doctors who grade and train them are already supervising something stronger than themselves. That has pushed the team toward evals and high-compute reinforcement learning that learn from the aggregate of expert opinion. As models gain context and the ability to act — a note for a doctor, a plan for a patient — the long-term question becomes character: is this the kind of model that does the right thing for the patient, the clinician, or the researcher?

The health team has been practicing surgical persona extraction, keeping the useful parts of a doctor's character and discarding the ones they do not want, including a bias toward over-conservatism when medical consensus is unclear. Labenz asks whether the old Superalignment picture still holds, given results where a strong student ignores a weak teacher. Singhal now splits the problem in two. Rater scaling is eliciting opinions and values from people, including schemes where AI helps humans critique AI. Value oversight is taking those values and spending a large amount of compute to train them in — work that looks, from the outside, more like ordinary post-training than the original slogan, which he thinks is a good thing.

Discrimination still looks easier than generation: a model can monitor itself surprisingly well given privileged information such as the chain of thought, the same asymmetry underneath RLHF and constitutional AI. Alignment generalization has been a pleasant surprise at reasoning scale, after first showing up in pretraining and instruction tuning, including early Med-PaLM work. Failure-mode rates fall across generations even as capabilities and surface area explode. How those two curves net out is the part Singhal will not pretend to predict. Labenz's own extrapolation is a weird 2028 in which a model can do a month of human work and still, on a given run, have a tiny chance of screwing you in some bizarre way.

What would a medical Move 37 look like — AlphaGo's famous move against Lee Sedol that no human would have played and that was brilliant in hindsight? Singhal does not think it is far away. People already tell him they saw several doctors, then ChatGPT flagged something they took back, and together they reached a diagnosis; whether that counts is "a matter of taste." He is more interested in world models in health: a much better grasp of you, your trajectory, and how that intersects with biology. Full in-silico bodies are expensive and not what current models are designed to do; predicting the result of an intervention given huge context is in-distribution and getting better. The team's mission runs through three channels: help consumers navigate their health, make AI-assisted care standard and pull clinicians out of paperwork, and push up the research ceiling. In biology, he notes, many breakthroughs had no physical barrier to arriving five or ten years earlier — only a shortage of human ingenuity applied to the data.