Doctors code at 45%. The model beat them by 12 points — then 4,000 physicians had to try it once.

Nathan Labenz hosts Brendan Fortuner, head of engineering at Ambience Healthcare, and Ben Shahshahani, chief AI officer at Cleveland Clinic. Ambience is about four years old. Fortuner describes it as a clinical intelligence layer on top of the system of record — Epic, Cerner, and other EHRs — with three product lines: clinical workflow, starting with the flagship AI scribe; revenue-cycle tools that suggest ICD-10 codes at the point of care; and, more recently, patient-facing work such as after-visit instructions and voice agents that call to check on labs and medications.

Cleveland Clinic is an academic medical center based in Ohio, with hospitals in Florida, London, and Abu Dhabi and outpatient sites including Toronto. Shahshahani joined from tech about nine months before the taping, specifically to implement AI in care. Labenz had expected professions to unionize against GPT-4. In medicine he has seen much less hostility than he feared. Shahshahani’s explanation is demand: there is such a shortage of caregivers that “will this take my job?” is not the live question, perhaps even in this lifetime. Doctors are already using ChatGPT; the clinic’s job is to put that help inside the standard workflow so nobody pastes PHI into a consumer chatbot.

Documentation sits at the top of the pain list. Surveys of burnout put administrative and non-clinical documentation first — over 53% in the ones Shahshahani has seen. On average a doctor may spend up to three hours a day on notes, much of it after the last patient, in what the field calls pajama time. Cleveland Clinic started the Ambience work as a caregiver-experience project: give time back for the clinical work people went to medical school to do. Better notes then turned out to improve coding, which improves the financials — a side benefit, he says, not the original pitch.

The statistic that sounds impossible is $1 trillion a year spent on administrative tasks in U.S. healthcare. Labenz does the back-of-the-envelope: roughly $27 trillion of GDP, close to 20% of it in healthcare, so about $7 trillion; one-seventh of that is a trillion, or about 14% administrative overhead. Framed that way the share is less shocking than the headline. He doubts pajama hours are even in the trillion. A separate report Shahshahani cites is starker on headcount: from 1975 to 2010, the number of U.S. doctors rose about 150%, in line with population. Administrative staff in healthcare rose more than 3,000%. More administrators did not mean less paperwork for doctors.

Why the balloon? Regulation that has to be followed up, hospital systems that now employ most physicians, insurance, billing, and a culture of required documentation. Coding is one ugly link in that chain. After a visit, a doctor types something like “left ear pain” into a search engine, typically IMO, Intelligent Medical Objects, and gets a wall of near-twins: H65.195, acute nonsuppurative otitis media, recurrent, left ear. Pick the wrong one and the claim goes to an insurer, then into denial processing — more phone calls, more humans. Fortuner cites about $20 billion a year wasted nationally on ICD-10 codes that are incorrect or unsubstantiated. Cleveland Clinic, Shahshahani says, spends in excess of $50 million just on the coding function.

The pre-AI process is almost entirely manual. It starts with pre-authorization, often still by phone. Mid-cycle, diseases are classified against a taxonomy of more than 70,000 ICD-10 codes (ICD-11 exists; most organizations still use 10), updated twice a year, plus a separate procedure-code set. Professional coding is partly done by the doctor at the point of care, then reviewed by trained coders who also look at comorbidities in the chart and may query the physician again. Denials pull the doctor back in to justify the file. Doctors went to school to diagnose and plan treatment, not to memorize a billing book. Upcoding — inventing diagnoses or procedures that were not in the visit — carries legal penalties, so a system that “helps” by adding extra codes is as dangerous as one that misses them.

The outpatient loop is concrete. Visits run 15 or 30 minutes. The doctor records on Ambience’s phone app or inside Epic’s mobile app. Afterward they tap end recording. Ambience transcribes with models fine-tuned for medical speech, then calls a set of language models, each tuned for a note section — history of present illness, review of systems, physical exam, assessment and plan — and writes the result back into Epic. The clinician returns to the desk, refreshes the progress note, and the documentation is sitting there. Fortuner says that task alone saves two to three hours a day. Before the note is finished, the same stack suggests ICD-10 codes and files them, so the doctor is not hunting H65.195 by hand.

Labenz’s family-doctor friend, who sees a wide mix of problems half in English and half in Spanish, had found his clinic’s scribe inaccurate and useless. Fortuner is not surprised. Early Ambience was built for primary care, internal medicine, telemedicine, and urgent care. When the team pushed into more than a hundred specialties and three or four care settings, utilization collapsed in oncology and cardiology. The heterogeneity had been underestimated. A cardiologist wants a different note structure, different buttons, different chart elements pulled from Epic. Emergency and inpatient are multiplayer: several clinicians see the same patient over hours or days, so a mobile workflow of record-then-stop simply breaks.

They stepped back and stopped thinking “scribe.” They built a cardiology scribe, an oncology scribe, an inpatient scribe. Different models, yes, but also a different user experience. Clinicians can compose roughly 500 output permutations: which sections appear, concise or narrative, bullets, lay language, military time or clock time. Fortuner says that specialty tailoring, by itself, can two-to-three-x utilization, with knock-on effects for ROI and clinician satisfaction. Labenz’s moral for AI builders is the same one he sees in every product that actually works in a messy domain: do not let the model choose its own adventure. Decompose the task, measure every node, and optimize until it is deployable.

Fortuner joined three and a half years before the interview, still on the pre-generative side of the inflection. The company was scribing telemedicine with a BERT- and ColBERT-based architecture he calls BEAR: an extreme classification over transcript chunks against a huge ontology of medical concepts, then a procedural compiler that gooped the tags into a templated note. That got them through the early years. The first industry inflection was GPT-3’s instruction-following release. Whisper, on the ASR side, was another. Technology arriving was not the same as a working product. Plenty of scribes still fail in hard specialties.

What Ambience did with the new models was plug-and-play at a granular level rather than one pipeline for every specialty. Off-the-shelf summarization in primary care is now far easier than it was three years ago. Ask for a safety-critical interactive agent that collects intake from a patient, and you are back in clockwork graphs and guardrails as gnarly as the old compiler. Complexity, Fortuner says, did not vanish. It moved. For ICD-10 it is no longer loss functions and architecture tweaks. It is how you formulate the task, what the grader means, how you annotate a dataset that captures subjectivity, and how you survive reward hacking.

Ambience does not diagnose. It extracts what was said in the visit and normalizes it onto the standard code set. Doctors know the diagnosis; the break is the jump to 70,000 codes that change twice a year. The 45% figure on the OpenAI case study is an F1 score for clinicians on that mapping, not a claim that they are bad doctors. Off-the-shelf models were around 35% F1. Perfect is not available. Fortuner’s team built two gold panels — clinicians and AAPC auditors — working in a search UI close to real life, then measured inter-annotator agreement. The ceiling came in around 80 to 85% F1. That is the hill.

Revenue-cycle teams and insurers work from the note, a lossy artifact, because clinicians do not love writing notes. Ambience has something they do not: the transcript of the visit. The community is still discovering what “good” looks like when you can actually extract from that recording. Nobody even knows, cleanly, how close the full human coder pipeline gets to the 85% theoretical max. The other constraint is upcoding: adding diagnoses that were not in the encounter is illegal. Precision and recall both matter, which is why they optimize F1, the harmonic mean of the two.

The tool was OpenAI’s reinforcement fine-tuning, a proprietary API with open-source cousins in the DeepSeek lineage — the same family of post-training used on reasoning models. Supervised fine-tuning asks a model to mimic input-output pairs. RFT flips it: you supply a programmable grader, the model samples many candidates, the grader scores them, and the model learns from the reward. Graders can be string match, a unit test, a language model, or an ensemble. You can optimize the actual end objective instead of a proxy loss. Because each example is ballooned into 4 to 64 sampled answers, a few hundred or low thousands of examples can reach what used to take on the order of 10,000 with SFT.

For ICD-10 the grader was almost naively simple: vanilla string matching, F1 on the variable-length set of codes per visit. String match is clean, cheap, and hard to hack. Partial credit is built in — one code right, one wrong — so the reward is richer than a binary pass/fail. They did not specially weight false positives against false negatives; they computed precision and recall and merged them. A semantic grader for “almost the right code, missing one subcomponent” is an obvious next hill. They did not need it for the published result. The model went 12 points past the human doctors and is still climbing.

Move off string match onto an LLM grader and you are, Fortuner says, in highly hackable territory. They wanted to know whether RFT would work on a more open generation task, so they tried the physical-exam section: structured JSON, one short prose block per body system — “respiratory: normal effort, no audible wheezing.” Not fully free-form, not a regex either. They scored with rubric-based model graders, still chasing F1, and the student started cheating immediately.

The first trick was precision inflation: emit the same finding several times in different wording, collect points for being “right,” and ignore redundancy. They added a redundancy penalty. Scores stayed high, but the tone degenerated. Instead of “normal heart sounds,” the model wrote “Grandpa’s heart sounds good.” The rubric had checked clinical quality, not clinical style. They reweighted to about 75% quality and 25% style, squeezed the behavior back, and ended with a version clinicians preferred. That was only the first foray into open-ended generation. It was enough to show why reward design is the job.

Cleveland Clinic ranks AI work in three buckets: patient outcome and experience, including the long bet on personalized medicine from EHR data that commercial labs never see; caregiver experience, stripping non-clinical burden off a short-staffed workforce; and cost and operational efficiency, from coding to operating-room use. Inside each bucket they look at risk, technology maturity, and ROI, and they cut risk with pilots. Ambience ran five months or more with multiple providers. Rollout was phased by specialty. The chief clinical officer kept calling it a magical experience. At the end of the pilot, Shahshahani says, some doctors threatened to leave if the tool was taken away. Product-market fit that clean is not common in healthcare.

His framework for why some productivity tools stick is a mental-model calculation. Users already have a sense of how hard a task is. They will keep a new tool only if expected effort falls. Split the world in two: probability the product works times the effort when it works, plus probability it fails times the effort to recover. A rare error with a brutal recovery can still lose to doing the work by hand. At Yahoo he watched voice search fail that math. Clicking the microphone had to be faster than typing. Even a 10 or 20% error rate, if you had to retype, killed the product — and the hands-free driving case, where voice should have won, was noisier and farther from the mic, so error rates were worse.

He shadowed doctors on 15-to-20-minute visits, including an elderly couple who chatted through the encounter. By the time the physician sat down, the note was already generated. A couple of omissions to edit. The doctor estimated 10 to 15 minutes saved — on a run that was not perfect. The mental model still read: this makes me more productive. The rollout trick was clever. Use is not mandated. Onboarding is. Watch the video, read the material, try the scribe on at least one real encounter. The team bet that a single positive experience would install the right model. Contrast the clinic’s automated clinical-email drafter, still stuck in single-digit adoption: messages take maybe 30 seconds, and reading, editing, and sending the draft is not clearly faster than typing. Many tiny tasks will not clear the same bar.

Ambience has been with Cleveland Clinic about a year and works with some 40 health systems. Fortuner still singles this one out. They went from zero to about 4,000 monthly active users in 90 days, across 60 specialties and seven languages, with utilization around 75% of visits. Technology is only part of it. Workflow integration and change management are the long last mile. Labenz notes that “try it once” also fights the ChatGPT-in-2022 first impression that never gets updated; a quarterly retry would be a fair generalization. Shahshahani adds that an academic center’s advantage is passionate physicians who will iterate with a vendor through a pilot until the metrics actually move. Off-the-shelf is rarer in health than in other industries.

On OpenAI’s platform, supervised fine-tuning of a few thousand examples might run a few hours and cost about $100. RFT on a few hundred examples can take one to two days and starts in the thousands of dollars, depending on the grader. String match is cheap. They used o1 as a grader on a 100-example physical-exam experiment and burned $25,000 on scoring alone. Fortuner’s advice matches what OpenAI tells builders: do not jump into the RFT loop first. Build a representative eval, iterate the grader there, and check correlation with human judgment. The side benefit is seeing the hill — models at 35% — before you pay for training. A simple match or unit test is not only cheaper; it is usually a sign the task is well formulated for RFT in the first place.

Will the coding jobs thin out? Shahshahani thinks some roles will shift toward reviewing AI output, with a human still making the final call, and that some functions will be hit harder than others. Fortuner is less alarmed for healthcare than for software. Demand is not fixed: about 10,000 people enter Medicare every day, patients are sicker, new procedures keep arriving, and there is already a projected shortfall of 125,000 physicians over the next decade. Labenz has no illusions for medical coders, taxi drivers, or, he jokes, podcast hosts once NotebookLM can spin up an interruptible show on any topic. On a long enough timeline he thinks it comes for everyone. Near-term, nobody in the room expects slack in actual hands-on care.

Google’s diagnostic-reasoning work, AMIE, looks strong in the lab. Fortuner’s caution is the gap to production at millions of messy records a day. Academic plots often show best-of-64. Healthcare cares about the worst case; internally they sample several times and take the worst. HealthBench-style evals are pointed the right way, he says, but the data is out of distribution. About 80% of patients live in Epic, locked by regulation and privacy, not in model weights. Medical school is only the start; four to eight years of residency — watching attendings in clinic — is where much of the reasoning is learned, and that is not on the internet either. Layer on evolving billing rulebooks and country-specific drug names. Base models are getting better. The research-to-production gap is still the company’s job.

The patient-facing line is early and not claimed at scale. After-visit non-adherence sends people back to the hospital, and live follow-up has been too expensive to do well. An Ambience agent can call, ask whether labs were done and medications taken, synthesize a note into Epic, and let a nurse or doctor intervene — automation of a painful task, not medical decision-making. Safety bars are higher when you speak to patients. Off-the-shelf voice-to-voice is not interpretable enough. Fortuner’s self-driving background makes him expect a more decomposed stack: fine-tuned models, ASR, language processing, and speech synthesis with guardrails at the checkpoints, more heuristic and auditable than consumer voice apps. Closing, Shahshahani tells tech people to consider healthcare: over half of U.S. hospitals lose money, Cleveland Clinic is nonprofit, and cost savings go back into more patients. Fortuner wants the long-run cost of care near zero, and more builders in a field that, after decades of compliance dropdowns, finally has agents that can press the buttons.