Oscar Health's AI Playbook: Translate the Bureaucracy First, Then Reinvent Care

Mario argues that LLMs are unusually well matched to healthcare because they move between informal language and formal language in both directions. Healthcare combines highly structured artifacts such as ICD-10 codes, CPT codes, utilization-management guidelines, and regulated rule bases with deeply human material such as provider-patient conversations and electronic medical record notes. That mix, he says, is why healthcare had relatively little algorithmic coverage over the past 15 years: there was no equivalent of PageRank, and predictive models usually stopped at the boundary between formal and informal language. LLMs are strong exactly at that boundary.

The first wave of use cases is therefore administrative. When Oscar pays a claim, it runs the claim through a large rule base and can produce a long trace of which rules fired, in what order, and which ones failed. An expert can read that trace, but a layperson cannot; an LLM can translate the formal trace into plain language that explains what happened. Mario expects the next five years to make this kind of explanation real-time, bidirectional, and much more transparent.

In that near-term vision, a patient could know immediately what a visit will cost, whether a better alternative exists, and why a claim or authorization is progressing the way it is. Longer term, Mario says the bigger prize is replacing some clinical intelligence with machine intelligence and bringing down the cost of doctor visits by an order of magnitude. Oscar is sequencing accordingly: three of its top four current use cases are administrative, and only one is clinical, with the administrative ones treated as deadlines rather than open-ended research.

Mario contrasts administrative and clinical use cases by the cleanliness of their inputs and outputs. In claims, the system has already spent decades standardizing the data: clearinghouses, EDI formats, and clear outcomes such as paid, denied, or adjusted. The model can operate on a relatively well-defined playing field. Clinical use cases lack that discipline, because the inputs and outputs are messier and often depend on context that never reaches the record.

Oscar runs a medical group with about 150 caregivers, records virtual care conversations, and has an LLM generate summaries and medical records from them. Mario says the biggest difference between LLM-written and human-written summaries is that humans include subtle contextual knowledge the model could not have had, such as remembering a prior conversation that is not in the medical record. In that setting, the LLM is on an unfair playing field unless the system improves the model's horizon of knowledge, not just the model itself.

Oscar's care teams are organized around geographies because local context changes the conversation: a member in Manhattan may be deciding whether to take a bus or subway, while a member in Phoenix is deciding whether to drive. That kind of everyday context is obvious to a human agent but absent from the LLM unless it is explicitly supplied. Mario's lesson is that many clinical and service improvements will come from giving the model far more contextual knowledge, not only from a better base model.

Mario frames Oscar's business as selling a package of healthcare services for a subscription-like premium, then paying for whatever care members use. That creates three levers: grow and retain members, run operations efficiently, and reduce medical costs while improving outcomes. AI work is mapped to those levers rather than treated as a single innovation program. The financial lens matters because the levers have very different payoffs.

Oscar entered the Affordable Care Act individual market partly to have more control over retention than an employer-focused insurer would. In a typical insurer, about 85% of costs are medical and about 15% are administrative, including commissions and marketing. With margins often between 2% and 5%, a small percentage saving on the 85% medical-cost base can double margins, while automating internal admin saves money but has a lower ceiling. Mario says the medical-cost side is more fun, more powerful, and better for the healthcare system.

The business has an extreme seasonality: six weeks at the end of each year when members re-enroll or new members sign up, which Mario compares to a Christmas retail peak multiplied by ten. Oscar has therefore used AI heavily in retention campaigns that remind members why they should stay and what the plan did for them in the past year. The company has found persona-level differences, such as colorectal-cancer screening reminders working particularly well with Asian-American members in its population.

Mario also sees a sharp split between chronically ill and generally healthy members. Chronically ill members already know how to navigate the healthcare system, so messaging about convenience and removing steps resonates more; generally healthy members respond better to empathy and encouragement to go back to the doctor for the right reasons. Language and phrasing matter, and LLMs can extract concerns and persona signals from customer-service conversations. The same models help fill gaps in ethnicity data by inferring from names, conversation language, or other signals, which supports matching members to the right doctors and conversations.

Mario says GPT-4 works well out of the box for extracting issues, personas, and language from customer-service conversations. Oscar has ethnicity information for roughly 65% to 70% of members and uses LLMs to infer the rest from names and prior conversation language, which is easier than building bespoke models. The company has deployed call summarization for care guides, lab-test summarization in the Oscar Medical Group, secure-messaging medical-record generation, and claims explainers for internal care guides. These are among the most tangible administrative use cases Oscar is running today.

The PMPM math is unglamorous but clarifying. Oscar collects roughly $600 per member per month in premium, with about 85% going to medical costs and 15% to administration, so one or two cents PMPM does not shoot the lights out. Ten such savings start to become interesting. Mario calls PMPM the magic four letters in healthcare because it forces every AI project to connect to the economics of the plan.

Compliance is a real design constraint, not a reason to avoid healthcare. HIPAA limits the sharing of patient-specific information, and vendors must sign business associate agreements (BAAs); Mario says Oscar was the first organization to sign a BAA with OpenAI directly. When a provider such as Google releases a new model like Gemini Ultra, it does not automatically fall under the existing HIPAA agreement, so Oscar uses a mix of anonymized and synthetic test data until the model is covered. That creates a typical three-to-four-month lag between a model's release and its use on real medical data.

Selling into hospitals is less about certifications than about trust. Mario says HyTrust certification helps clear final hurdles, but hospitals and insurers still run long security and policy checklists, and the decisive factor is whether the customer trusts the team and the solution. He warns that health tech often overinvests in product tweaking and underinvests in enterprise process, even though the best products do not always win; the best enterprise sales processes often do. He also helped write a consortium document on healthcare AI principles that emphasizes democratizing analytics and getting more people into contact with models such as GPT-4 and Claude.

Mario gives a deliberately simple failure case: take a hundred customer-service calls, ask GPT-4 to characterize each call, invent a taxonomy, and count how many calls fall into each reason. GPT-4 fails miserably when the task is packed into one pass, and it fails in a strange way if the context is too large. The problem is not that the model cannot understand a single call; it is that the task combines identification and counting in one token-generation process. That composition is where the model breaks down.

He explains this through the layer structure of Transformers. Interpretability work on early models showed that when a model fills in a sentence like 'the capital of France is blank,' the answer can be identified at a middle layer and then remain stable through later layers. The model is doing layer-by-layer processing that ends in a synthesis. When a task asks it to process many calls and then count them in the same pass, it effectively runs out of layers for the combined operation.

Chain-of-thought works, in Mario's framing, because it chains multiple model calls together and expands the effective layer space. Each step gives the model a new bottleneck and a new chance to process part of the problem. The hard part is knowing how composed a task really is: counting call reasons is easy to decompose, but deciding whether a patient is a smoker may hide many subtasks. Mario says that ambiguity is still one of the biggest things Oscar has to think through.

In utilization management, Oscar answers questionnaire-style questions that decide whether a member gets care, and one question - whether the member has a post-traumatic injury - produces the highest false-positive rate of any LLM question Mario has seen. His speculation is that the concept is broadly familiar to laypeople, so the training data contains many references that have nothing to do with the utilization-management definition. The model brings a general-person association to a highly specific clinical and administrative context, and that mismatch creates errors.

Oscar's workaround is a self-consistency questionnaire. The team gives the model the definition and a few short examples, then asks it to generate 30 different ways the condition might appear in medical records. Those 30 formulations are fed back into prompts and executed independently, then the model rates the independent reviews. Mario says this adds roughly 10 to 20 percentage points of accuracy, and the independence matters because a single pass causes token pollution or cross-pollination between the subproblems.

The claims assistant faced a different version of the same problem. Oscar's claim system is a giant, audited rule base with its own domain-specific languages, including one called Layer Cake, and a claim can produce a thousand lines of rule logic. Early attempts ran into context-window limits and, even when the trace fit, GPT-4 became less reliable as the decision tree grew longer. The part Oscar has more or less cracked is explaining why a claim was denied; explaining exactly why a claim paid one dollar rather than ten remains harder.

The solution was to give the model traces at a chosen level of hierarchy. Oscar omits subprocedure calls and function outputs, then asks the model to decide which functions it wants to double-click into for more detail. In the denial case, it is usually possible to find the function call that caused the denial, so the model can focus there instead of reading the entire trace. Mario describes this as systems design: the prompts matter, but the chaining and the order of calls matter more.

Mario estimates that 90% of Oscar's prompt work is trying things out. He reads papers because he likes to, takes notes on them, and publishes the notes, with a strong bias toward interpretability and weird failure cases because they reveal how the models work. What he sends to the team most often is work on prompting strategies and systems design: what the first call is, what the second call is, and how the chain is arranged. The self-consistency workaround was discovered internally before the team mapped it to the literature.

He offers a custom-GPT example to show how strange the programming model can be. Mario tried to make a custom GPT follow a formal state machine, and it kept losing track of state or inventing a new state; he eventually used Python code inside the GPT to force it back to state zero. He calls this an unbelievably weird programming paradigm and a novel kind of intelligence, which is why sharing experiments across companies matters. The ecosystem learns faster when teams publish what worked and what failed.

On model choice, Mario says Oscar has consistently found that specializing a model in a domain causes it to lose alignment. Even a simple instruction such as 'your answer needs to be in JSON' can fail half the time with a specialized model, while a general model with a JSON mode is more reliable. He therefore prefers the biggest general model for reasoning plus retrieval and fine-tuning rather than a smaller healthcare-specific model, unless the architecture can separate symbolic planning from content generation. He cites a paper showing RAG and fine-tuning provide independent gains, so teams should do both rather than treat them as alternatives.

Oscar's AI team grew out of a hackathon and became a seven-person pod with two product managers, data scientists, and engineers. The pod holds weekly office hours where anyone working on AI can bring a prompt, runs a Monday-night hacking session, and keeps its own three priorities so it does not drift into endless research. It also maintains a list of AI projects across the company. Mario is proud of the mix: something centralized for shared practice, something decentralized for product teams, and a low bar for anyone to show an experiment, even a failed one.

When Jacob asks what tooling Oscar wishes existed, Mario points first to a safety layer that verifies model output for insults, errors, or policy violations before it reaches a user. Oscar currently relies on human-in-the-loop review: lab-test summaries go to a doctor, and claims explanations go to care guides rather than directly to members. He also wants faster inference, because running GPT-4 a thousand times in parallel and picking the best output could outperform waiting for a better single model, especially where outputs can be verified such as code.

For a new healthcare AI startup, Mario would avoid generic clinical chatbots and look for an obscure, unglamorous niche where non-technical buyers have a specific pain. He suggests regulatory filings and documentation, including reports to state regulators, state health departments, and private quality organizations such as NCQA, because those reports are natural-language documents generated from data flows. He is more cautious about prior authorization startups because that capability is close to an insurer's core competency and risks a low ceiling unless it is highly interactive or platformized. He sees fraud, waste, and abuse as another area dominated by old-school players and ripe for better software.

Mario sees no reason healthcare should not eventually have AI doctors, because medicine is algorithmic and knowledge-based. The barriers are safety, physical interaction, and business model. Oscar once had physicians review a sample of claims and found that about 35% needed to be in person, meaning roughly two-thirds of claims could be virtual, but that is two-thirds of claims rather than two-thirds of people; a diabetic still needs foot exams and other hands-on care. Oscar also sees leakage: only about 28% of members who name a primary care physician actually see that physician in a year, which shows how weakly patients are attached to a specific doctor or group. The third barrier is that health systems have little incentive to shift to lower-cost virtual channels because that would invite pressure to cut reimbursement and capacity, while insurers could deploy automated virtual primary care but often lack member engagement.

In the quick-fire close, Mario calls clinical chatbots overhyped and voice outputs underhyped, especially for non-clinical uses where a voice interface can get far quickly. He names OpenAI and Hugging Face as companies he is excited about, citing OpenAI's model lead and Hugging Face's role as a shared home for models. He points listeners to Oscar's public AI papers and insights site and to his Twitter account for experiments, and he mentions long-running side interests in games, including an Oscar role-playing game and an LLM that adds balanced mechanics while a player plays.