A $9 AI nurse, an 86 on the medical boards, and a 10% extinction line

Nathan Labenz usually spends Cognitive Revolution episodes in long interviews with researchers and builders. This one is a scouting report he first gave to a room of retail investors who were mostly doctors and lawyers: a fast, analogy-free survey of medical state of the art, how the systems are made, how one might think about capital—explicitly not investment advice—and how the whole thing might hit society. Co-host Eric Torenberg is named at the open; the hour is Labenz talking, without a guest.

He is available for more of these talks, preferably remote with three little boys at home, and he flags new work besides Waymark and an advisory seat at Athena. Stellus AI, a new advisory and implementation firm, is aimed at companies of 50 to 1,000 people. One of the first projects is a plumbing-parts supplier whose decades of paper catalogs sit on a giant bookshelf; the job is to make that inventory actually findable. He is also floating a niche job board for past guests, with Elicit, Athena, and Stellus already hiring.

He refuses analogies because he thinks AI is as fundamental as DNA and natural selection, and walking around without those concepts leaves too much unexplained. He will describe circuits, loss, and scaling in literal terms, as simple as possible without being wrong. The two numbers that bookend the hour are a $9-an-hour AI nurse and a survey in which 48% of recent top-venue authors put at least a 10% chance on human extinction. Both, in his telling, are currently true.

Keep tabs on capability, he says, because the floor has already moved. Since the middle of 2022 it has been true that the best AI systems beat the average human at the average task: pick a typical piece of white-collar computer work, give it to a typical intern and to the model, and the model wins. That has held for almost two years. The next layer is expert performance on routine tasks, where routine means common and we know what good looks like—the standard of care in medicine.

The speed still startles him. GPT-2 shipped in 2019; he read about it in the hospital while his wife was about to deliver their son, who has just turned five. Five years later GPT-4 averages 86 on MMLU, a pile of undergrad and graduate exams across many fields. That is closing in on the human expert in-domain. Ask one human to sit every exam in every domain and GPT-4 is notably stronger.

The medical curve is sharper. Passing a U.S. licensing exam sits in the low 60s. Google’s Med-PaLM was the first system that pretty clearly passed, in December 2022; Med-PaLM 2 hit 86% in March 2023. In a nine-dimension head-to-head, doctor-judges preferred the AI on the good traits—consensus, reading, recall, reasoning. The one bad trait the AI did more of was bringing up inaccurate or irrelevant information; humans did all the other bad things more. On differential diagnosis versus primary-care physicians, the AI’s accuracy is higher across exact match and more permissive top-k definitions. Labenz’s personal line: if he had a condition that genuinely worried him, he would not be without an AI doctor, and he would not be without a human doctor either. Cross-reference both.

Two applications he thinks fly under the radar. Virtual tissue staining can turn a biopsy process that takes hours—too slow for the operating room—into images in seconds, so a surgeon removing a tumor can see stained results in real time. Literal mind reading is further along than most people know: models trained on fMRI reconstruct the image a person was looking at, and the data needed to decode one brain has fallen from 30 to 40 hours of scanning to about one hour. As a kid, he says, that would have been major news. Today it gets a few retweets.

The systems arriving are not the ones people expected. The forecast was that AI would automate low-wage, low-status labor first. Instead, doctors and lawyers are feeling it while the proverbial plumber or building custodian is not. A study that split jobs into five quintiles of wages and status found the fourth quintile most exposed. Read the area under the curve as work AI seems poised to do, and the area outside as work it is not. Roughly, about half of people have about half their tasks in range. He thinks society will be adjusting to that mismatch for a long time.

Commercialization is already quoting labor-market prices. Hippocratic AI charges by the hour, $9, for AI nurses that are scalable and clonable like software but billed like a shift. The company is deliberately not trying to diagnose, even given the head-to-head results above. The bet is follow-up work nurses do, or that nobody does because there is no time: pre-op calls, discharge instructions, medication checks, how are you feeling. Software replicability plus an hourly wage is a combination he says has not really existed before.

Where it stops is anything that is not routine. Can AI do new science? In general, today, no. Narrow, purpose-built systems such as AlphaFold are the exception, and they matter. General-purpose chat systems cannot handle novel, unfamiliar situations where somebody has to figure it out for the first time. Labenz used to summarize this as “no Eureka moments.” He has had to change it to “precious few.”

One exception is a robot hand learning to twirl a pencil with reinforcement learning. Early in training the agent needs a reward function, a score that tells it when it is doing better. Writing that function is expert work; a person on the street would not even parse the sentence. GPT-4, he says, is significantly better at writing those robotics reward functions than the human specialists who do it. That does not happen often. It was enough to move his bottom line.

A second example is still landing. A ChatGPT-like model trained on 300 billion base pairs of bacterial and phage DNA can generate new CRISPR-like complexes. Researchers can also perturb the model, watch how confused it gets, and use that as an in-silico test of which genes are essential to a cell’s survival—a cumbersome wet-lab process that now runs in seconds. This, he notes, is from the last few weeks. The impact across fields is just beginning.

The core inputs are data, compute, and algorithms: a three-legged stool, none optional. All of these systems are information-processing circuits, inspired by neuroanatomy and not much like it. They are unidirectional and differentiable, proceeding layer to layer, without the brain’s cyclical cross-talk. A toy example is classifying a handwritten seven from raw pixels until the right output lights up. For intuition he points to 3Blue1Brown’s visualizations rather than a metaphor.

Learning sits on a loss function, a score for the output. Weights usually start random, so the first pass is garbage. Back-propagation, the chain rule of calculus, then asks every parameter whether a nudge up or down would make the score a bit better. On a GPT-3-scale net that question is asked 175 billion times. Loop it and you have gradient descent, walking a loss landscape toward lower loss. The system has to be differentiable and you have to have a score. That is the core, with many tactical complications piled on.

The old game was small curated sets such as MNIST, handwritten digits with labels, and a split into train and test. Train long enough and training loss keeps falling while validation gets worse: overfitting, memorizing idiosyncrasies that do not generalize. The unlock was clever losses that turn unlabeled web-scale data into training signal. Next-token prediction makes all text usable because the text is the answer key, with a vocabulary of 50,000 to 100,000 possible outputs. Image denoising adds noise on purpose and trains a model to remove it, which makes all pictures usable. He cannot overstate that shift from labeled sets to unsupervised, self-documenting data. Every major scaled system now sits on it.

Compute is why GPUs matter: massive parallelization, so you are not asking 175 billion tweak-up-or-down questions in sequence. The dominant circuit is the Transformer. He runs a jargon glossary—tokens, embeddings, neurons, activations, attention, MLPs, nonlinearities, parameters, logits, the forward pass—because people get lost in the words. Then you still need an algorithm that turns data and crunching into something that works. All three legs, or it does not stand.

Scaling laws ask how good a system will get for a given budget. There is always a Pareto trade: a bigger model costs more per token, a smaller one can see more data but learns more slowly, and each compute budget has an optimum. A 6×10^18 FLOP study is already large and still eight orders of magnitude below the White House reporting threshold of 10^26. Those small curves let labs project loss, pick parameter counts and token counts, and back into a bill. GPT-4 was trained on 10 trillion tokens at a reported tens of millions of dollars of compute, not counting salaries or data. GPT-5 is rumored to be aiming at 100 trillion tokens and a few billion in compute.

What unnerves him more than the bill is that the models learn things nobody asked them to learn. Early OpenAI trained a small next-character model on Amazon reviews and found a neuron that lit up on positive reviews and went strongly negative on bad ones. Isolated, it would have been a state-of-the-art sentiment classifier, and sentiment was never in the loss. Greg Brockman’s line is the one Labenz wants people to take home: semantics emerged from a syntactic process. An Othello model that only ever saw move lists such as F4 internally built a two-dimensional board. Vision nets grow window, wheel, and car detectors no one specified. High-order, human-recognizable concepts fall out of a purely syntactic training loop.

Grokking makes the same point on a clock. On modular addition, a model first memorizes, validation gets worse, and the old rule said stop: you have overfit. Someone ran it out toward a million optimization steps, on a log scale, and the net suddenly generalized. Reverse engineering showed an alien algorithm of sines, cosines, and Fourier rotations no human would use, but a reliable one. Loss can be predicted from scaling laws. When a particular skill shifts from memorization to a general algorithm cannot. That is why he calls “certain capabilities remain hard to predict” the most important sentence in the GPT-4 technical report. Inverse-scaling contests even found tasks, including hindsight neglect, that got worse for several generations and then, with the next one, grokked to perfect.

They can be human-level without being humanlike. AI wins on breadth—it has read the internet and speaks the languages—plus speed of at least 10×, cost usually 90% lower, parallelism, cloning, and 24/7 availability. A paper titled along the lines of “natural selection favors AIs” notes that an advance can be ported across models immediately, while humans learn slowly. Evaluations even give AI the edge on bedside manner. Humans still win on expert depth, breakthrough insight, and coherent memory. Model memory today is limited and brittle. The glaring weakness is adversarial robustness: they are easy to trick, and AI Twitter is full of jailbreaks that would never fool a person.

He skips his business-practice slides and turns to the not-advice. A friend’s question is whether we are approaching a “big tech singularity,” in which dominance of data, compute, and algorithms lets the giants walk into other industries. Labenz says that looks increasingly plausible, and prices seem to agree. On a compute chart the big four are Google, Microsoft, Amazon, and Meta; Apple is fifth; Alibaba leads China; Oracle, a multi-hundred-billion-dollar company, is not in the game. There is no realistic on-ramp for a newcomer to train at that scale if the incumbents are already spending billions per model.

He does not pick stocks, with one historical exception for a friends’ club: Nvidia about two years ago, because everyone chasing scaling laws buys compute from the same place and algorithm progress has been incremental. AMD and Intel get attention because Nvidia’s margins are extreme, and Washington wants manufacturing out of Taiwan. His safest bet is still the compute layer, chips and big tech, because those firms hold all three legs. Live players: OpenAI leads; Google has the deepest bench and broadest agenda; Anthropic is a small dark horse tied to both Amazon and Google; Meta is shipping Llama 3; Microsoft is less cutting-edge in research but rich in data centers and bound to OpenAI. Never count out Elon Musk at Tesla and X. Character.AI’s girlfriend products hold attention long enough that birth rates become a long-term question. France’s Mistral may live on EU industrial policy. Chinese researchers are thick on the NeurIPS floor, but a chip ban without a domestic industry could stall the next scale jump. Apple is behind on Siri and fifth on compute, with more money than it can spend.

Whether we all die, he says, depends on whether scaling continues. They are called laws; they are better called trends, and trends can break. The single biggest question is whether capability levels off at human expert or blows past it. His guess is something alien: superhuman in some ways, weird and weak in others, and hard to predict. What he finds strange is the race itself. Nobody has a real plan for controlling superhuman systems. Sam Altman has said the worst case is “lights out for all of us.” At least, Labenz says, they know they are playing with fire.

A survey of people who published in top AI venues in recent years found that 48% gave at least a 10% chance of human extinction from AI. For his money, that is worth taking seriously. A statement that mitigating extinction risk should be a global priority was signed by Altman, Anthropic’s CEO, Bill Gates, the chief scientists of OpenAI and Google DeepMind, and two Turing Award winners. Forecasters’ consensus timelines sit in a range of a few years to maybe eight or ten. It may never happen. The field’s center of gravity is that it might, soon.

What he will watch: whether raw scaling holds all the way to superhuman systems; whether a better-than-Transformer architecture arrives, and recent results suggest it will; whether mechanistic interpretability catches up, and a survey of some 2,000 researchers said no, do not count on it to save us; which new modalities go immediately superhuman, as in cells, brains, and weather; whether systems become agentic over medium horizons, which hundreds of startups are already trying to make true; whether frontier labs actually pause when things get scary, as they have said they would; when governments lean in; and whether the U.S. and China slide into an AI arms race, which he calls definitely really bad.

He loves the technology. Waymark’s product is literally ten times better; he uses ChatGPT many times a day and likes having AI servants. The caution is about what comes next. Figure out how to use the current systems at work and in life. Hyperscaling another 10× or 100× past where we already are is something society would be wise to slow down. Learn what we have before trying to make a superhuman AI scientist, because he does not think we have the strategies that would keep that under control and make it go well.