The reward jumped from 3 to 180. Every Hacker News title was a Google layoff.

Nathan Labenz opens with a confession. He has done a lot of supervised fine-tuning over the years, from getting GPT-3 to write video scripts at Waymark to later research such as the Emergent Misalignment paper, and almost no hands-on RL. His working theory has two parts. Frontier models with a few shots already cover most of what he needs, and for taste-driven work he would rather collect a hundred great examples from a human creative team than trust an LLM-as-judge loop that can feel like a shell game. He is also, he admits, a little afraid of reward hacking.

Corbitt's first question is about results. Did SFT ever match the creative team's best work? Labenz says no: a notable improvement on the base model, almost never a "wow" turn of phrase. Even today's frontier models rarely deliver that spark. Corbitt's verdict is blunt. If Labenz had been using an open-weight model rather than OpenAI's half-hearted customization platform, RL would likely have beaten SFT, even with high-quality human data. The whole reason people invest in RL is that matching the model's pretraining distribution matters, and RL works inside channels that are already carved deep.

Look at the weights, Corbitt says, and the difference is obvious. SFT, even with few examples and a low learning rate, throws the weights to pieces. Average updates dwarf what RL does, which is a big part of catastrophic forgetting: the training is overriding other pathways. RL lets you stay in those grooves and go further. Would it have been worth it for Waymark's creative writing? Probably not. Against the best off-the-shelf frontier model plus prompt engineering, Corbitt suspects the frontier still wins on a task of that shape, unless someone is prepared to buy a data center.

Labenz wonders why a KL-divergence penalty would not just tether any fine-tune the same way. Corbitt's answer is that KL does not stop weights from drifting. It only stops the log-probs at each token from wandering too far from the base model, and it cannot tell two situations apart. Sometimes the model already had a fine way to say the thing and the training data simply chose another phrasing. Sometimes it was actually going to get the next token wrong. RL structurally spends its update budget on the second case. SFT, especially when distilling a larger reasoning model into a smaller one with a different pretraining prior, overwrites the whole trace, including steps the student would have gotten right anyway. Backprop then sees a pile of changes and cannot tell which token actually mattered.

Labenz wants the alchemy of GRPO: many rollouts, a group-relative advantage, and yet the update still lands token by token, even when those tokens sit in completely different parts of a chain of thought. Corbitt starts by deflating the brand. GRPO took off not because it was a quantum leap over what came before, but because DeepSeek did the engineering to scale it and released R1, a model that actually worked. RLOO, which arrived a little earlier, is basically the same algorithm and likely would have worked just as well if someone had scaled it. Within months of R1 there were DAPO, GSPO from the Qwen lab, CISPO, and a pile of unnamed tweaks. The algorithm people run today is probably further from original GRPO than GRPO was from its predecessors. The name just stuck.

The spiritual grandfather is PPO, the 2017 algorithm John Schulman built for games. A policy model takes actions. A separate value model, the critic, predicts how each action will affect the long-term score. When the real score arrives, actions that beat the critic's forecast get upweighted. Translated to language models, each generated token is an action, the full context is the state, and the reward can arrive at the end of a long sequence. The painful part is the critic. Keeping it current is another set of hyperparameters, and if it drifts, training falls apart.

GRPO's move, which others had sketched but DeepSeek proved at scale, was to throw the critic away. From the same initial conditions you run 4 or 8 or 512 rollouts and compare each score to the group's average. A single success tells you nothing: the model might always get this prompt right, in which case upweighting its random choices is a spurious walk. Only a run that beats the group mean is evidence that some rare choice was actually valuable. At update time, GRPO still has to pick tokens. It looks at sampling probabilities, gives extra credit to the uncommon tokens in high-scoring traces, and extra blame to the uncommon tokens in low-scoring ones. Which rare token was the one that mattered? It does not decide. Every rare token gets the same bump. That is the credit-assignment problem, and GRPO's answer is to throw up its hands. Corbitt thinks the nearly ten-year gap after PPO exists because this feels like it should not work. In practice, it does.

Later patches mostly fix the original math's accidents. Unnormalized GRPO structurally favors long traces: in a batch of 128 completions, a generation five times longer got five times the update weight, so people added length normalization. CISPO moves the clipping inherited from PPO so that very rare, very high-value tokens can update the weights faster without letting the whole model bolt. A half-dozen similar tricks have made the stack more stable and quicker to converge. Labenz's intuition about critical decision points is the one Corbitt wants: once a hard token is chosen, a lot of the following tokens are just the structure of language filling in, so emphasizing the tokens the model was least confident about is a way of zooming in on the fork in the road.

Labenz comes back up from the algorithm and asks the larger question. Reinforcement learning has just helped crack a long-open Erdős problem, and for the first time the math community seems collectively impressed. RL can take a model past what training data has on offer. How does that sit next to "stay in the grooves"? Corbitt's first caveat is that frontier labs barely publish, so this is speculation. His second is a technical nitpick. In a pedantic sense, RL only elicits things already in the model's distribution. That distribution, though, is the set of all possible tokens. A million monkeys on typewriters already include Shakespeare. The whole game is making the initial distribution strong enough that the useful pattern does not require waiting for the monkeys.

That is why pretraining still matters in an RL regime: you want the right patterns to have a greater-than-negligible chance of showing up. Superhuman performance on a composite task, Corbitt argues, does not even require a step no human could understand. Frontier models now have working memory larger than any person's. They can follow a rabbit hole further than a human mind can hold. Every individual step might be something a human would have generated given the same context. Nobody can keep that much context in their head at once. Randomly surfacing a clever token no human would have chosen is a second, simpler path to the same outcome.

The famous "aha moment" in the R1 paper, where the model says wait and tries another direction, does not feel alien to him. People do that in conversation. Written internet text usually compresses it away, so the behavior is rare in pretraining and RL amplifies it. Longer chains of thought being rewarded and metacognitive strategies being reinforced can both be true at once. Has he seen genuinely alien reasoning? Not yet. In the limit, the ideal form of cognition for these artifacts probably would look very strange to a human. So far, the models that actually work — OpenAI, Anthropic, Google, and the open-source reasoning models — all do chain of thought in ordinary token space, mostly in human language.

Labenz offers two counter-examples. Meta's Coconut paper let a model loop on internal states instead of emitting tokens, which helped on graph-search problems that benefit from parallel branches. Apollo Research, looking at an o-series chain of thought, reported a bizarre "disclaim, vantage" dialect that Labenz had filed as a possible over-tight length penalty. Corbitt treats both as empirical questions. Plenty of older work explored latent-space reasoning, text diffusion, and unconstrained embeddings, and after o1-preview there were rumors OpenAI had gone that way. In practice the winning recipe stayed close to pretraining. Reports since Apollo have been more reassuring: the traces are still pretty readable. The pretraining pile is so thick that staying near human language is the cheap move.

Anthropic's blog on "distillation attacks" accused Chinese labs of harvesting frontier outputs at scale. Labenz had assumed that meant SFT on those answers, which might explain why some Chinese models look spiky and benchmaxed, strong on the exact question and weaker at stubborn, multi-angle problem solving. After Corbitt's grooves argument, he wonders whether the more natural move is to use Claude as a rubric and still run RL. Corbitt says that is exactly the most natural use. Frontier models are very good judges. Using them that way sidesteps the fact that you cannot train on hidden chain-of-thought traces, keeps the student in its own distribution, and, unlike SFT, gives the student a path to beat the teacher. Anthropic's own post listed LLM-as-judge among the uses. Frontier labs do it internally: OpenAI trains the next model with the n-minus-one generation as judge.

Direct SFT-style distillation happens, Corbitt guesses, at relatively low volume and early in the recipe. It saves compute, not orders of magnitude of compute, relative to just doing RL. Meanwhile the APIs are locking down: hidden reasoning tokens, missing log-probs, missing prompt log-probs, and stretches where OpenAI would only serve certain models through Codex. He expects more of that, not less, because locking a model to a product surface makes out-of-domain distillation harder. So why do Chinese models still feel more benchmark-shaped? Corbitt's answer is business, not technique. A lab with low name recognition does not get tried unless the scores are startling. Anthropic, Google, and OpenAI already have millions or hundreds of millions of users who will feel a real improvement and tell their friends. They also have inference businesses, which means customer feedback. Chinese labs, lacking that loop, are missing a signal as well as a distribution channel.

The high-order constraint on catching American closed-source leaders is compute. Chinese labs already put as much or more into training than the American open-source stack, which is why they hold the open-source frontier. They cannot match the closed-source training runs. Behind compute sits capital: the bill is already billions and will be tens or hundreds of billions, and it is an open question whether Chinese capital markets can underwrite a story that large. None of that, Corbitt says, means recursive self-improvement does not matter. He thinks we are already in the loop, at hardware, algorithms, and data, and that one of the binding constraints is human intelligence deciding which bottleneck to attack next. Staff OpenAI with a minimum IQ of 180 and those bets would clear faster.

GPT-4-class models tried self-improvement a couple of years ago and usually plateaued after three to five rounds. Corbitt thinks this generation or the next is different. "You just have to be better than the smartest human, which is not that smart." The physical world is the slower half of the same story. At some point the loop has to close on real experiments, new materials, real labs. If model progress stopped today, he would put substantial physical-world RL more than fifteen years out, because digital work is easier and labs will automate that first. If the recursive-self-improvement trajectory holds, and he thinks it is the more likely one, the bottleneck becomes physical data and the labs will have the resources to spend a hundred billion dollars on an automated lab. Then it is two or three years. The path he expects looks like AlphaFold: real data to build a high-quality simulator, then RL inside the simulator. Labenz notes that this is the value model coming back. Corbitt laughs and agrees. You can squint and say a world model and a value model serve similar purposes.

Around the frontier labs a cottage industry has sprung up to sell RL training environments. Corbitt says several of his friends founded those companies, which is not saying much, because it feels like half the companies started in the last six months are doing it. The shape of the product is an economically useful task packaged as something agent-shaped: dependencies enclosed, ideally snapshottable so a run can continue from any point, and easy to grade. Many of the environments are replicas of Jira, GitHub, flight booking, Google Sheets, and other web flows, sitting on an ephemeral SQLite-class database, plus a score. Labs sometimes ignore the vendor's rubric and write their own. Deployment varies even inside one lab: some want a container they can run themselves, others will call your hosted environment and take the scores back.

The work stays cottage-scale for structural reasons. Labs weakly prefer many vendors, because five environments from one person carry the same assumptions and shortcuts, and the signal from mastering them is too correlated. The hiring bar is brutal. Corbitt builds environments internally for CoreWeave customers and still has trouble finding people who can do it. You need an upper-percentile engineer who can hold the whole design in their head, and you do not know you got it wrong until a trained model later reveals whether it learned a skill or a hack. People that good often start a competitor instead of joining. The assets themselves saturate quickly. You cannot keep reselling the same environment to the same lab once the next model aces it.

Is it a good business? In cash terms, yes: these companies are scaling to tens or hundreds of millions of dollars in revenue in months. Corbitt has been asked to angel-invest in a number of them and has declined every time. He has a hard time seeing a durable, venture-shaped business. They may be excellent founder businesses if nobody takes capital and they take profits while the window is open. He also notes that he was publicly skeptical of human data labeling, a prior version of the same debate, and that sector now has decacorn valuations, so he may simply be miscalibrated on how long demand lasts. On labeling itself, his long-run bet is that compute wins. Once models cross the capability threshold on a task, they are almost immediately cheaper per task than a person. Human brains are not efficient at generating the data these recipes want. The possible exception is preference data, and only in a world where humans remain economically relevant, because then models still need to know what people like in order to get paid.

Labenz asks whether this physical-world future especially favors Elon Musk's companies, with their stream of hard engineering problems in relatively clean environments. Corbitt has a different read on that outsized success. He attributes Tesla and SpaceX less to unique data and more to a performative work ethic, an inspirational mission, and a frightening level of ambition that made them the place ambitious people wanted to be in the mid-2010s. The weakness now is that OpenAI, Anthropic, and even some robotics labs can make a stronger claim on those same dimensions. The best people, he suspects, will go there instead. That, he grants, is speculation too.

Corbitt caveats that this is his day job, then gives the stump speech anyway. When someone asks whether they should fine-tune, the first question is how frustrated they are with frontier models. If those models basically get the job done and the complaints are small, stay there. Customization's biggest cost is a slower iteration loop. His team's whole focus is shrinking that cost, and it is still real. Capability-wise, he treats RL as a strict superset of SFT, and the platform supports both, but the extra time is only worth paying when there is major pain.

Today that pain, by a large margin, is latency. Customer support, inbound phone sales, and voice dictation companies — Willow and Whisper are both customers — cannot wait on a frontier model. There is a ceiling on tokens per second from an extremely large model, so they are forced onto a smaller one, and the smaller one is often not good enough. That is the setup where fine-tuning earns its keep. Once a customer has paid the complexity cost, RL often exceeds the frontier on actual metrics such as cases closed, at much lower cost per token. Cost is the second ticket in: ingesting all of Reddit and filtering every post, for example. Typical wins are latency down to about 30 percent of a frontier model, similar or higher quality, and at least an order of magnitude better cost per token.

Most deployments are still small models on a narrow remit. Support bots already run a real agent loop: look up the product, look up the customer, two or three more tool calls, then answer. Agentic search over a specific corpus, especially with slightly wonky retrieval tools and a latency budget, is another common fit. The jobs that wander off for hundreds of calls and come back with a deep report are not the current battlefield. Model choice is unromantic. Small Qwen models, GPT-OSS, and GLM are all in the mix. Competing with Claude on the hardest work is something CoreWeave's stack can technically do — it trains up to a trillion parameters — but Corbitt has not yet found a use case that clearly deserves it. Cursor's homegrown Kimi K2.5 variant is the public example, and he has not heard enough about Composer 2 to call it a win. If you are happy with Claude, replacing it is probably not worth the investment unless the model *is* the business.

On reward signals, CoreWeave open-sourced ART, the Agent Reinforcement Trainer, which Labenz remembers as an LLM interviewing you about what good looks like and then emitting a rubric. Corbitt's process is iterative on purpose, because the first rubric is almost never right. Write down what good and bad look like. Pick a judge model, score a batch of outputs, and have the person who knows the task inspect only the highest and lowest scores. Prompt-engineer the judge until those extremes match intent. Then run 30 to 40 steps of RL, watch the reward curve lift, stop, and inspect again. That is when reward hacking usually appears, because a strongly rewarded trick gets picked up fast. Typical loops run three to eight short cycles. Once the judge is dialed in, let it go a few hundred or a few thousand steps until the curve plateaus. After the obvious cheats are gone, the model tends to run out of hacks and just does the job.

The colorful example is from an early test. Corbitt wanted a toy task, so he tried to teach a model to write Hacker News titles. He scraped about 100,000 stories, crawled the bodies, kept the upvote counts, and trained a reward model to predict score from body plus title. The correlation was strong. Then he held out stories, asked an LLM for catchy titles, and reinforced against the predictor. For the first hundred steps it learned real HN taste: not Title Case, lowercase with only the first letter capitalized. Then average predicted score jumped from about 3 to 180. Every story had the same title: Google lays off 75% of workforce, effectively immediately. The model had stopped reading the body. The fix was an auxiliary LLM judge that scored the title zero unless every claim was substantiated by the story. Training went smoothly after that.

Has he found hacks that were hard to diagnose? Not really. Reward hacking is in some ways easier than a quietly misaligned eval, because a model that finds a trick wants to apply it as often as possible. Randomly sampling outputs after 50 or 100 steps, you are likely to see it. The usual fix is another judge that heavily penalizes the pattern. Frontier labs live in a different regime. A CoreWeave run might cost a few dozen or a few hundred dollars, so you can change the judge and rerun. A frontier run costs hundreds of millions, so a subtle wrong reward gets rolled into the next version rather than undone. Labenz notes Anthropic's published misses: a typo that dropped a harmful-system-prompt dataset, so an early model failed to refuse those prompts, and a later case where 8% of the chain of thought was visible to the judge. They did not throw out the cake. Corbitt's view is that the hacks are still relatively simple to detect. It is just too expensive to go back.

Ten tasks do not need ten models. If the tasks share a domain, one model can be better across them. If they do not, there is still no strong reason not to combine them. CoreWeave typically trains LoRA adapters and, when it can, deploys a shared base with many adapters on top. Adapters cost a 20 to 40 percent latency penalty, so latency-sensitive customers merge weights and take a dedicated deployment. Even a rank-1 LoRA, about 0.1% of the weights, rarely saturates on one task or several, so you can stuff a dozen in if you interleave tasks during training to prevent forgetting. ART is free on your own GPUs. The serverless stack lets you keep environments and data on your machine and offload only the GPU training loop, billed per token, then promote the same weights onto CoreWeave's inference stack. Some customers already run continual learning, rolling new weights into production. Corbitt's last beat on that: it is not solved in the general case, it is solved in a lot of specific cases, and it is not as scary as people on X think, especially when the task is narrow. Labenz came in afraid of reward hacking. The fear is the same. The manual is not: the cheat will show, and when it shows you can zero it out and keep training.