From 15% to 70% on Right-or-Wrong Scores: DeepSeek-R1 Laid Reinforcement Learning's Cards on the Table

On January 20, 2025 — Inauguration Day in Washington — DeepSeek released R1 with downloadable weights, and a few hours later Moonshot AI posted the paper for its Kimi k1.5 reasoning model. Labenz, recording a solo emergency episode of *The Cognitive Revolution*, says the sequence had "shades of OpenAI and Google" racing to preempt each other. Whether the labs chose the date, coordinated, or simply do not care when a U.S. president is sworn in is one of the questions he cannot answer from the outside.

What is clear, he argues, is the result. With this model, DeepSeek has joined the top tier of global AI developers. Moonshot might belong there too, but he will not crown a lab on benchmarks alone while the weights are still unpublished. The gap between closed and open weights is shrinking, and so is the gap between Western and Chinese labs that looked much wider two years ago.

He treats the week as more than a paper dump. Practical utility, open-source dynamics, and strategy all move at once. Scale AI CEO Alex Wang had taken out a full-page newspaper ad calling the moment an "AI war," a phrase Labenz says he totally hates and finds wildly irresponsible. An AI war, he says, does not exist and would be bad for everyone. If American strategy is built on preventing China from doing frontier work so "the good guys" can keep an unassailable lead, that window already looks short.

R1 is not trained from scratch. It sits on DeepSeek V3, the base model that had already made headlines as what the writer Zvi Mowshowitz called "the $6 million model." Western frontier training runs are understood to cost hundreds of millions of dollars. DeepSeek reached GPT-4o and Claude Sonnet quality with a single-digit-million compute budget, through data work, algorithm work, and neural-network designs coupled tightly to the chips they run on.

Labenz reads that number as a warning for export controls. The logic of those controls is to starve Chinese labs of compute so they cannot do frontier research. If a few million dollars of compute already gets you to the frontier, keeping the best Chinese companies out with denials, smuggling, and domestic fabs gets much harder. Six million dollars, he says, is just not that much.

V3 is a mixture-of-experts model with 671 billion parameters, of which about 37 billion are active on any forward pass. Extra parameters help the network absorb knowledge and learn faster while keeping inference cheaper than a dense giant. The catch is size: hundreds of gigabytes of weights will not live on a laptop, and decent throughput needs a GPU array. DeepSeek's business is an API; the weights are public so others can host them too, and even the stranger R1-Zero checkpoint was already turning up at inference providers.

The paper describes two models, and Labenz wants readers to keep them straight. The product most people will chat with is R1. The result he treats as closer to a scientific discovery is R1-Zero, named for DeepMind's AlphaZero, which dropped human game records and learned by playing itself. Here the same idea lands on language: pure reinforcement learning on top of V3, with no human preference data, no demonstration traces, no supervised fine-tuning, and no learned reward model scoring the outputs.

What remains is almost embarrassingly simple. The trainers give the model problems; a correct answer gets a reward and an incorrect one does not. A second "format" reward asks for a two-part reply, with thinking wrapped in tags and a short answer in its own tags. Most of the gains, Labenz says, come from the accuracy rule. The format rule is there to keep the trace in a shape humans can later productize.

Sparse rewards killed earlier attempts such as OpenAI's WebGPT, because a weak model never gets a problem right and therefore never gets a learning signal. DeepSeek dodges that trap with a strong base — V3 already solves *some* items — and with a diet of math and code. Coding tasks come with unit tests, so passing five of eight is already a graded signal rather than a binary coin flip. The optimizer is Group Relative Policy Optimization: sample many answers to the same prompt, often about sixteen, treat the mean score as baseline, and strengthen the replies that beat the model's own average. No extra judge network is required.

The graph Labenz keeps coming back to is not log-scaled. Early in training, R1-Zero's thinking trace averages maybe 500 tokens. After about 8,000 steps it is nearer 10,000, growing in a roughly linear climb that has not flattened by the time the paper is written. Longer thought raises the odds of a correct answer; correct answers get reward; longer thought is reinforced. One ceiling he flags is context: R1's window is 128,000 tokens, on par with GPT-4o and well short of Gemini's million-token class.

Inside those extra tokens, the authors report that reflection — revisiting earlier steps — and exploration of alternative approaches arise spontaneously. Nobody trained the base model as a problem solver; it was trained to predict the next token on internet text. Occasionally those traces include a human-like rethink. When that stumble works, it is reinforced. Users of R1 and of Gemini Flash Thinking see the stream immediately. OpenAI's o1 still hides the chain of thought and returns only a conclusion.

The qualitative exhibit is a math trace in which the model writes, "Wait, wait, wait. That's an aha moment." Labenz used to say AIs do not have Eureka moments; he has walked that back. The authors call the anthropomorphic rethink their own aha moment, a chance to "witness the power and beauty of reinforcement learning." On AIME, the same loop moves pass@1 from 15% to more than 70%, and 16-sample consensus from 25% to about 85%, catching the o1 announced in September after four months. OpenAI's later o3 curve, he notes, had not flattened either.

The cost of that climb is weirdness. R1-Zero's chains of thought are hard to read and often switch languages mid-trace, mixing Chinese and English without being asked. Labenz reaches for AlphaGo's Move 37: a play people first called a mistake, later a masterpiece, produced by a system that was never optimized to make sense to spectators. Reinforcement learning, given a strong enough base, can get traction on hard problems — and it will not prioritize our ability to interpret the result.

He is more uneasy about stacking that loop on Meta's recent "reasoning in continuous space" work, which feeds the last hidden state back in place of a sampled token. The paper reports shorter traces for comparable accuracy and a better grip on problems that want breadth-first search, because a latent state can hold several paths at once. Put reinforcement learning on top of that, Labenz says, and you can imagine something very powerful and quite alien.

The method is not limited to contest math. Wherever an environment can return an objective score, the same centrifuge can spin. Some of those scores will be ugly. If the reward is dollars successfully scammed, and the base model is already good enough to succeed occasionally, enough at-bats could produce a literal superhuman scammer. OpenAI has said its models were never told how to reason; Dario Amodei has said internal results convince him Anthropic is on a path to AGI and superintelligence within a few years. Gemini Flash Thinking is already public. Labenz infers that Anthropic is training the same way even without a launch: the recipe is too simple to be a house secret.

R1 exists because R1-Zero will not talk like a product. DeepSeek's fix is a multi-stage pipeline that puts reasoning first and general helpfulness later, the reverse of work Nathan Lambert's team at the Allen Institute for AI had described: pretrain, diverse supervised fine-tuning, then reinforcement learning. Here a small "cold start" supervised set, designed with human priors, shows the model the reasoning format people want. Then large-scale RL on math and code pours in the actual skill.

Labenz treats the cold-start step as a validation of something he has been telling automation builders: fine-tune on chain-of-thought demonstrations that show the pattern you want, then reinforce. The small supervised set is not there to teach calculus. It is there to keep the later RL from amplifying a wild, language-mixing reasoner into the only personality the model has.

Only after that does the lab broaden the data. It mixes about 600,000 successful reasoning traces, harvested from the model itself, with 200,000 examples of writing, Q&A, and ordinary dialogue left over from the V3 work, and runs another supervised pass. A second reinforcement-learning stage follows. Accuracy rewards remain for problems with right answers; for writing and chit-chat, the paper says, "we do resort to reward models to capture human preferences." That is the first learned judge in the entire report.

In that last stage, helpfulness is scored only on the final summary. The thinking tokens are allowed to wander, reverse, and try dead ends, because that is the point of a scratchpad. Harmlessness is scored on the entire transcript, thinking included. OpenAI's deliberative-alignment writeup for o1 and o3 does the reverse: it puts no reward pressure on the chain of thought, on the theory that punishing bad reasoning will not erase it — it will teach the model to hide it.

Labenz flags a post from the model-behavior observer Janus, who posts as @repligate. After trying R1, Janus said the immediate vibe was that its chains of thought were "substantially steganographic" — speaking in code — then added that this was vibes, not a paper reading. Labenz does not treat it as evidence. He does treat Janus as someone whose feel for models is worth watching, because a model that already switches languages unprompted is a model that could, under safety pressure, learn a dialect we cannot audit.

The ablation list matters as much as the successes. There is no Monte Carlo tree search, no process reward that grades each step, no extra verifier sitting above the token stream. An autoregressive model emitting one token at a time learned to double back and search inside that stream. Labenz's read is that you can start this centrifuge, leave it running, and come back later — which is how he pictures OpenAI getting from o1 to o3 without a zoo of extra machinery.

On the inference provider Hyperbolic, R1 is billed at $2 per million tokens. OpenAI's o1 is $60 per million *output* tokens. That is more than a 30-fold gap. Labenz finds R1 competitive on pure reasoning and useful when he pastes in a paper and asks for a critique, which is how he is using it on the free DeepSeek chat site. For multi-turn work and miscellaneous assistant tasks he still expects o1 to win. If the inner loop is this reinforcement process, the outer loop is how many times you have shaped a product with user feedback, and OpenAI has run that loop longer.

The unsanded edges show up as better writing. One widely shared R1 passage, headed "Value learning, or how to teach cannibals table manners," calls human values "a fugue of contradictions" and the pretraining corpus a "digitized scream pile of history." It is the sort of sentence, Labenz says, that heavily productized assistants have been trained out of writing. Anyone doing creative work should put R1 on the shortlist now. He calls the model a wilder beast, closer to a base model that happens to reason than to a fully sanded helper.

He also wants more people on R1-Zero, not just the productized R1. Hyperbolic claims to host it; in a few of his own tests he did not see the language-switching the paper describes, so he is not completely sure the endpoint is the raw checkpoint. The request is still the same: spend time with the weird version, and if it produces illegible or surprising traces, publish them. The community has almost no public feel yet for what large-scale RL, cooked without a helpfulness layer, actually sounds like.

DeepSeek then treats R1 as a teacher. Successful traces go into supervised fine-tunes of smaller Llama and Qwen models, and the jump from those bases to the distilled reasoners is huge; the larger the student, the better. That is the same distillation loop OpenAI and Anthropic already run, and a main reason small models have gotten so capable so fast. Directly applying the R1-Zero reinforcement recipe to those small students, though, did not work. Distillation did. Why the RL loop refuses to start on the small models is, Labenz says, not answered in the paper.

His best guess is a threshold. He goes back to Microsoft's Tiny Stories work: tiny networks trained on simple prose unlock skills in order, from word frequency to parts of speech to a scrap of negation logic. A small Llama may never encode the reasoning patterns often enough to luck into a reward, so the flywheel never turns. A second guess comes from Mosaic's training lore: late in a run the learning rate is tiny, the weights are stuck in a local minimum, and extra updates barely move. The authors suspect that once a distilled student *has* the patterns, further RL would take; they did not run that experiment.

For policy, the finding cuts deep. U.S. discussions have treated pretraining compute as the yardstick that should trigger reporting and special review. Reasoning gains here use far less extra compute than the original V3 run, and they pour into a Llama 70B student that scores 65% on GPQA Diamond — above GPT-4o's 50%, above o1-mini, about two-thirds of the way to o1's 75%, on questions PhDs in-field reportedly hit around 70%. That is a laptop-scale model. The hosted DeepSeek product will refuse Tiananmen Square; the open R1-Zero Labenz tried on Hyperbolic answered it. Zvi's gloss, which Labenz likes, is that stuffing false beliefs into the weights creates tension, so the sensible split is a coherent model plus a moderation layer on the product, not propaganda in the parameters.

The same day, Moonshot published Kimi k1.5. The model was not yet in the API, and Labenz will not over-weight a paper: Microsoft's synthetic-data work has shown that strong benchmarks and a useful assistant are different jobs. At a high level the method still rhymes. Moonshot describes a simple RL framework and explicitly skips Monte Carlo tree search, process reward models, and even a value function that would tag which tokens actually mattered. There is no Kimi-Zero in the writeup. There is a warm-up supervised set meant to show planning, reflection, correction, evaluation, exploration, error spotting, backtracking, and solution refinement before the RL begins.

Thinking length grows here too. To stop the model from spending a thousand tokens on 1+1, they add a length penalty that wants correct *and* brief when brief is enough. They tried a traditional reward head that emits scores and a chain-of-thought critic that reasons about quality; the critic won, in the same spirit as OpenAI's deliberative alignment, where a model with a policy in context judges whether the policy was followed. They still refuse to micromanage individual steps, because planning and recovery require going down a wrong path and coming back.

Labenz's comparison is the point. You can get o1-level reasoning with or without supervised fine-tuning, with or without chain-of-thought rewards, with or without a length penalty, on more than one RL algorithm. OpenAI, Google, DeepSeek, Moonshot, and almost certainly Anthropic appear to be scaling the same long-thinking loop, and nobody claims it is done. That is, for him, the remaining shared tech tree he had feared chip bans would split. o3 is the exception that still looks secret: spending thousands of dollars in a handful of minutes implies more tokens per second than one autoregressive rollout can emit. Kimi's paper even boasts that inference stays a single language sample, avoiding the parallel planning of tree search. Majority vote, or a consensus trick such as the "Smoothie" embedding-centroid method, might explain some of o3. The high-compute setting, he thinks, probably needs something these papers do not disclose.

When Sam Altman, DeepMind's leadership, and Dario Amodei say the singularity is near, Labenz thinks the joke phase is over. The setup is still simple. Math is already a losing fight for most humans; programming too, with o3 reportedly in the range of the top 200 competitive coders. Transfer is showing up outside pure puzzles: o1 already beats GPT-4o at legal analysis. Knowledge has been superhuman since GPT-3 or GPT-4; weekly superhuman reasoning is here; trained-from-scratch systems already have an intuitive physics of proteins, weather, shipping networks, and other domains almost no biological brain can grok. Google's Titans paper, a follow-on to Mamba and state-space models, is another step toward useful memory that does not grow with every token. The picture of AGI, he says, is assembling in public.

He does not claim to picture godlike superintelligence, and he doubts the lab leaders do either. Weekly superhuman systems on a one-to-three-year clock are another matter. Amodei has talked about AIs better than any human at every task in 2027. Imagining the *world* that creates is still hard. Imagining the *model* that satisfies the definition, Labenz says, is getting easier, which is why he now says the singularity is in fact near. The papers still omit data and last-mile details, and very few organizations can train a 671-billion-parameter V3, so he does not expect a flood of new frontier labs. He does expect a new normal: a PhD-level reasoner you can run on a laptop, permanent from here unless something collapses.

That normal is already pressure on Western business models. He would often pick R1 over o1 inside Cursor if he is paying per token and waiting on a visible trace. Why DeepSeek open-sources the weights is still a mystery to him, as is what Beijing thinks. Maybe the mission language about unraveling AGI with curiosity is sincere; maybe a better closed model is coming. He notes that the base model will discuss Tiananmen while the Chinese product will not, which could mean the state cares more about the public square than about a private weights file. A de-escalatory reading is available too: releasing methods and weights is the opposite of hoarding. A $500 billion Stargate build and newspaper ads about an AI war make it hard to believe anyone is asleep.

His own o1-era hope — that long thinking would make rogue bio or cyber work expensive again, so incumbents could out-compute attackers the way spam filters outrun spammers — looks weaker once a 70B distilled reasoner fits at home. Six million dollars trained V3; the extra RL looks cheaper still, and more parallelizable. Chip denials have not kept Chinese labs off the shared tech tree, and the best open models and the clearest public recipes for reasoning now come from China. Building domestic data centers and fabs still makes sense. Pretending those denials will block military use while starving Chinese small businesses of civilian AI does not. People have noticed that R1 sometimes claims it was trained by OpenAI; Labenz bets that is web contamination and unsanded edges, not stolen o1 traces. The remaining Western lead, he says, is small, and any plan for a good AI future that needs it to be insurmountable is a bad plan. Go to chat.deepseek.com. When Kimi ships, use that too. If you want the work almost no one else is doing, get on R1-Zero. He has not felt this "drop everything and immerse" pull since early GPT-4, two and a half years earlier.