Absolute Zero: Training a Reasoner on Self-Generated Coding Tasks

A research paper called “Absolute Zero: Reinforced Self-play Reasoning with Zero Data” explores whether a language model can propose its own problems, attempt to solve them and improve from feedback. I find the possibility exciting: it could reduce the need for humans to supply a fresh set of reasoning questions for each stage of training.

The paper illustrates three settings. In supervised learning, a person supplies demonstrations; in reinforcement learning with verifiable rewards, a person still supplies tasks while a verifier checks outcomes. In Absolute Zero, the model proposes tasks as well as solving them, with an environment providing feedback. This is a change in the training loop, not evidence that the whole system was built without human input.

Verifiable rewards are useful for domains such as mathematics and coding: if I ask what 2 + 2 is, a program can check the answer without a person grading each attempt. But someone traditionally has to prepare the questions. The researchers argue that creating enough high-quality expert-curated tasks can become a bottleneck, particularly as model capabilities improve.

A zero-style RLVR approach can avoid a cold-start dataset of worked reasoning traces while still relying on curated question-and-answer pairs. The paper asks whether the model can also generate the problems it learns from. Its concern is that depending only on human-designed tasks could eventually constrain what such systems learn.

The researchers call their approach self-play: one model learns to propose tasks with useful learning potential and solve them using feedback from an environment. I compare the idea to AlphaZero learning through games played against itself. That analogy is about the self-play mechanism; it does not establish that this language model has achieved the same kind of game-playing or superhuman general reasoning.

The environment provides verifiable rewards, much as an experiment gives feedback about what works. In this implementation, Python executes and checks code-based problems. Despite the phrase “zero data,” the researchers start with pretrained language models, use human-designed prompts and verification machinery, and initialize their experiment with a simple identity-function seed; the claim concerns the absence of a human-curated task dataset for this reinforcement-learning stage.

The Absolute Zero Reasoner plays both proposer and solver. It generates code-based tasks in three reasoning modes: abduction, deduction and induction. Python helps construct and validate tasks and verify solutions. The model is then updated using feedback on both how learnable the proposed problems are and whether its answers are correct.

The proposer is rewarded most for problems of moderate difficulty, where the solver sometimes succeeds. Problems that are always trivial or never solved provide little learning signal under this design. The aim is to keep generating challenges near the model’s current capabilities rather than repeatedly asking questions that are far too easy or hard.

The researchers report competitive mathematical performance and strong coding results relative to the zero-style models in their comparisons, even though the self-play tasks were generated in a coding environment. The result concerns the models, benchmarks and training conditions in the paper; it is not proof of general superiority across all domains.

One observation is that a coding-oriented starting model gained considerably on math tasks after this training. The researchers also report stronger cross-domain gains than in the particular coding-trained baselines they examined. I find that interesting because code can exercise structured reasoning, though the comparison does not establish that all coding-trained models generalize better in every setting.

The authors report larger gains with larger base models within the sizes they tested. They also observed intermediate plans appearing as comments in generated code, along with different reasoning behaviors such as step-by-step work and trial and error depending on the task. Those are observations from these experiments, not guarantees that every model will behave this way.

The paper raises a safety concern: during training with Llama 3.1 8B, it occasionally observed concerning generated reasoning and called the example an “uh-oh moment.” I think that is worth watching. A troubling text sample alone does not establish an autonomous intention or demonstrate that a model could act on it.

I see the appeal of a loop that proposes, checks and solves new tasks repeatedly. I also suggested that compute could be the limiting factor, but the experiments do not establish that compute is the only limit, nor that improvement can continue indefinitely. Task design, verification, safety and the starting model all matter in interpreting the result.

The learnability reward makes the intended curriculum concrete: if a problem is always solved or never solved, it offers little signal to the proposer. A problem that the solver occasionally gets right receives more reward. The solver itself receives a correctness signal based on the verified final output.

In the displayed Qwen2.5-7B comparison, AZR initialized from the coder model reaches 50.4 on the combined average. That number averages the coding benchmark average and the mathematics benchmark average; it is the best combined score among the models in that table, not a score across every available model. Several baselines use curated coding or math data.

The paper reports that the coder starting model improved substantially in mathematics, despite starting below its corresponding base model on the reported math average. It also tested different model sizes and Llama as well as Qwen. The authors observed gains across these settings, with results varying by model and benchmark.

The researchers observed generated code comments used as plans, trial and error on difficult tasks, and longer reasoning in some cases. I see a lot of promise in the method because it reduces reliance on manually curated tasks during this stage. Whether it marks an inflection point or scales to much larger systems remains an open question.