TinyZero: A $30 Reinforcement-Learning Experiment on a Small Arithmetic Task

A Berkeley research project called TinyZero demonstrates a DeepSeek-R1-Zero-style reinforcement-learning effect on a small arithmetic game. The reported cost is under $30 for a particular training run on an already pretrained model; it does not include building a frontier base model, and the project did not clone DeepSeek-R1 or establish equivalent general reasoning.

The DeepSeek paper describes an “aha moment” in an intermediate R1-Zero model: while working on a problem, it reevaluated its initial approach and allocated more output to reasoning. The authors present this as an example of strategies emerging during reinforcement learning rather than being explicitly supplied as a step-by-step demonstration.

My view is that reinforcement learning will become increasingly important after base-model pretraining. A clear reward is easier to define when an answer can be checked, as in many math or coding tasks. Open-ended questions such as which color is best do not have one objectively correct answer and are harder to reward in the same way.

Jiayi Pan, a UC Berkeley PhD student, and collaborators applied this idea to Countdown. They used already pretrained Qwen2.5 models, including a 3B base model, and rewarded the model for producing valid solutions. This is a small-scale reproduction of a behavior studied in R1-Zero, not distillation of the full DeepSeek model.

The project reports that the model developed written search and self-verification behaviors during reinforcement learning. That observation concerns its generated solutions on the tested tasks; it does not show that the model learned general reasoning from scratch. In the DeepSeek work, too, the starting model had already undergone substantial pretraining.

A reward function tells the system how well an attempt meets a goal. In Countdown, the player combines given numbers with basic arithmetic to reach a target, so an answer can be checked automatically. That makes it easier to try many candidate solutions and reward correct ones than in a subjective creative task.

One example asks for an equation using 19, 36, 55 and 7 to reach 65. The displayed response tries and revises intermediate approaches before producing 55 + 36 - 7 - 19 = 65. It is an illustration of one output, not a measurement of reliability on all possible problems.

The recipe starts with a base language model, prompts and a ground-truth reward, then applies reinforcement learning to the game. Pan reports that the initially unhelpful outputs gradually developed revision and search. The experiment logs are available for closer inspection. AWS sponsored the video; its Bedrock promotion is separate from the TinyZero research.

I speculate that small models adapted for narrow jobs might eventually combine task-specific reinforcement learning with additional computation or training at inference time. That is a possibility to explore, not a result shown by this experiment. Test-time compute and changing model weights at inference are distinct ideas, and the video does not demonstrate their combination.

The team compared Qwen2.5 base models of 0.5B, 1.5B, 3B and 7B parameters. Pan reports that the smallest model tended to guess and stop, whereas models from 1.5B upward learned search, verification and revision in this setup. The displayed training curves suggest that the starting model’s capabilities matter for the task.

They also compared base and instruction-tuned variants. The instruction model learned faster and produced more structured, readable responses, while the reported final Countdown performance was roughly similar. That is evidence about these particular runs, not proof that instruction tuning is unnecessary for every application.

Pan says the team tried PPO, GRPO and PRIME, and observed long reasoning traces with each. The researchers had not tuned their hyperparameters and explicitly declined to rank the algorithms quantitatively. The experiment therefore does not establish that the algorithm choice never matters.

The observed strategy also depended on the task. For Countdown the model searched and checked proposed answers; for multiplication, Pan reported a step-by-step strategy based on the distributive rule. I find that variation interesting when considering small systems specialized for particular problems.

The central caveat is scope: Pan said the main result had been validated on Countdown rather than general reasoning, with compute limiting further experiments. A separate Open LLaMA run had a termination problem: the model often continued generating because the implementation did not stop after the answer, so the team did not report those results in the thread.

Asked about the price, Pan identified the run as a 3B model trained with PPO and estimated about ten H100 GPU-hours, corresponding to roughly $30 at the assumed rate. This is an estimate for the additional experiment, not the cost of pretraining Qwen, conducting all research, or reproducing DeepSeek-R1. Actual costs depend on hardware prices and configuration.

In preliminary observations, the 3B model’s output length first fell as it learned the expected format, then its chain-of-thought length increased alongside improved performance. Pan cautioned that minor code mismatches could affect this early result, so it should be interpreted cautiously.

Pan also considered DeepSeek’s published training-cost figures plausible under the stated mixture-of-experts and FP8 assumptions; that is his assessment, not an independent audit. TinyZero’s code, dataset and experiment logs were made available for others to study and reproduce the narrow experiment.