How Researchers Think o1-Style Reasoning Models Work

A paper from Fudan University and the Shanghai AI Laboratory surveys how reasoning models like OpenAI's o1 might be built. The authors propose a framework for reproducing aspects of test-time computation; their account is an informed hypothesis, not confirmation of OpenAI's proprietary methods.

The idea behind a thinking model is to spend more computation at inference: after receiving a prompt, it generates a longer reasoning process before answering. I see this as a way to improve performance on difficult math, science, logic and coding tasks. The paper discusses o1; it was written before o3 was announced.

The paper places o1 in OpenAI's five-stage AGI framework, at the 'reasoner' stage. The other stages are chatbots, agents that act, innovators that assist invention, and organizations run by AI. I think progress toward agents is already visible, although the later stages remain speculative.

The o1 system card reports improvement with increased reinforcement-learning and inference computation. For me, that suggests a second axis of scaling beyond training alone: spend more compute on a problem when the model is answering. I consider this one of the major AI developments of 2024.

The researchers organize their discussion around four components: policy initialization, reward design, search, and learning. Initialization establishes the model's starting capabilities; reward design supplies feedback; search explores possible answers; and learning changes behavior based on experience.

Initialization includes pretraining, instruction fine-tuning and human-like reasoning behaviors. Examples include clarifying goals, breaking down tasks, proposing alternatives, generating solutions, evaluating them and correcting mistakes. These provide ingredients for the later search and learning stages.

A reward tells a system what to pursue. In chess, a simple signal could distinguish a win from a loss; language tasks are harder because correctness and quality may not be obvious. Designing feedback for these problems is consequently a central challenge.

Search can occur both during training and during inference. A model might generate possible solutions, assess them and iterate. Reinforcement learning uses feedback from interactions with an environment; automated feedback can reduce dependence on expensive human judgments, although the quality of that feedback matters.

In reinforcement learning, a policy specifies how an agent chooses actions in a given state. For language models, actions can be viewed at three levels: an entire solution, individual reasoning steps, or the tokens that make up those steps.

The authors describe six reasoning behaviors that can be prompted or learned from expert trajectories. The first is problem analysis: restate the question, identify implicit constraints and turn abstract requirements into actionable specifications. When prompting, you can ask a model to identify missing information before it proceeds.

Task decomposition divides a complex problem into manageable subtasks, like following the steps in furniture assembly. In a coding example, the model might separate reading an input string, removing spaces and parsing it. Task completion then works through those clarified subtasks.

When a path fails, alternative proposal generates other possible approaches. The paper uses cipher solving as an example; varying sampling settings can produce a more diverse set of candidates. Step-by-step reasoning can make a complex task easier to work through, though it is not a guarantee of correctness.

Self-evaluation asks the model to inspect a proposed solution. I often find evaluation useful alongside generation, including in multi-agent systems where one agent produces an answer and another critiques it. Self-correction closes the loop by proposing a fix and testing again.

The researchers speculate that a reasoning model needs enough context capacity to generate long, diverse reasoning traces. They also discuss evidence that exposure to code and structured logical data can improve reasoning, along with training for self-reflection. These are proposed ingredients, not a disclosure of the o1 training recipe.

Reward models can assess an outcome or a process. An outcome reward checks the final answer; a process reward assesses intermediate steps. If the first several steps are correct but a later one fails, process feedback can identify where to revise rather than treating the entire attempt as equally wrong.

Feedback may come from a real environment. For code, running a program against tests can tell the model whether its output meets an expected result. But real feedback can be expensive or unavailable at test time, so a learned reward model may be used to estimate it.

Another option is an AI judge: one model evaluates candidate answers generated by another. The authors speculate that a system handling many domains may combine reward methods, with process supervision particularly useful for long mathematical and coding reasoning. This remains their hypothesis.

The paper also raises the question of world models. Researchers disagree about whether language alone is sufficient for modeling the world; others explore video or representations that predict future states. I mention Yann LeCun's work and Fei-Fei Li's World Labs as examples of different approaches.

Search explores multiple attempts or possible reasoning paths, guided by rewards or heuristics. The paper argues that it can improve results at both training and inference time. One implication is that a smaller model with an effective search procedure may sometimes outperform a larger model without one.

Self-consistency samples multiple solutions and selects a consistent answer, for example by voting. Self-evaluation instead asks a model to assess its own output. Both depend on the quality of the selection signal; the paper calls the idea that evaluation is easier than generation an assumption.

Search can operate at the level of complete solutions, intermediate steps or individual tokens. At the solution level, candidates are assessed as whole answers. At the step level, the search compares possible next steps; token-level search makes still finer decisions.

The paper distinguishes tree search from sequential revision. Tree search explores branches of possible solutions, whereas sequential revision repeatedly refines an existing attempt. The choice involves a tradeoff between breadth of exploration and the efficiency of iterative improvement.

Best-of-N sampling generates several candidates and selects one using a reward model. Beam search expands and prunes partial sequences. Monte Carlo tree search uses simulated exploration to estimate the value of possible actions; additional simulations can improve its estimates, at a computational cost.

Sequential revision takes an answer, reflects on it, improves it and repeats. The paper speculates that training-time search might favor tree-based approaches with external guidance, while inference-time search might use sequential revision and internal guidance. The authors do not claim access to OpenAI's implementation.

Reinforcement learning can generate experience through interaction, reducing reliance on scarce human expert examples. I use Go as an illustration of how repeated play can find strong strategies. AlphaGo's famous move 37 surprised human observers; its history also included human game data, so self-play should not be conflated with a complete absence of human examples.

The paper surveys open implementations and research projects inspired by o1-style reasoning. I also mention DeepSeek R1 and QwQ as examples of work toward more accessible reasoning models. Availability of methods and projects gives researchers ways to experiment, even if their performance differs from proprietary systems.

The open questions include adapting reasoning to domains without easily verified answers, adding multiple modalities, and learning or searching within world models. These problems connect powerful inference-time reasoning to messy real-world environments, where feedback may be incomplete or ambiguous.