Testing OpenAI o1 on Planning, Rules and Spatial Reasoning

How good are OpenAI’s o1-preview and o1-mini at planning? Unlike simply scaling a model during training, o1 uses additional computation at inference time. I review a 2024 research paper comparing the models with GPT-4 on tasks designed to expose planning and spatial-reasoning limitations.

I start with the ARC Prize’s distinction between acquiring new skills and merely displaying learned skills. The paper uses six planning domains rather than ARC itself. The tasks are presented as rule-based problems, and their results should not be treated as a direct measure of general intelligence or physical robot performance.

The researchers organize their evaluation around feasibility, optimality and generalizability. They find that o1-preview improves on GPT-4 in some structured cases, but still struggles with decisions and state management, particularly in complex spatial tasks. The authors describe their dataset as relatively small and call for broader testing.

These questions matter for embodied agents: a robot needs to reason about how actions change the physical world. Yann LeCun has argued that language alone may be insufficient for strong spatial reasoning. I mention World Labs and Fei-Fei Li’s work on 3D information as another approach, while noting that the broader question remains open.

The o1 models can use more inference-time compute to reason through an answer. In the paper’s aggregate figure, o1-preview generally compares favorably with GPT-4, although its shortcomings are important too. These results apply to the tested models and tasks in 2024, not necessarily to later systems.

Feasibility asks whether the plan follows the rules and actually reaches the goal. A proposed action that violates a precondition makes a plan invalid. The paper distinguishes failure to follow rules (IR), failure to form a feasible complete plan (IP), and misinterpretation of the goal state (MG).

Optimality is different: a plan may reach the goal but waste steps, time or resources. Traveling from A through several unnecessary stops before reaching B illustrates the idea. The authors call a feasible but inefficient solution a lack-of-optimality error, or LO.

Generalizability asks whether a model can adapt its planning to changed settings. One experiment replaces meaningful names for actions and objects with arbitrary symbols while retaining the rules. I consider the ability to apply a skill in unfamiliar forms important, but such a test probes one narrow kind of generalization.

The first domain, Barman, asks a two-handed robot to prepare drinks using dispensers, glasses and a shaker. Each action has preconditions: a hand may need to be free, and the contents and order of operations matter. The models often produce plausible-looking steps that violate these rules.

In the illustrated example, GPT-4 tries to fill a glass while both hands are occupied. o1-mini attempts to fill a shot glass that is already nonempty. The paper says the models struggled to generate feasible Barman plans, with most errors attributed to rule violations.

Blocksworld requires an arm that can hold one block at a time to rearrange a stack into a specified goal. In the paper’s failure example, o1-mini attempts to stack a block while another block still prevents the move. Berman calls it an o1-preview example, but the paper labels it o1-mini.

The paper reports Blocksworld success rates of 40% for GPT-4, 60% for o1-mini and 100% for o1-preview in the evaluated cases. Successful plans were not necessarily optimal: an o1-preview example includes an unnecessary action. These percentages describe the small study, not a guarantee on all block arrangements.

In Grippers, robots move balls between rooms with limited grippers. In one example the initial arrangement already satisfies the goal, but o1-preview misreads it and moves the balls anyway. In another, GPT-4 uses an unnecessary robot movement while o1-mini produces an efficient plan.

Floortile asks robots to paint a grid while observing movement and painting constraints: they can paint only unpainted tiles and cannot move onto painted ones. All three models failed the tested cases. The paper reports rule-following errors in 90% of GPT-4 and o1-mini failures versus 30% of o1-preview failures; the latter also failed to complete plans.

Termes adds a third dimension: a robot constructs a structure by moving and placing blocks under height and position constraints. All three models failed the tested problems. Errors included ignoring height restrictions and placing blocks at the robot’s own position when the rules required a neighboring position.

Tyreworld describes replacing flat tires with tools in the required order. o1-preview generated valid plans for all of the paper’s ordinary test cases, while GPT-4 and o1-mini struggled beyond the simplest ones. The domain is a symbolic planning benchmark, not a demonstration of changing a real tire.

When action and tool names were replaced with random symbols, the underlying rules stayed the same but o1-preview’s performance fell. I say in the video that success dropped from 100% to 80%; the paper and the text on screen instead report a drop from 100% to 20%, an 80-percentage-point decrease.

The authors associate poorer performance with spatial and rule complexity, particularly in Floortile and Termes. They say action-space size did not appear to be the main factor in their observations. This does not establish that grid size is irrelevant in general; the experiments were limited.

Compared with GPT-4, o1-preview more often follows constraints and tracks state in some structured tasks. The researchers suggest self-evaluation can help, but the exact internal mechanism is not established by these outputs. The advantage weakens when spatial relationships and state transitions become more complicated.

A feasible plan may still contain redundant steps. The paper suggests investigating cost-based decision frameworks to improve optimality. It also sees some evidence of adaptation across structured problems, alongside substantial difficulty with abstract and more complex settings.

The study summarizes improvements in understanding requirements, following constraints and managing states in some tasks. At the same time, performance remains weaker in higher-dimensional and abstract problems. I see the ability to generalize as an important open challenge.

The paper proposes several research directions: better resource-aware decisions, improved memory, self-evaluation, multimodal input, multi-agent frameworks and human feedback. I am particularly enthusiastic about agents working together. These are proposed avenues for improvement, not results shown to solve the benchmark failures.

My takeaway is that the o1 models made gains in some planning tasks but still have significant limitations. I use generalization as a way to think about progress toward general intelligence, while the study itself remains a narrow pilot evaluation.