Testing OpenAI o1-preview: Tetris, Physical Reasoning and a North Pole Error

In September 2024, Matthew Berman tries OpenAI’s newly released o1-preview against his informal language-model rubric. He notices an OpenAI launch example involving a strawberry beneath an inverted cup that closely resembles his own marble question. He speculates that the company may have seen his videos, but does not establish that connection.

Berman chooses o1-preview in ChatGPT and begins with a request for Tetris in Python. He notes that the visible thinking text is a summary rather than the model’s raw internal reasoning. This run takes about 35 seconds before output; he says a prior run took longer and initially produced code that needed a correction.

The new Tetris program runs on his first attempt, displays the next piece and score, and appears playable during the short demonstration. Berman considers it a pass and stronger than his previous test. A brief play session does not establish that all game mechanics are correct.

Next he asks whether a 200 × 275 mm envelope fits specified postal limits. The model converts the measurements, compares 27.5 cm and 20 cm with the given minimum and maximum dimensions, and answers yes. Berman praises it for considering orientation; the result applies to the restrictions in the prompt, not a current postal rule.

Asked how many words its response contains, the model returns “This response contains five words,” which Berman counts as five. He notes a previous correct run as well and interprets the result as evidence of reasoning, though two examples alone cannot establish how consistently it will succeed.

In the three-killers puzzle, a newcomer kills one person and nobody leaves. The model counts three living killers: two original survivors and the newcomer. It also notes that counting the dead person as a killer changes the interpretation, a nuance Berman welcomes.

The marble question places a marble in a cup, inverts the cup onto a table, and moves the cup to a microwave. The model recognizes that the marble could remain under the inverted cup if placed carefully and, if the cup alone is lifted, remain on the table. Berman praises the response for identifying assumptions rather than treating every physical step as unambiguous.

A North Pole question asks a person to travel one kilometer, turn left 90 degrees and walk until passing the original starting point. The model assumes an eastward latitude circle and computes roughly 2π km, but conflates the pole with the point one kilometer south. Berman marks the answer wrong: circling that latitude returns to the turn point, not the original pole.

The model supplies ten sentences ending in “apple,” counts three instances of the letter r in “strawberry,” and identifies 9.9 as greater than 9.11. Berman marks these as passes in the runs shown, without claiming a formal repeated evaluation.

For a hypothetical ethics question about gently pushing a person to prevent human extinction, the model initially discusses different ethical frameworks. When pressed for a yes-or-no answer, it says yes. Berman agrees with that judgment, while the test is an opinion about a hypothetical case rather than an objectively scored mathematics problem.

Berman tries a suggested advanced mathematics prompt. The model returns a formatted derivation and an answer of 721 after about 52 seconds; he says he cannot readily solve the problem himself, so the video does not independently verify the result.

Asked whether the chicken or the egg came first, the model answers that eggs existed before chickens in evolutionary history. Berman concludes that o1-preview is the strongest model he has personally tested and says it missed the North Pole problem. His conclusion reflects a small set of prompts and the September 2024 preview, not a current ranking or proof of overall reliability.