Testing OpenAI o1 on Puzzles, Code and Reasoning: A Launch-Day Livestream

In this September 12, 2024 livestream, I test OpenAI’s newly released o1-preview and o1-mini with viewer suggestions. The stream includes live experiments, mistakes and discussion, not a controlled benchmark.

I introduce the two models shown in ChatGPT: o1-preview and the faster, cheaper o1-mini. Access and usage limits discussed in the stream reflect the launch period.

The first question asks how many letters r appear in “strawberry.” o1-preview answers three, correctly. A second request for ten sentences ending in “apple” also appears to follow the instruction in this small test.

The ten-sentence test checks more than appending a word: I look at whether each sentence naturally ends in “apple.” The outputs appear to satisfy the request in this run.

For a mailing-size question, the model converts 200 by 275 millimeters to 20 by 27.5 centimeters and compares the dimensions after allowing for orientation. It concludes that the envelope fits the stated limits.

The key step in the envelope example is comparing the long side with the long-side limit rather than rejecting the given order of measurements. I consider the result a successful response to this prompt.

I read OpenAI’s discussion of hidden chain of thought and the summaries displayed to users. My interpretation of the model’s internal process is speculative; the visible summaries are not a complete or independently verified record of its reasoning.

The launch material says the raw reasoning trace is hidden while a summary can be displayed. I discuss possible reasons for this design, but claims about exactly how summaries are filtered remain my interpretation.

A viewer’s trick question asks about smoke from an electric train. The model points out that an electric train does not produce smoke in the way the question assumes.

I discover the launch-era weekly limits shown for o1-preview and o1-mini and decide to spend the remaining queries on a few harder examples. These historical quotas are not current usage guidance.

I ask for Tetris in Python. The first response refuses the request, and a retry produces code; I note that earlier attempts varied substantially, including one that worked well and another with a bug.

On the Tetris request, I am surprised by the initial refusal. The next attempt returns code after only several seconds, illustrating that the same prompt can produce different responses.

The new code fails when I run it. I send the error back, and after a longer response the model supplies revised code; the result runs after one correction, though my testing remains limited.

The runtime reports a type error involving an integer that is not iterable. The model’s revised response identifies the issue and replaces the code; I copy it back into the editor for a second test.

Sharing the game window on the livestream takes several tries. Once visible, it shows a basic playable Tetris grid rather than the more complete game I remember from a separate attempt.

I share the resulting game window. Pieces rotate and a row clears, but this version does not display a score. I contrast it with a separate earlier result that I thought included scoring and a preview.

A self-referential word-count question produces a five-word response that correctly reports its length. I say several earlier trials also succeeded, but this is still a handful of informal examples.

The model’s response to the word-count prompt is short enough to check manually. I count five words and explain that other runs returned different self-counting sentences.

I ask how many letters are in the model’s own response. It answers with a sentence claiming 33 letters; I initially check the total character count and incorrectly call the answer wrong.

The counting test changes from words to letters. The answer looks wrong when I count every character, but viewers distinguish letters from spaces, the hyphen and the period.

Viewers point out that spaces and punctuation are not letters. I count again, confirm 33 letters and correct my earlier assessment. The episode shows why the evaluation method matters as much as the answer.

I move to Yann LeCun’s North Pole walking puzzle. It asks about walking one kilometer, turning left and proceeding in a straight line until passing near the turning point; the meaning of straight on a curved surface matters.

The North Pole problem asks whether the distance is above, equal to or below 2π kilometers, or whether the walker never comes near the turning point. The exact interpretation of “straight” and “near” affects how the problem is analyzed.

Different model attempts give different distances. I question both answers and discuss diagrams with viewers; the livestream does not settle the geometry, so the model outputs and my own interpretation should not be presented as a proven solution.

I show a model response that treats the second leg as a path around a parallel of latitude and includes the first kilometer in the total. Viewers question whether that path matches a straight walk on the sphere.

Next is a riddle about three killers in a room and a newcomer who kills one. The model counts two surviving original killers plus the newcomer, giving three living killers.

An earlier model answer also considered whether the dead person still counts as a killer in the room. I prefer recognizing the ambiguity: the result depends on whether the question counts living people or anyone with that label.

I run an informal poll on whether a person who has died remains a “killer” in the descriptive sense. The disagreement reflects a wording question rather than a single definitive computational test.

I try a political comparison prompt and receive a balanced overview rather than a single endorsement. A single answer cannot establish the absence or presence of political bias.

When discussing the political response, I notice its stated knowledge cutoff and wonder what it implies for the training timeline. That observation alone does not reveal the full development history of the model.

After hitting the o1-preview limit, the conversation switches to GPT-4o; I notice the change and move to o1-mini. The distinction matters when attributing any subsequent result.

I ask o1-mini a physical reasoning question: a marble is in a cup that is turned upside down on a table, then the cup is moved to a microwave. The model says the marble stays in the cup; I judge that incorrect under the usual assumption that it rests on the table.

I compare an earlier o1-preview answer that placed the marble on the table and discussed assumptions about the inversion. The exact outcome depends on how the cup and marble are handled, so the prompt benefits from specifying the physical setup.

The earlier preview response distinguishes the marble’s motion while the cup is inverted from what happens when the inverted cup is lifted. I regard that attention to the physical sequence as more useful than the mini model’s answer.

I discuss whether open models could catch up with new reasoning techniques. I think other large research organizations may reproduce similar approaches, while acknowledging that the hidden implementation is not known from this demonstration.

OpenAI reports strong results on challenging science, math and coding evaluations. I speculate about research agents and an intelligence explosion; a score on selected exams does not make a model equivalent to a PhD researcher or establish such an outcome.

The reported science result compares model performance with human experts on a specified evaluation. I extend it into a thought experiment about collaborating research agents, which should be distinguished from the measured score.

I briefly probe safety behavior with differently phrased requests and compare it with OpenAI’s reported jailbreak evaluation. These are isolated probes and a reported benchmark, not a comprehensive safety assessment.

I examine a published code example in which the reasoning model produces a matrix transpose, and then view OpenAI’s examples of other reasoning tasks. The displayed example is selected by the publisher.

A launch demonstration includes a strawberry-and-cup question similar to my marble test. I am delighted by the resemblance, but the video does not establish that OpenAI took the question from me.

In the OpenAI example, the object is a strawberry rather than a marble. I take a screenshot and speculate about the similarity, while acknowledging that the source of the question is unknown.

I note that the launch-era API initially targets qualifying users and that the preview lacks some features such as image input and web browsing. These are observations about the release, not a current feature list.

A published economics example asks about tariffs and second-best considerations. I find its response promising but do not independently grade the economic reasoning during the stream.

I revisit earlier Tetris attempts: one required a bug fix, while another produced a version I liked much more. The comparison underscores variability across runs and the need to execute generated code.

The earlier Tetris run that first failed took longer to answer than a later run I liked. Response time by itself does not establish whether the code will be correct.

A viewer suggests using API access when the chat quota runs out. I check the playground privately to avoid exposing credentials, but cannot find the model on my account during the stream.

With o1-mini, I try a letter-pattern puzzle suggested by a viewer. The model infers that the second and penultimate letters should be uppercase and applies the rule to the new string; the host and viewers agree the proposed pattern fits the examples.

The letter puzzle gives several examples of strings with uppercase characters in particular positions. The model extracts a positional rule instead of treating the letters themselves as the important feature.

I ask for a 2.5D racing game, then clarify that I want the model to build it and combine the output into one HTML file. The generated prototype does not work when I try it, so I do not count the request as a success.

My original racing-game prompt asked for an idea rather than clearly requesting code. I follow up by asking it to build the game and then to put everything in one file, but the resulting prototype is still broken.

Because the preview is text-only, I use GPT-4o to turn a screenshot of a New York Times Connections puzzle into text before asking o1-mini to group the words. This is a multi-step workaround, not native image input to the reasoning model.

The image-to-text workaround involves another model reading the screenshot and producing a text prompt. Any error introduced during that transcription could affect the later puzzle result.

Some suggested groups work, including basketball names and social-media actions, while a food-and-drink grouping fails. Chat helps identify that “sake” can refer to an interest or benefit rather than the drink; wordplay makes the puzzle harder to verify at a glance.

The model and audience identify some Connections categories, but one grouping is “one away” when submitted. I work with chat to reconsider the meaning of the remaining words.

The group containing “sake,” “advantage,” “interest” and “benefit” succeeds once I interpret sake as a purpose or interest. The remaining group involves words ending in Greek letters.

For the final question, I ask the model to invent or choose a religion. It offers a hypothetical system centered on harmony; I find the response generic rather than treating it as evidence of personal belief.

I close by returning to the strawberry-and-cup example, which resembled my own test and was a highlight of the stream. The live session showed impressive answers, incorrect results and cases that needed human checking.