OpenAI o1’s 2024 Launch: Reasoning Benchmarks and Early Tests

OpenAI introduced o1-preview and o1-mini on September 12, 2024, describing a new family trained to spend more computation on difficult problems before responding. I was excited by the reported math, science and coding performance. This discussion reflects the launch, not current product availability; OpenAI has since retired the preview and mini from its API.

OpenAI said the models learned through training to refine approaches and identify mistakes. Its launch materials included results for both the available preview and a further o1 model still in development. Those are different versions, and performance on a particular test should not be treated as a blanket claim of human PhD-level ability.

On 2024 AIME problems, OpenAI reported 13.4% for GPT-4o, 56.7% for o1-preview and 83.3% for the later o1 using consensus among 64 samples. For o1 with a single sample, it reported 74.4%. The video’s 83% figure is therefore not the preview's single-answer score; the company also reported an 89th Codeforces percentile for the later o1, versus 62nd for the preview.

OpenAI compared o1 with PhD experts on GPQA Diamond science questions. The company explicitly cautioned that performance on that benchmark does not make the model more capable than a PhD in every respect. In 2024 the preview lacked some ChatGPT features, including web browsing and file or image uploads, so GPT-4o could be more useful for many ordinary tasks.

OpenAI described safety training that uses reasoning about its rules and reported scores of 22 for GPT-4o and 84 for o1-preview on a particular difficult jailbreak evaluation. These are results on that test, not a guarantee of safety in every situation. The company also described preparedness evaluations, red-teaming and collaboration with safety institutes.

I imagined combining stronger reasoning models with agents and automated research tools. The potential is exciting, but my suggestion that this marks an imminent intelligence explosion is a prediction, not a demonstrated scientific outcome. Research applications still require verification by qualified people.

OpenAI had not disclosed all details of the method. I speculated about connections to earlier rumored projects, but those connections and the exact implementation were not established by the release. A brief biology demonstration showed the model responding after a pause; it was illustrative, not an independent study of scientific accuracy.

A Cognition AI demonstration showed Devin working on a sentiment-analysis task. I wondered whether it used the new model and argued that stronger base models might change the amount of orchestration needed by products like Devin. The clip did not establish that Devin was powered by o1; my competitive analysis was speculation.

OpenAI also announced o1-mini, described at launch as a faster, less expensive reasoning model particularly suited to coding. The company said it was 80% cheaper than o1-preview at that time. Those historical pricing and feature descriptions are not current product advice.

OpenAI demonstrated a browser Snake game created from a prompt. I noted that earlier models could also produce similar games, so the example was interesting but not a decisive comparison. My speculation that o1 might be the same as another rumored model was not confirmed by the announcement.

The technical report describes reinforcement learning applied to a model's internal reasoning process and reports gains with more training and test-time compute under the evaluated conditions. It does not mean that additional thinking always improves every answer, or that the model is literally a human scientist.

The report's graphs compare AIME, Codeforces and GPQA results. Its AIME bars mix single-sample accuracy with 64-sample consensus, while its 89th percentile coding score applies to the later o1 model rather than the preview. These are OpenAI-reported benchmark results, not independent proof of broad competence.

OpenAI provided selected examples including a cipher, a shell-script transformation and math problems. In the displayed comparison, GPT-4o struggled with the cipher while o1-preview produced the intended answer; the examples are demonstrations, not a representative failure-rate study.

A human-preference evaluation in OpenAI's report favored o1-preview in some reasoning-heavy categories, while it was not preferred for some natural-language work such as personal writing. The result depends on the prompts and raters and shows that a specialized improvement need not make one model best for every use.

OpenAI reported safer completions on a challenging set of jailbreaks and edge cases for o1-preview than GPT-4o. The company also said it would not expose raw chains of thought to users, instead showing model-generated summaries. A summary is not the complete internal process, and the report's proposed monitoring assumes that an internal trace is faithful and interpretable.

Greg Brockman compared ordinary model responses and extended reasoning to different styles of thinking. This is an analogy, not evidence that a model thinks like a person. Training and inference details remain partly undisclosed, and longer reasoning is not a guarantee of correctness.

I asked o1-preview to write a Python Tetris game. After a long wait the first program failed with an attribute error. I supplied the error and received corrected code; that second version ran and cleared a line, but the interface had an odd embedded game and no score.

I was impressed that the repair made the game playable despite its flaws. The video description later notes that I had mistyped the Tetris prompt, which may explain the unusual nested game. This is one interactive test, not a controlled measure of coding reliability.

Finally, I asked the model to count the words in its own response. It returned a seven-word sentence that I counted as correct. I wanted to test the model further; one successful answer and one repaired program do not establish general reliability or an intelligence explosion.