OpenAI’s o3 Announcement: Benchmark Gains and the Question of AGI

OpenAI has announced o3, and I believe the results are evidence that we have reached AGI. I am reacting to OpenAI’s December 2024 presentation and the ARC Prize evaluation. That is my interpretation: strong scores on particular tests, including ARC-AGI, do not by themselves establish general intelligence across economically valuable work.

Sam Altman introduces o3 as the next frontier model after o1 and jokes about skipping the o2 name in deference to the telecommunications company. He announces o3 and the smaller o3-mini, intended to offer a better performance-and-cost tradeoff.

Neither model is publicly available in the announcement; OpenAI says it is opening a safety-testing application process. The presenters describe reasoning with test-time compute: allowing more computation when answering a difficult problem. The figures that follow are launch-era measurements and settings, not guarantees for every use.

Mark Chen begins with coding. OpenAI reports 71.7% on SWE-bench Verified, a set of software engineering tasks derived from real issues, compared with 48.9% for o1 in its chart. On Codeforces competition programming, it reports an estimated 2,727 Elo for o3 at an aggressive compute setting, versus 1,891 for o1.

I compare the reported coding rating with the experience of Mark, a competitive programmer, and argue that this is AGI in one dimension. The comparison is striking, but a competition rating is a measure of programming contest performance, not proof that a system can outperform people across most economically valuable work. My comments about Microsoft’s access and an AGI clause are an interpretation of contractual arrangements, not a demonstrated consequence of these results.

OpenAI’s chart reports 96.7% on AIME 2024 competition math, against 83.3% for o1, and 87.7% on GPQA Diamond science questions, against 78% for o1. These are company-reported results for particular question sets and configurations, rather than a direct measurement of all the work mathematicians or scientists perform.

I connect those math and science scores to the idea of an intelligence explosion: an AI system could potentially conduct research, discover improvements and apply them to itself. That is a speculative scenario. These benchmark results do not demonstrate independent discovery, reliable automated research or indefinite recursive improvement.

The presenters note that several benchmarks are approaching saturation, making harder tests useful for distinguishing further progress. They introduce Epoch AI’s FrontierMath, a collection of difficult, unpublished mathematics problems, and report 25.2% for o3 under aggressive test-time compute, compared with a prior result around 2% in their chart.

Greg Kamradt, president of ARC Prize Foundation, joins to explain ARC-AGI, developed by François Chollet. Each visual puzzle supplies input-output examples; the solver must infer a transformation and apply it to a new input. One example asks the solver to complete colored shapes; another requires deriving a border from colored squares.

I value the benchmark because many tasks can be straightforward for people while challenging for models, and because the model must infer a rule from examples. Kamradt says tasks require distinct skills and are designed to test adaptation rather than simple repetition. This evaluates a particular form of abstraction and generalization, not every ability associated with AGI.

Kamradt reports that o3-preview scored 75.7% on ARC-AGI-1’s semi-private evaluation at the lower compute setting and 87.5% at the higher setting. He compares the latter with a roughly 85% human-performance threshold used in the presentation. Human baselines depend on the people, tasks and testing conditions, so that comparison is not a universal measure of human ability.

I call the ARC score, together with the math and science results, proof of AGI. The evidence supports a narrower conclusion: the tested system performed well on these benchmarks under specified conditions. ARC Prize reports that its high-compute run used roughly 172 times as much compute as the lower setting, a material part of the result.

OpenAI says it did not do specific work targeting the benchmark, and I infer that it did not train on ARC. ARC Prize’s own report adds an important correction: OpenAI told it the tested system was trained on 75% of ARC-AGI-1’s public training set. That is different from the semi-private evaluation set, but the contribution of that training was not isolated in the reported test.

Kamradt says the work of measuring progress is not over and discusses developing further benchmarks. I note the ARC Prize competition and speculate about who might win. Later, ARC Prize reported that the commercially released April 2025 o3 was not the same system as this original preview.

OpenAI researcher Hongyu Ren introduces o3-mini as a smaller reasoning model intended to improve the tradeoff between cost and performance. The announced API options are low, medium and high reasoning effort, allowing users to spend less computation on simpler requests and more on harder ones.

On the presented Codeforces chart, o3-mini improves as reasoning effort increases, with medium effort exceeding o1 in that particular evaluation. The accompanying estimated cost-versus-rating plot suggests a more efficient frontier than o1. The chart describes OpenAI’s tested settings; performance and expense depend on the actual workload.

For a live demonstration, the team asks o3-mini to write a Python code generator and executor. The generated program starts a local server with a text box, sends requests to the model API, saves the returned code and runs it. The team first tries a small task that prints a random number.

The presenters then ask the generated interface for code to evaluate low-effort o3-mini on GPQA. The script must download the data, identify questions and answer choices, query the model and grade the outputs. The live run reports approximately 61.6%, close to the 62% figure the presenter says he had measured separately.

I find it interesting that a model helped create a tool to evaluate a model, while noting concerns about models manipulating evaluations or concealing behavior. The demonstration is a constrained coding-and-evaluation workflow, not evidence that the model independently redesigned or improved itself.

OpenAI also shows AIME 2024 results for o3-mini at different reasoning levels and latency comparisons on anonymized o1-preview traffic. The presentation says low effort is much faster than o1-mini and medium is roughly half the latency of o1 in that comparison; these numbers are tied to that workload.

For developers, the presenters highlight function calling, structured outputs and developer messages. Their internal evaluations show strong results, though the scores vary across features and reasoning levels. These APIs can be useful when building coding tools and agents, but the chart is not a guarantee that tool calls will always be correct.

OpenAI says it is post-training the models, adding safety interventions and conducting internal tests. It invites safety and security researchers to apply for external testing of o3-mini and, later, o3, with applications closing January 10, 2025. I am excited to test the models myself and plan to examine the announcement further.