Reactions to OpenAI’s 2024 o3 Preview: Benchmarks, Cost and Limits

OpenAI's December 2024 preview of o3 produced striking benchmark results and a range of reactions. I gathered responses from researchers, entrepreneurs and commentators, from enthusiasm about mathematics to warnings about cost and reliability. These reactions concern the preview evaluated at the time; ARC Prize later said the publicly released April 2025 o3 was not the same system it had tested.

Balaji Srinivasan focused on FrontierMath, a collection of challenging mathematical problems. OpenAI claimed a 25.2% score after earlier models had scored much lower. This was a reported benchmark result, not evidence that o3 could solve 25% of all research mathematics or outperform every mathematician.

The sample problem shown involves number theory, and FrontierMath's creators say many tasks require hours or days of specialist work. I found the result remarkable. Individual problems vary in difficulty, and the video does not establish how many problems any particular human expert could solve.

Srinivasan also pointed to ARC and competitive programming. The figures show performance under specific benchmark and testing conditions, not a universal rank among all human coders or a direct head-to-head contest. The value of these comparisons depends on the task, rules, tools and evaluation method.

Mathematicians Terence Tao and Timothy Gowers had described reviewed FrontierMath problems as extremely difficult. Those comments concern the problems they examined; they do not establish that no mathematician could solve 25% of the full benchmark. The video’s broader comparison is speculation.

Ethan Mollick likened more inference-time computation to the fictional supercomputer in Douglas Adams's work: difficult answers may require more time and the right question. His analogy highlights both potential and cost, not proof that more computation always yields a correct answer.

The video includes a sponsored segment for Vultr, a cloud-computing provider. The claims about availability, performance and pricing in that segment are sponsor claims.

ARC creator François Chollet reported 75.7% on the ARC-AGI-1 semi-private evaluation in a lower-compute configuration and 87.5% with much higher compute. ARC Prize said the tested system had been trained on part of the public training set. These scores measure that evaluation setup, not a general percentage of progress toward AGI.

Chollet called the result a significant advance in adapting to novel tasks, while noting its expense. I suggested costs could improve over time; that is a forecast. ARC Prize later revised its estimated retail costs as pricing assumptions changed, so the video’s rough dollar figures are not a settled invoice or current price.

Chollet did not consider the result AGI: the model still failed some apparently simple ARC tasks. An 85% threshold in a particular competition is a prize condition, not an agreed definition of AGI. The video speculates about his incentives, but provides no evidence that they determined his assessment.

The ARC Prize competition had separate rules: a fully private set, an open-source submission and a fixed compute budget, described in the video as ten cents per task. The high-compute semi-private score did not by itself win that competition. Benchmark versions, access and budgets matter when comparing scores.

One example o3 failed shows red shapes and blue markers in grids. I infer a cross from the blue markers, with intersected red shapes turning blue. It feels simple to me, but individual human reactions are not a controlled measurement of how easily all people solve it.

OpenAI researcher Steven Heidel described the moment in AGI-like terms, while Emad Mostaque predicted major economic disruption as AI performs computer-based work more cheaply. These are reactions and forecasts, not established conclusions. My suggestion that an AGI declaration would automatically end Microsoft's agreement is not supported by the information shown.

Immunologist Derya Unutmaz described a different model, o1 Pro, critiquing a review of MAIT cells and cancer. He found the feedback unusually insightful and wondered what a stronger model might mean for biology. This is one expert's reported experience, not an independently controlled evaluation of o3 or evidence of a clinical discovery.

Riley Goodside shared a graph of ARC scores rising across model generations. The jump in this particular benchmark is notable, but plotting a few historical results does not establish an exponential law or guarantee future progress. Deedy Das and others also emphasized the reported FrontierMath jump.

OpenAI researcher Noam Brown highlighted the roughly three months between o1 and the o3 announcement. David Dohan discussed a 16-hour figure and improving ARC results; the video interprets it as time spent on each answer, which is not established by that post alone. ARC Prize's later table reports about 13.8 minutes per task on average in the high-compute semi-private run, alongside substantial parallel sampling.

I repeated an estimate of roughly $350,000 and 5.7 billion tokens for the high-compute test. ARC Prize did report 5.7 billion tokens and a 172-fold compute increase, but its cost estimates depend on pricing assumptions and were subsequently revised. These are estimates for the benchmark run, not a confirmed per-user charge or an amount spent on one answer.

Gary Marcus predicted that the system would impress initially but might be unreliable, especially outside closed domains. I wanted hands-on testing before drawing that conclusion. ARC puzzles probe a form of adaptation, but their performance alone does not settle reliability in open-ended real-world situations.

Santiago Valdarrama called o3 a significant advance while arguing it was not AGI, remained costly and still failed easy tasks; he cautioned that generalization to other domains was uncertain. Ben Thompson saw it as a consequence of the earlier o1 insight that inference-time scaling can matter. Neither view proves that more computation always improves an answer.

Epoch AI's Tamay Besiroglu regarded the reported 25.2% FrontierMath score as earlier progress than he expected. Epoch later clarified that OpenAI commissioned and owned core benchmark questions and had access to statements and solutions, with a holdout arrangement. The access and funding relationship matters when interpreting the reported result; it does not by itself prove training contamination.

Sam Altman predicted that o3-mini would surpass o1 on many coding tasks at lower cost, while extra performance could demand much greater spending. These were 2024 forward-looking product claims, not independent results for every task.

NVIDIA researcher Jim Fan compared specialized AI achievements in Go, StarCraft, robotics and mathematical benchmarks. He argued that exceptional performance in a domain can coexist with failures on simple tasks, an illustration of the gap between narrow prowess and broad competence. I found the reactions compelling, especially the mathematical results, while the limitations remain important.