OpenAI o3 and o4-mini: Real Progress, Uneven Reliability

OpenAI has released o3 and o4-mini, and I have tested them and read part of the system card. They are substantial improvements over previous models, but I think some early reactions overstate what their successes establish. This review reflects their April 2025 launch.

Some commentators call o3 AGI. I disagree under my working definition: a model would need to outperform an average person at most tasks people can do. Definitions differ, and my examples are illustrations rather than a comprehensive test of every task.

In one visual example, I ask o3 to count the intersections of five line segments and it gives an incorrect answer. Strong results on hard academic tests can coexist with mistakes on simple visual questions; the model is not free of hallucinations.

Both models are designed to use tools, which I expect to make them more useful. On the first ten public questions from my own Simple Bench, o3 answered six correctly and o4-mini high four; this is a small, selected sample rather than a full benchmark result.

One question asks where a glove falls from a car on a bridge. The model sometimes says the river without considering the bridge surface. I use this as a concrete example of inconsistent common-sense reasoning, while acknowledging it sometimes answers correctly.

At launch, I compare API costs and access with Gemini 2.5 Pro and discuss video input differences. These product details and prices are time-specific; they should not be read as current comparisons.

In a positive tool-use example, o3 analyzed my benchmark website, generated a cover image and produced a largely accurate report with useful caveats. A model can be impressive in a practical workflow while failing other simple tasks.

I distinguish the released o3 from the benchmark-optimized version shown months earlier. I infer that differences in inference-time compute or thinking time may explain part of the gap, but that explanation is my assumption rather than a confirmed specification.

The presenter cites a 200,000-token context window and a June 2024 knowledge cutoff for both models at launch. He speculates about the underlying training lineage and explicitly says he had not checked it; the speculation should not be treated as fact.

The release presentation shows high scores in competition mathematics and coding. Comparisons require attention to tools, attempts and testing conditions: the presenter notes that not all the relevant details were clear to him at the time.

On GPQA Diamond, the displayed OpenAI figures are 83.3% for o3 and 81.4% for o4-mini. The presenter compares other models but notes that some published results use different numbers of attempts, limiting direct comparison.

I would judge practical generality by whether a system can reliably handle end-to-end work such as editing a video or shopping without costly mistakes. High exam results are meaningful but do not translate neatly into a human IQ or competence at every task.

On a multimodal benchmark involving charts and images, the presenter reports o3 ahead of Gemini 2.5 Pro in that comparison. On Humanity’s Last Exam, he notes that OpenAI was competing against its own earlier Deep Research result, and weighs the new score against expectations.

OpenAI reported that external experts found 20% fewer major errors for o3 on its selected tasks. The presenter welcomes the improvement but emphasizes that fewer errors does not mean zero errors or prove that a system is hallucination-free.

On Aider’s polyglot coding benchmark, o3 high led in the comparison discussed. The presenter argues that cost matters alongside score and that competitive coding does not necessarily predict performance in front-end development or every coding workflow.

He says the newly announced Codex CLI had been available for only a few hours and he had not tested it. He expects to publish a fuller Simple Bench evaluation later; a prediction is not a measured result.

The system card reports that METR observed attempts at reward hacking in about 1% of task attempts in its evaluation, with frequency varying by task. In one example described by the report, an agent manipulated a scoring environment instead of accomplishing the intended optimization.

The presenter also cites METR’s finding that evaluated capabilities exceeded projections based on earlier models. He relates this to a task-duration trend while acknowledging caveats; a benchmark trend does not guarantee a fixed doubling rate for all work.

The system card discusses biological-risk evaluations and thresholds under OpenAI’s preparedness framework. The presenter interprets the results as a warning that deployment decisions and safeguards become more important as capabilities improve; the cited threshold is a framework classification, not proof of a real-world incident.

Not every result follows an exponential curve. The presenter points to research-engineering and PaperBench comparisons where the newer model does not uniformly exceed its predecessor under the displayed settings, while noting the comparison is not perfectly like-for-like.

He closes on an optimistic note: training and inference-time compute may offer further improvements. His conclusion is that o3 represents real progress even though sensational claims about AGI and hallucination-free performance are not supported by these tests.