OpenAI o3 and o4-mini: Tool Use, Benchmarks and the First Codex CLI

SUMMARY

In 30 seconds

Matthew Berman examines the launch of o3 and o4-mini, emphasizing their ability to combine reasoning with tools. An OpenAI livestream clip demonstrates analysis of an old physics poster, followed by benchmark charts, cost comparisons and discussion of Codex CLI and platform risk. His theory about the models’ training origins and a new scaling law is speculation.

1Reasoning with toolsBerman highlights the models’ ability to call tools repeatedly during a task rather than only producing a single response.
2Physics poster demonstrationAn OpenAI presenter asks o3 to estimate a missing result from an old poster and compare it with later research. The presenter assesses the returned estimate as plausible.
3Benchmark conditions matterThe launch charts show different scores across mathematical, coding and scientific tests, including results with and without tools.
4A speculative training theoryBerman hypothesizes that the models came from GPT-5 checkpoints; he does not present evidence confirming that theory.
5Codex CLI and platform riskOpenAI introduced a terminal coding agent, prompting Berman to consider the risk of a model provider entering its customers’ markets.

Takeaways

OpenAI launched o3 and o4-mini in April 2025 with tool-assisted reasoning and image analysis.
In a livestream demonstration, o3 inspected a physics poster, estimated a missing result and searched for relevant literature; the presenter judged the estimate reasonable.
Benchmark results vary by task and tool access; tool-assisted and tool-free scores should be compared in context.
OpenAI’s API documentation lists text and image input and text output for these models, not native audio input and output.
Codex CLI began as an open-source terminal coding agent that connected local files and commands to cloud models.
Berman’s GPT-5 checkpoint theory and interpretation of tool use as a scaling law are conjectures, not established training details.

TopicsOpenAI Reasoning & test-time compute Benchmarks Coding agents Agent engineering

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

The o3 and o4-mini launch

Berman introduces the two reasoning models and emphasizes iterative use of tools. He frames several broader implications as his own views.

OpenAI has released o3 and o4-mini, and what excites me most is their ability to use tools as part of reasoning. Rather than taking one action and stopping, an agent can inspect a result, decide what to do next and use another tool. I think this could be an important direction for improving performance, though calling it a new scaling law is my interpretation, not an established result.

OpenAI describes o3 as its more capable model and o4-mini as a smaller, faster option. The models can reason over images, including by using tools to inspect them. Although I say in the video that they also accept and output audio, OpenAI’s API model documentation lists text and image input and text output; audio is not native support for these model APIs. Product-level tools can provide additional capabilities.

A physics poster demonstration

An OpenAI presenter asks o3 to analyze an unfinished physics poster and compare an estimated result with literature. The presenter comments on the result.

In a clip from OpenAI’s launch livestream, a presenter gives o3 a screenshot of a physics poster from a 2015 internship. The task is to estimate the proton isovector scalar charge and compare it with recent work. The final value is not actually printed on the poster: the model has to interpret the plot and infer what additional steps are needed.

SLIDE 1 / 7Analyzing a physics posterThe prompt asks o3 to estimate an unfinished physics result from an image and search the literature.

The presenter says the task requires extrapolating the plotted values to a physical quark mass and applying a normalization factor. The model zooms into the image and searches the web for later results. The presenter says reconnecting with the project and reviewing the literature would have taken substantial time by hand.

SLIDE 2 / 7The model’s estimated physics resultThe model describes its extrapolation and normalization assumptions for the presenter to assess.

The livestream presenter judges the model’s estimated values reasonable and says the final paper’s value was around 1.2. The answer also notes that the older estimate has more uncertainty than more recent results. This is a single demonstration assessed by its presenter, not an independent validation of all scientific research tasks.

The demonstration illustrates the combination of image inspection, calculation and web browsing. I see tool use and agent scaffolding as important alongside pretraining and reinforcement-learning-based post-training. My further suggestion that tool use is itself a separate scaling law is a hypothesis, not a quantified law shown by this demo.

Benchmarks and cost

Berman reviews math, science, coding and visual benchmarks, then speculates about model training. He compares OpenAI’s estimated inference-cost charts.

SLIDE 3 / 7AIME 2024 benchmarkThe launch chart shows model scores and identifies tool-free configurations.

OpenAI’s launch charts compare competition math results. For AIME 2024, the chart shows o3 at 91.6% and o4-mini at 93.4% without tools; it also presents AIME 2025 and Codeforces results. These are benchmark-specific measurements, not a ranking of all uses, and tool availability matters when comparing results.

On the displayed Humanity’s Last Exam chart, o3 rises from about 20.3% without tools to 24.9% with Python and browsing, while o4-mini moves from about 14.3% to 17.7% with tools. Deep research scores 26.6% in that chart. I speculate that additional agent scaffolding and tool use help explain the difference; the chart alone does not establish the cause.

SLIDE 4 / 7Humanity’s Last Exam with and without toolsThe chart distinguishes tool-assisted scores from tool-free results.

OpenAI also presents visual-reasoning, coding and instruction-following benchmarks. Its displayed SWE-bench Verified results are 69.1% for o3 and 68.1% for o4-mini, compared with 48.9% for o1. The SWE-Lancer chart expresses performance in terms of the dollar value attached to benchmark tasks; that is not evidence the models independently earned that money in the market.

I offer a theory about how the models were made: perhaps OpenAI took intermediate GPT-5 checkpoints and applied reinforcement learning to produce different reasoning models. That is my speculation based on public comments, not information confirmed by OpenAI. Strong benchmark results alone do not establish the models’ training lineage.

The cost charts compare estimated inference costs and performance at different reasoning levels. In the displayed AIME and GPQA examples, o4-mini improves on o3-mini at comparable estimated costs, and o3 medium and high compare favorably with o1 in the shown settings. Actual cost and performance vary with configuration and workload.

SLIDE 6 / 7Performance and estimated inference costThe chart compares o4-mini and o3-mini at different reasoning settings.

Codex CLI and access

He discusses OpenAI’s new coding agent and the platform risk of building on a single provider. He closes with launch-era access and his plans to test the models.

SLIDE 7 / 7The original Codex CLI previewOpenAI demonstrates its early terminal interface for a coding agent.

OpenAI also unveiled Codex CLI, an open-source coding agent that runs in a local terminal while using cloud-hosted models. It can work with local files and commands, bringing model reasoning into a coding environment. The original release described an experiment; its capabilities and approval mechanisms have evolved since this 2025 video.

This raises platform risk for developers building on a model provider: the provider can decide to build a product in an adjacent market itself. Codex CLI is a coding-agent example, even if it was initially a lightweight reference project. I think developers should consider diversification and open-source alternatives rather than relying on a single provider.

The launch also announced a $1 million initiative for projects using Codex CLI and OpenAI models, with grants in $25,000 API-credit increments. At launch, Plus, Pro and Team users were offered the models in the selector, Enterprise and Edu were scheduled to follow, and free users could try o4-mini through the Think option. These are historical launch details, not current availability.

I plan to test the models against the alternatives available at the time, including Gemini 2.5 Pro, particularly on coding. I am also interested in what developers think about the platform risks of building on top of OpenAI.