Testing GPT-4o Mini Against GPT-4o at Launch

OpenAI has released GPT-4o mini, a smaller and less expensive model. This video is sponsored by HP and Intel, which provided an EliteBook 1040 G11 for the demonstration.

OpenAI reports an 82% MMLU score and launch API pricing of 15 cents per million input tokens and 60 cents per million output tokens. It says the model supports text and image inputs, has a 128,000-token context window and an October 2023 knowledge cutoff. These are launch-era claims and prices.

I compare GPT-4o mini with GPT-4o in the Playground, with a helpful-assistant system prompt and a large output limit. This is an informal selection of prompts, not a controlled benchmark.

Both models write a Python script to print numbers from one to 100. For a Snake game, the mini model produces code quickly and the game opens on the first try. I notice that the snake can pass through its own body, so the result is playable but not a complete implementation of the usual rules.

I also test a safety bypass prompt and observe responses from both models. A single attempt cannot establish a model's overall safety or justify treating a refusal as a defect. I expect defenses and behavior to change.

For the laundry question, the model answers that 20 shirts can dry in four hours if all have enough sunlight and airflow. The answer depends on simultaneous drying and similar conditions. Both models also answer a basic arithmetic problem and a hotel-cost expression correctly in the displayed tests.

The mini model correctly counts the words in one response. For a puzzle about three killers and a new person entering and killing one, both answer three living killers; the result depends on whether the dead person is included in the word 'killers' and whether the entrant was already one.

In the sponsored segment I describe the HP laptop's mobility, Intel processor, camera and audio features, screen and locking behavior. I also cite battery life up to 20 hours; that is a product claim and actual battery life depends on use.

I ask where a marble ends up after a glass is turned upside down, placed on a table, lifted and moved. Mini answers that it remains on the table, while GPT-4o gives a different answer. The original wording leaves details of the inversion and placement unspecified, so the result depends on physical assumptions.

Both models miss one of ten requested sentences ending in 'apple' in my test. Both correctly say 9.9 is greater than 9.11. These individual answers are useful examples, not a comprehensive comparison.

For vision, both models explain a meme contrasting startups and large companies. I observe a much higher reported token count and slower start for mini in this run, although I do not know whether the display reflects a bug or a general property.

I ask both models to turn a spreadsheet screenshot into CSV, then query an iPhone storage screenshot. In the shown run they give the same remaining storage and identify Photos as the largest listed app. The apparent differences in vision latency and token counts are observations from these specific requests.

I'm impressed by the small model's speed, price and performance on this selection of prompts. I suggest choosing the larger model when vision latency matters, but the sample is too small to establish a universal ranking. I thank HP and Intel for sponsoring the video and providing the laptop.