A No. 1 Gemini Ranking and the Limits of Early Model Tests

In November 2024, Google’s Gemini Experimental 1114 appeared at the top of a human-preference leaderboard. I want to look beyond that headline at what the ranking measures, what early tests do and do not show, and how model scaling may change.

At the time of recording, the API was having technical difficulties, so I could not run the full Simple Bench evaluation. This is an early snapshot, not a current comparison of Google, OpenAI and Anthropic models.

The leaderboard asks people to vote blindly between two model answers. Those preferences contain useful information, but they can also reflect response length and presentation style rather than reasoning quality alone.

In the snapshot I show, Gemini Experimental ranks first on the displayed overall measure but fourth on the style-controlled measure. For mathematics and hard prompts, o1-preview leads the categories I examined. Changing the evaluation changes the picture.

I also find the launch unusual: earlier Gemini releases came with more conventional benchmark materials, whereas this experimental model arrived with limited information. Reports about Google’s internal plans and disappointment are attributed to unnamed sources and should be read as reports, not confirmed design decisions.

Simple Bench tests questions designed to expose gaps in everyday reasoning. Without the working API, I could only try the ten public examples, not the full set. I guessed at a possible overall score, but that guess was not a measurement.

In my informal trials, the new Gemini usually answered three of the ten public questions correctly, occasionally four; I report around four or five for o1-preview and Claude on that sample. Ten public items and informal runs are too limited to settle a general ranking.

I also notice a 32,000-token limit in the interface and speculate about possible compute constraints. The limit alone does not establish the model’s size, architecture or the reason Google chose that setting.

The video turns to conversational behavior. Two troubling examples come from the separately available Gemini 1.5 Pro, not necessarily the new experimental model; one is a hostile response to a student. Such incidents matter for reliability, especially in sensitive conversations.

In a separate reasoning example, the experimental model revises an answer during its response but still arrives at an incorrect choice. Self-correction is interesting behavior, but an attempted correction is not evidence that the final answer is right.

Reports at the time suggested several leading labs were seeing less dramatic gains from the next round of models. Those reports cite sources inside the companies; they are not a complete view of internal research or a proof that progress has stopped.

Anthropic CEO Dario Amodei describes scaling laws as empirical regularities rather than laws guaranteed to hold forever. He expects the pattern to continue but acknowledges uncertainty. That distinction matters when extrapolating past trends.

My interpretation is not that language models have hit a permanent wall. I expect advances to continue, but think approaches such as giving a model more computation at inference time may become increasingly important alongside pretraining. That is an assessment, not a demonstrated inevitability.

Ilya Sutskever told Reuters that scaling pretraining had plateaued and that finding the right thing to scale mattered. The presenter connects that view to a broader search for new techniques, without claiming to know the details of his lab’s work.

Some OpenAI researchers expressed confidence in a path toward AGI, while also describing remaining engineering work and uncertainty. These are their views at the time, not evidence that AGI had been achieved or that a particular timeline was assured.

A public exchange between Sam Altman and ARC creator François Chollet illustrates the gap between hints and independently verified benchmark results. Chollet offered to evaluate a qualifying solution; the exchange itself did not demonstrate that the challenge had been solved.

The video closes its main discussion with changing positions on AI governance and a departing OpenAI employee’s concerns about trust. The common thread is uncertainty: leaderboard scores, anonymous reports and confident forecasts each answer different questions and have different limits.