Gemini 3.1 Pro and the Limits of AI Benchmarks

After testing Gemini 3.1 Pro and reading its model card at its February 2026 release, I want to look beyond the headline scores. Different users may see conflicting rankings because models are optimized and evaluated on different tasks.

I describe a shift from broad pretraining toward post-training on specific domains and internal evaluations. My estimate that pretraining accounts for only 20% of training compute is a claim about the industry at the time, not a documented universal ratio for every model.

If a lab has relevant data and optimizes for a domain, its model may excel for one user while lagging on another benchmark. The older assumption that improvement in one area predicts improvement everywhere is less reliable.

For example, I cite an Epoch AI chess puzzle benchmark where Sonnet 4.5 scored about 12% and the later Opus 4.6 about 10%, despite the latter’s strong coding abilities. These particular results do not rank the models universally, and the tests and settings matter.

Gemini 3.1 Pro is competitive across many reported tests, including scientific reasoning and coding, but Google’s launch table also shows a lower GDPval-AA score than competing models. One table does not capture all uses or guarantee independent replication.

Google reported 77.1% for Gemini 3.1 Pro on ARC-AGI-2, compared with around 69% for Claude Opus 4.6 in the displayed table. Melanie Mitchell and others pointed out that changing numeric encodings can reduce accuracy because the original format may offer unintended arithmetic cues.

That does not mean a model is deliberately cheating. It does mean that a benchmark can reward shortcuts unrelated to the intended abstraction, and that small changes in the way questions are encoded can change the result.

François Chollet compares sufficiently advanced agentic coding to iterative machine learning: an agent optimizes toward a goal but the resulting code can be hard to inspect. A system can overfit a specification or drift from the actual intent, so benchmark success still needs human review.

Gemini’s reported LiveCodeBench Pro result is strong, but my own experience in Cursor included unhelpful output. A competitive programming score and a user’s experience of a particular coding task are different measurements.

On my private Simple Bench, Gemini 3.1 Pro scored 79.6%, close to the 83.7% human baseline shown in the video. That baseline came from only nine participants, so it is a limited sample and cannot establish broad parity with people.

I argue that this is a notable moment for text-only English tests aimed at non-specialists, although models still fail some visual or audio puzzles people solve. My broader claim about what an average person could outperform is an interpretation, not something proved by this small sample.

Simple Bench has its own shortcut problem. Multiple-choice options can reveal that a question is a trick: an option of zero in an ice-cube puzzle, for instance, may prompt the model to consider melting rather than perform only the arithmetic.

When I remove the choices and have a blind grader compare open-ended answers with the hidden solution, scores fall by roughly 15–20 percentage points. That shows how evaluation format matters, even though the models still solve some questions.

Hallucinations are another dimension. Artificial Analysis’s Omniscience index rewards correct answers and penalizes hallucinations; the chart shown gives Gemini 3.1 Pro a higher aggregate score than the compared Claude models under that test.

Looking only at how incorrect responses are handled changes the comparison: I report that roughly 50% of Gemini’s non-correct responses were hallucinations, versus 38% for Sonnet 4.6 and 34% for GLM-5. These figures describe the cited evaluation and definitions, not a universal hallucination rate.

Google’s model card provides additional context. In the cyber evaluations discussed, the card says Deep Think did not show higher capability than the model without it when inference costs were accounted for. A more intensive mode does not automatically win every test.

The model card also reports a machine-learning research task in Deep Think mode: a fine-tuning script’s runtime fell from 300 seconds to 47, against a 94-second human reference. The displayed card says the average across challenges remained below its alert threshold; one strong task is not proof of autonomous self-improvement.

I suggest that domain-specific data or internal evaluation might help explain strong fine-tuning performance. That is a hypothesis, not a disclosed finding about Google’s training data.

Anthropic CEO Dario Amodei presents a different argument for specialization: training across many environments can develop capabilities that generalize beyond any single domain. I interpret this as a bet that enough varied specializations may produce broad transfer.

Amodei argues that much progress toward highly capable systems may be possible without continual learning on the job. He also discusses longer context as a way to supply missing domain information. Whether this works across real tasks remains an open question.

A model may see extensive domain material in a context window and infer patterns without retraining its weights. The presenter asks how much application-specific training is needed versus generalization and in-context learning; the balance can differ by task.

I question the hunt for one definitive benchmark of general intelligence. If such a measure existed, labs would have a strong incentive to optimize directly against it, which could make the score less useful as a general measure.

Small independent teams face a difficult task creating realistic evaluations, while labs have greater resources but also incentives that can bias benchmark selection or presentation. Results need independent scrutiny and attention to the test design.

Forecasting offers another evaluation: Metaculus reported improving model performance approaching its average human forecaster, though not the professional level in the comparison cited. I also speculate that agents able to influence real events might game prediction markets; that is a hypothetical risk.

Speed is another metric. A demo produces an answer extremely quickly, but the video also marks a problem: the generated explanation of Simple Bench is incorrect. High throughput alone says nothing about whether the answer is useful or true.

I mention my own comparison site and close with a subjective comparison of generated video realism, showing examples from Seedance 2.0 and Veo. Different tasks call for different evaluations, so a single headline ranking is an incomplete guide.