30초 요약
In this February 2026 short, Matthew Berman highlights Google’s Gemini 3.1 Pro benchmark table and explains the pattern-matching structure of ARC-AGI-2. Google reports 44.4% on Humanity’s Last Exam without tools, 77.1% on ARC-AGI-2, 94.3% on GPQA Diamond and 80.6% on SWE-Bench Verified under the listed settings. The 94.3% figure is for GPQA, not ARC-AGI-2.
- 1Humanity’s Last ExamGoogle reports 44.4% without tools on the full text and multimodal set.
- 2Abstract reasoningARC-AGI-2 uses example transformations; Google lists a verified Gemini 3.1 Pro score of 77.1%.
- 3Separate science scoreThe chart’s 94.3% corresponds to GPQA Diamond, not ARC-AGI-2.
- 4Coding scoreThe reported SWE-Bench Verified score is 80.6% for a single attempt, near the table’s 80.8% for Opus 4.6.
핵심 정리
- Google reported 44.4% on the full Humanity’s Last Exam without tools for Gemini 3.1 Pro Thinking (High).
- Its ARC Prize-verified ARC-AGI-2 score was 77.1%; the nearby 94.3% is the GPQA Diamond score.
- Google reported 80.6% on SWE-Bench Verified using a single attempt, close to the listed Opus 4.6 result of 80.8%.
- These historical benchmark results depend on the model and evaluation settings and do not establish performance on every task.
챕터별로 다듬은 전문 읽기
Gemini 3.1 Pro benchmark highlights
Berman points to the no-tools Humanity’s Last Exam score and explains ARC-AGI-2. He then highlights the GPQA and coding results shown in Google’s benchmark table.
Google’s launch table shows Gemini 3.1 Pro Thinking (High) at 44.4% on the full Humanity’s Last Exam without tools. I also highlight ARC-AGI-2, which gives examples of a visual transformation and asks the system to solve a new instance. Google reports an ARC Prize-verified score of 77.1% on that benchmark.
The 94.3% figure shown next is for GPQA Diamond, a scientific-knowledge benchmark, rather than ARC-AGI-2. Google’s table also reports 80.6% on SWE-Bench Verified with a single attempt, close to the 80.8% listed for Claude Opus 4.6. These are launch-era results under specific configurations, not a guarantee of equivalent performance across all coding tasks.
같은 채널의 다른 글
34:21EN
Matthew Berman· 2026년 10월 27일Matthew Berman Examines Anthropic’s IPO Filing, AI Growth and GovernanceMatthew Berman, Sam Altman · Forward FutureThe presenter reviews the S-1’s financial metrics, compute strategy and voting structure while questioning the tension between rapid AI development and safety.
0:26ENMatthew Berman· 2026년 10월 25일Sam Altman on Human Values and Problem-Solving in an AI WorldSam Altman · OpenAIIn a short interview excerpt, Altman considers the enjoyment of solving hard problems and the human role in interpreting results.
0:36ENMatthew Berman· 2026년 10월 25일Kimi K3.1 and a Rubik’s Cube Solver DemonstrationMatthew Berman · Forward FutureMatthew Berman shows a scrambled virtual cube being solved and gives the Kimi K3.1 test a favorable assessment.
0:21ENMatthew Berman· 2026년 10월 24일An AI-Generated Kimi Benchmark Chart Used Invented ScoresMatthew Berman · Forward FutureMatthew Berman asks for a graph of Kimi K3.1 Pro benchmarks, then discovers that the displayed figures are illustrative placeholders.
25:50ENMatthew Berman· 2026년 10월 24일Kimi K3.1 Pro Tested: Strong 3D Demos, Uneven Results and Competitive PricingMatthew Berman · Forward FutureMatthew Berman tests the open-weight model on coding, 3D rendering and visual explanations, then compares benchmark results and prices.
32:20ENMatthew Berman· 2026년 10월 23일Who Is Local AI For? The Costs, Trade-offs and a Hybrid FutureMatthew Berman · Forward FutureMatthew Berman weighs data control and independence against hardware cost, setup complexity and inference speed.