Grok 4: Benchmark Results, Multi-Agent Reasoning and the Launch Demos

ZUSAMMENFASSUNG

In 30 Sekunden

Matthew Berman reviews xAI’s Grok 4 launch and explains why he sees reinforcement learning and additional inference compute as important. He surveys benchmark claims and launch demonstrations, tests the Heavy interface, and questions ambitious claims about AI-created games. The vending benchmark discussed is simulated, and the reported scores and prices reflect the launch period.

1Reinforcement learning at scalexAI depicts increased post-training compute as central to Grok 4; Berman explains rewards on checkable problems.
2HLE depends on configurationThe launch chart reports 26.9% without tools, 41% with tools and 50.7% with tools and additional test-time compute.
3Four agents in the interfaceBerman starts a difficult math question in Grok 4 Heavy; no final answer is shown in the video.
4ARC-AGI resultsARC Prize independently tested the model, reporting 66.6% on version 1 and 15.9% on version 2.
5Vending-Bench is simulatedThe business-management benchmark uses a simulated environment, despite the host describing it as real-world.
6Launch-era pricingSuperGrok was presented at $30 monthly and Heavy at $300 monthly or $3,000 annually in July 2025.

Das Wichtigste

Reported Humanity’s Last Exam scores rise with tools and additional test-time compute, making configuration essential to comparisons.
Grok 4 Heavy presents multiple agents working on a question, but the host’s example remains unfinished.
ARC Prize independently tested Grok 4 on ARC-AGI; benchmark results alone do not establish general intelligence.
Vending-Bench simulates a vending business rather than measuring real-world machine earnings.
A game built with model assistance and human direction does not establish that an AI can independently produce a major commercial game.
The video’s prices and product roadmap describe xAI’s July 2025 launch.

ThemenFoundation Models Benchmarks Reinforcement Learning Multi-agent systems AI cost & performance

VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Transkript zeigen

Nur das Transkript

Training and reinforcement learning

Berman introduces the launch and explains why he thinks verifiable rewards contributed to the model. He considers where future training feedback might come from.

Grok 4 has just launched, and I think it represents a significant leap among frontier models. That is my assessment of the results shown at the July 2025 launch, not a guarantee that it will be best on every task.

The launch chart traces the Grok family from next-token prediction through Grok 3 and its reasoning version to Grok 4. xAI depicts a major increase in post-training reinforcement-learning compute alongside pretraining; the change in training investment is central to its account of the new model.

FOLIE 1 / 7Training compute across the Grok seriesxAI’s launch chart illustrates its depiction of pretraining and reinforcement-learning compute.

Reinforcement learning with verifiable rewards gives a model problems with checkable answers and rewards correct solutions. Think of asking it to solve 2 + 2: the reward follows an answer of 4. Apply that idea repeatedly to much harder problems and, I argue, it can help elicit reasoning behavior.

I understand xAI to have encountered difficulty finding enough written problems with known answers. Elon Musk has suggested reality as a source of feedback. I think interaction with physical environments could provide many more verifiable outcomes, although turning those outcomes into useful training rewards is itself a practical challenge.

Humanity’s Last Exam and Heavy

The presentation compares several benchmark configurations, then introduces the multi-agent Heavy product. Berman shows its interface after a sponsored segment.

Humanity's Last Exam spans expert-level questions across mathematics, physics, biology, social science, computing, engineering and other fields. The launch presentation used it to show how Grok 4's score changed as xAI added tools and computation.

The chart showed Gemini 2.5 Pro at 21.6%, OpenAI o3 at 20% and o4-mini at 18%, with Grok 4 at 26.9% without tools. These are the results shown at launch, not an enduring model ranking; comparisons also depend on each test's configuration.

With tools such as web browsing and code execution, xAI reported 41% for Grok 4. With tools and increased test-time compute, its chart showed 50.7%. These configurations use different resources, so the three numbers should not be read as interchangeable single-model scores.

FOLIE 2 / 7Humanity’s Last Exam resultsThe chart distinguishes results with tools and increased test-time compute from the no-tool result.

I used to associate extra test-time compute mainly with giving a model more time to think. Grok 4 Heavy takes a multi-agent approach: xAI describes agents working on a problem and combining or selecting results. The presentation links additional compute to better performance, though the precise configuration matters when comparing results.

The video includes a sponsored segment for Abacus AI, which offers access to multiple models and agent tools. After the sponsorship, I show the Grok 4 Heavy subscription I bought and a short demonstration of its interface.

I give Heavy a mathematical question from Humanity's Last Exam, although I do not claim to understand the problem myself. Four agents appear and begin working independently. The screen shows their progress, but this brief view does not establish how they share information or whether their eventual answer is correct.

FOLIE 3 / 7Grok 4 Heavy interfaceFour agents begin the host’s difficult mathematics question; the screen does not show a final answer.

The distinction I draw is between Grok 4's single-agent experience and Heavy's multi-agent approach. Heavy is expensive, but I want to test it more thoroughly in another video. xAI also showed several live demonstrations during the launch.

Launch demonstrations

xAI presenters show a sports prediction, an illustrative black-hole visualization and an information-gathering timeline. Each demo has limitations.

In one launch clip, an xAI presenter uses Heavy and external information to estimate World Series odds. The model gives the Dodgers a 21.6% chance after roughly four and a half minutes. That is a forecast for a future event, not a verified outcome or proof of a betting edge.

Another presenter shows a visualization of two black holes colliding. The presenter says the waves and their falloff are exaggerated for visibility, and that the code uses post-Newtonian approximations rather than a full general-relativistic calculation near the black holes. It is an illustrative demonstration, not a validated simulation of the merger.

FOLIE 4 / 7Black-hole collision visualizationThe launch presenter explains that the visualization exaggerates wave scales and uses approximations.

A third launch demo builds a timeline of Humanity's Last Exam score announcements and reactions using X posts, including discussion of leaked scores. I find the ability to gather current information useful, but individual posts and rumored results still require verification.

Benchmarks and simulated business

Berman examines mathematics, coding and ARC-AGI scores as well as Vending-Bench. The vending test is a simulation.

The launch chart reports 87 on GPQA for Grok 4 without tools and 88.9 for Heavy; I assume the latter uses tools, although the video does not establish its exact configuration. It also shows 100% on AIME 2025, 79.4 on LiveCodeBench and 96.7 on Math Arena. These are reported results on particular tests, not evidence of uniform performance on every real-world task.

I consider Gemini 2.5 Pro a strong coder, but I have not yet tested Grok 4 myself. Returning to my Heavy run, nearly six minutes have elapsed and the interface shows each agent's progress without exposing its chain of thought.

ARC-AGI asks models to infer patterns from examples and apply them to unfamiliar visual tasks. I show results of 66.6% on ARC-AGI-1 and 15.9% on ARC-AGI-2. The ARC Prize team tested Grok 4 independently under its evaluation policy; the results are informative about those tasks, not proof that a system has achieved AGI.

FOLIE 5 / 7ARC-AGI-1 benchmark chartThe displayed results compare models on one version of the visual reasoning benchmark.

ARC Prize president Greg Kamradt described the 15.9% result as a breakthrough above a noisy low-score range and noted that the tested model was intended for public use. I see the result as evidence of stronger generalization on this benchmark, while recognizing that the benchmark tests a bounded set of tasks.

I contrast those tests with Vending-Bench, where a model manages inventory and a budget. The chart shows Grok 4 at roughly $4,700 and several other models substantially lower. I call it a real-world test in the video, but Vending-Bench is a simulated vending-business environment: its figures are simulated results, not earnings from an actual machine.

FOLIE 6 / 7Vending-Bench resultsThe chart reports model performance in a simulated vending-business benchmark.

Games, pricing and the roadmap

A game demonstration prompts a discussion of human judgment. Berman reviews launch pricing, an unfinished long-running test and xAI’s plans.

xAI also gave a game developer preview API access. A launch presenter says the developer made a first-person shooter in four hours and describes the model helping source assets. The demonstration includes human direction and existing assets; it does not establish that an AI independently built a commercial-scale game.

Musk predicts a really good AI video game in the following year. I am skeptical that AI will produce a game on the scale of Assassin's Creed or Halo on that timetable. He raises the need to play and judge games; to me, taste and the curation of experiences will keep humans involved for some time.

At the time of the July 2025 launch, Grok 4 was available in the app and via API. The announced features included a 256,000-token context window, multimodal reasoning and real-time search; I say the phrase 'enterprise-grade security' is too vague for me to assess on its own.

FOLIE 7 / 7SuperGrok subscription tiersThe screen shows launch-era prices and the benefits xAI listed for its subscriptions.

The launch pricing I show is $30 per month for SuperGrok and $300 per month or $3,000 per year for SuperGrok Heavy, with higher limits and early access to features. These are historical prices. My math problem is still running after almost 15 minutes, and three of the four agents appear far from completion.

Musk says Grok 4 uses a sixth-generation foundation model and that a seventh is in training. xAI's announced roadmap at the time included a coding model in August, a multimodal agent in September and video generation in October 2025. Those are launch-era plans rather than confirmed delivery dates; I am excited to test the model but uncertain whether the schedule will hold.