Grok 4: Benchmark Results, Multi-Agent Reasoning and the Launch Demos

Grok 4 has just launched, and I think it represents a significant leap among frontier models. That is my assessment of the results shown at the July 2025 launch, not a guarantee that it will be best on every task.

The launch chart traces the Grok family from next-token prediction through Grok 3 and its reasoning version to Grok 4. xAI depicts a major increase in post-training reinforcement-learning compute alongside pretraining; the change in training investment is central to its account of the new model.

Reinforcement learning with verifiable rewards gives a model problems with checkable answers and rewards correct solutions. Think of asking it to solve 2 + 2: the reward follows an answer of 4. Apply that idea repeatedly to much harder problems and, I argue, it can help elicit reasoning behavior.

I understand xAI to have encountered difficulty finding enough written problems with known answers. Elon Musk has suggested reality as a source of feedback. I think interaction with physical environments could provide many more verifiable outcomes, although turning those outcomes into useful training rewards is itself a practical challenge.

Humanity's Last Exam spans expert-level questions across mathematics, physics, biology, social science, computing, engineering and other fields. The launch presentation used it to show how Grok 4's score changed as xAI added tools and computation.

The chart showed Gemini 2.5 Pro at 21.6%, OpenAI o3 at 20% and o4-mini at 18%, with Grok 4 at 26.9% without tools. These are the results shown at launch, not an enduring model ranking; comparisons also depend on each test's configuration.

With tools such as web browsing and code execution, xAI reported 41% for Grok 4. With tools and increased test-time compute, its chart showed 50.7%. These configurations use different resources, so the three numbers should not be read as interchangeable single-model scores.

I used to associate extra test-time compute mainly with giving a model more time to think. Grok 4 Heavy takes a multi-agent approach: xAI describes agents working on a problem and combining or selecting results. The presentation links additional compute to better performance, though the precise configuration matters when comparing results.

The video includes a sponsored segment for Abacus AI, which offers access to multiple models and agent tools. After the sponsorship, I show the Grok 4 Heavy subscription I bought and a short demonstration of its interface.

I give Heavy a mathematical question from Humanity's Last Exam, although I do not claim to understand the problem myself. Four agents appear and begin working independently. The screen shows their progress, but this brief view does not establish how they share information or whether their eventual answer is correct.

The distinction I draw is between Grok 4's single-agent experience and Heavy's multi-agent approach. Heavy is expensive, but I want to test it more thoroughly in another video. xAI also showed several live demonstrations during the launch.

In one launch clip, an xAI presenter uses Heavy and external information to estimate World Series odds. The model gives the Dodgers a 21.6% chance after roughly four and a half minutes. That is a forecast for a future event, not a verified outcome or proof of a betting edge.

Another presenter shows a visualization of two black holes colliding. The presenter says the waves and their falloff are exaggerated for visibility, and that the code uses post-Newtonian approximations rather than a full general-relativistic calculation near the black holes. It is an illustrative demonstration, not a validated simulation of the merger.

A third launch demo builds a timeline of Humanity's Last Exam score announcements and reactions using X posts, including discussion of leaked scores. I find the ability to gather current information useful, but individual posts and rumored results still require verification.

The launch chart reports 87 on GPQA for Grok 4 without tools and 88.9 for Heavy; I assume the latter uses tools, although the video does not establish its exact configuration. It also shows 100% on AIME 2025, 79.4 on LiveCodeBench and 96.7 on Math Arena. These are reported results on particular tests, not evidence of uniform performance on every real-world task.

I consider Gemini 2.5 Pro a strong coder, but I have not yet tested Grok 4 myself. Returning to my Heavy run, nearly six minutes have elapsed and the interface shows each agent's progress without exposing its chain of thought.

ARC-AGI asks models to infer patterns from examples and apply them to unfamiliar visual tasks. I show results of 66.6% on ARC-AGI-1 and 15.9% on ARC-AGI-2. The ARC Prize team tested Grok 4 independently under its evaluation policy; the results are informative about those tasks, not proof that a system has achieved AGI.

ARC Prize president Greg Kamradt described the 15.9% result as a breakthrough above a noisy low-score range and noted that the tested model was intended for public use. I see the result as evidence of stronger generalization on this benchmark, while recognizing that the benchmark tests a bounded set of tasks.

I contrast those tests with Vending-Bench, where a model manages inventory and a budget. The chart shows Grok 4 at roughly $4,700 and several other models substantially lower. I call it a real-world test in the video, but Vending-Bench is a simulated vending-business environment: its figures are simulated results, not earnings from an actual machine.

xAI also gave a game developer preview API access. A launch presenter says the developer made a first-person shooter in four hours and describes the model helping source assets. The demonstration includes human direction and existing assets; it does not establish that an AI independently built a commercial-scale game.

Musk predicts a really good AI video game in the following year. I am skeptical that AI will produce a game on the scale of Assassin's Creed or Halo on that timetable. He raises the need to play and judge games; to me, taste and the curation of experiences will keep humans involved for some time.

At the time of the July 2025 launch, Grok 4 was available in the app and via API. The announced features included a 256,000-token context window, multimodal reasoning and real-time search; I say the phrase 'enterprise-grade security' is too vague for me to assess on its own.

The launch pricing I show is $30 per month for SuperGrok and $300 per month or $3,000 per year for SuperGrok Heavy, with higher limits and early access to features. These are historical prices. My math problem is still running after almost 15 minutes, and three of the four agents appear far from completion.

Musk says Grok 4 uses a sixth-generation foundation model and that a seventh is in training. xAI's announced roadmap at the time included a coding model in August, a multimodal agent in September and video generation in October 2025. Those are launch-era plans rather than confirmed delivery dates; I am excited to test the model but uncertain whether the schedule will hold.