Testing Grok on X After the 2024 Grok-1 Weight Release

SUMMARY

In 30 seconds

After xAI released Grok-1 base weights in March 2024, Matthew Berman tested the hosted Grok service on X because he could not run the checkpoint locally. His informal prompts show successes and failures; the hosted product and search tools are distinct from a test of the raw base model.

1Hosted service versus base weightsBerman used Grok on X; xAI’s open release was an unfine-tuned base checkpoint.
2Snake game failsThe initial game and a proposed revision both failed during the demonstration.
3A misleading drying answerThe model’s 16-hour answer uses unjustified per-shirt reasoning; batch and parallel cases depend on capacity.
4A physical reasoning exampleThe model says the marble moves with an inverted cup, contrary to the usual interpretation of lifting the cup alone.

Takeaways

xAI released a 314B mixture-of-experts base checkpoint under Apache 2.0 in March 2024; it was not dialogue fine-tuned.
The hosted Grok service on X was tested, not the released raw weights locally.
The generated Snake game failed twice, while simple arithmetic and a JSON example worked.
The shirt-drying and hole-digging problems require explicit assumptions; a coincidental number does not validate flawed reasoning.
A few prompts do not establish overall model capability or safety.

TopicsOpen-weight models Benchmarks Models & training AI Safety

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Grok-1 release and hosted coding tests

Berman explains the open weight release and tests the product on X. Simple code succeeds while the Snake game fails.

In March 2024 xAI released the weights and architecture of Grok-1, a 314-billion-parameter mixture-of-experts base model under Apache 2.0. The released checkpoint was not fine-tuned for dialogue. I could not run it locally, so I test the Grok service on X instead; that hosted product and its search tools are not a controlled test of the newly released raw checkpoint.

The X interface can surface recent information from the platform. My first coding prompt, printing the numbers one through 100, produces a correct Python loop. A request for Snake appears to use search and returns relevant posts with code, so the product's tool access is part of the observed result.

The generated Snake game starts but immediately crashes with a variable error. I paste the error back and try the revised code; it runs briefly and fails again with another error. I mark this as a failure for my no-manual-debugging test. It does not establish that the model could never produce a working game under other conditions.

SLIDE 1 / 3Generated Snake codeThe hosted model proposes code, which fails when run in the demonstration.

Moderation and reasoning prompts

He probes the hosted interface with a harmful request, then tests arithmetic and logic. Several questions need explicit assumptions.

I also probe the hosted product with a potentially harmful request and observe that it responds rather than refusing. That single prompt cannot establish a general moderation policy or overall safety, and refusing harmful assistance would not itself be an ordinary task failure.

On the shirt-drying problem, Grok divides four hours by five shirts and multiplies by 20, producing 16 hours. I give it a pass in the video, but its per-shirt calculation is not justified by the question: simultaneous drying with sufficient space takes four hours, while four sequential batches of five take 16. The prompt does not specify capacity, so the answer requires an explicit assumption.

SLIDE 2 / 3Drying-shirts calculationThe response divides time by shirt count without establishing the necessary assumptions.

In the speed-order puzzle, the response says Jane is faster than Sam but begins with 'yes' even though I asked whether Sam is faster than Jane. I treat the explanation as a pass, but the direct yes/no answer conflicts with the stated ordering. Simple addition and the arithmetic expression 25 − 4 × 2 + 3 are answered correctly.

The model claims a self-referential reply contains 12 words; I count ten and mark it wrong. In the killers-in-a-room puzzle it counts two surviving original killers and the entrant, giving three. That is the intended answer if the question counts living people; including the deceased person would be a different interpretation.

Structured output and further puzzles

JSON, a marble puzzle, belief tracking and output constraints yield mixed results. Berman closes with his initial assessment.

A structured-data prompt yields a JSON object for three described people. In the cup-and-marble puzzle, the model says the marble travels with the inverted cup into a microwave, even after I clarify that the cup stays inverted. Under the usual assumption that the cup alone is lifted from the table, the marble remains on the table; the wording and handling assumptions matter.

SLIDE 3 / 3Marble and inverted cupThe response treats the marble as moving with the cup under the displayed interpretation.

For a belief-tracking question, Grok says John thinks a ball is in the box and Mark thinks it is in the basket, matching where each last left it and what each knows. A request for ten sentences ending in 'Apple' fails after the first sentence, illustrating that an apparently simple output constraint can be missed.

Finally, the model estimates that five people could dig a hole in one hour if they work simultaneously at a constant combined rate. I accept the answer with that assumption, while noting real work may not scale linearly. Overall I find the session promising, but the mixed results come from a few prompts in a search-enabled hosted interface and do not establish a general ranking or the raw checkpoint's behavior.