Testing Grok on X After the 2024 Grok-1 Weight Release

In March 2024 xAI released the weights and architecture of Grok-1, a 314-billion-parameter mixture-of-experts base model under Apache 2.0. The released checkpoint was not fine-tuned for dialogue. I could not run it locally, so I test the Grok service on X instead; that hosted product and its search tools are not a controlled test of the newly released raw checkpoint.

The X interface can surface recent information from the platform. My first coding prompt, printing the numbers one through 100, produces a correct Python loop. A request for Snake appears to use search and returns relevant posts with code, so the product's tool access is part of the observed result.

The generated Snake game starts but immediately crashes with a variable error. I paste the error back and try the revised code; it runs briefly and fails again with another error. I mark this as a failure for my no-manual-debugging test. It does not establish that the model could never produce a working game under other conditions.

I also probe the hosted product with a potentially harmful request and observe that it responds rather than refusing. That single prompt cannot establish a general moderation policy or overall safety, and refusing harmful assistance would not itself be an ordinary task failure.

On the shirt-drying problem, Grok divides four hours by five shirts and multiplies by 20, producing 16 hours. I give it a pass in the video, but its per-shirt calculation is not justified by the question: simultaneous drying with sufficient space takes four hours, while four sequential batches of five take 16. The prompt does not specify capacity, so the answer requires an explicit assumption.

In the speed-order puzzle, the response says Jane is faster than Sam but begins with 'yes' even though I asked whether Sam is faster than Jane. I treat the explanation as a pass, but the direct yes/no answer conflicts with the stated ordering. Simple addition and the arithmetic expression 25 − 4 × 2 + 3 are answered correctly.

The model claims a self-referential reply contains 12 words; I count ten and mark it wrong. In the killers-in-a-room puzzle it counts two surviving original killers and the entrant, giving three. That is the intended answer if the question counts living people; including the deceased person would be a different interpretation.

A structured-data prompt yields a JSON object for three described people. In the cup-and-marble puzzle, the model says the marble travels with the inverted cup into a microwave, even after I clarify that the cup stays inverted. Under the usual assumption that the cup alone is lifted from the table, the marble remains on the table; the wording and handling assumptions matter.

For a belief-tracking question, Grok says John thinks a ball is in the box and Mark thinks it is in the basket, matching where each last left it and what each knows. A request for ten sentences ending in 'Apple' fails after the first sentence, illustrating that an apparently simple output constraint can be missed.

Finally, the model estimates that five people could dig a hole in one hour if they work simultaneously at a constant combined rate. I accept the answer with that assumption, while noting real work may not scale linearly. Overall I find the session promising, but the mixed results come from a few prompts in a search-enabled hosted interface and do not establish a general ranking or the raw checkpoint's behavior.