Grok 2 Beta: Matthew Berman Tests Reasoning and FLUX Image Generation

The mystery model called sus-column-r was an early version of Grok 2, according to xAI, and the Grok 2 beta has now been announced. I previously tested the anonymous model, so here I review the announcement and try the mini model and image generation. This is a snapshot of the August 2024 release, not a current product comparison.

xAI introduced Grok 2 and the smaller Grok 2 mini as successors to Grok 1.5, with chat, coding and reasoning capabilities. At launch the company said the two models were available to X Premium and Premium+ users and planned an enterprise API. Their weights were not released with this announcement.

xAI reported that the anonymous early version was ahead of Claude 3.5 Sonnet and GPT-4 Turbo on the LMSYS leaderboard at the time. I compare its position with several GPT-4o and Gemini versions. Those rankings are time- and methodology-dependent, not proof that one model wins every task.

The company describes having AI tutors compare pairs of responses, focusing on instruction following and factual accuracy. It also reports improvements in reasoning over retrieved content and tool use, including identifying missing information and discarding irrelevant posts. I value factual accuracy, but these are xAI’s evaluation claims rather than a guarantee that responses are true.

I compare the reported academic benchmark scores for Grok 2 and mini and note that many are fairly close. I expect a smaller model might become faster and cheaper, though that is an assumption rather than a measured result here. I also point out the strong comparison from Llama 3.1 405B, an open-weight model.

The release includes a redesigned interface and image generation in collaboration with Black Forest Labs using FLUX.1. I describe the model family as open source and attribute the team to Midjourney, but Black Forest Labs identifies its founders with latent diffusion and Stable Diffusion work; FLUX.1 variants also had different licenses: schnell used Apache 2.0, dev had non-commercial terms and pro was served through an API.

I am excited about an image generator that appeared less restrictive in my examples and about the competition among AI providers. Still, those examples do not establish that every request is allowed or that safeguards are absent. Capabilities, policies and availability can change.

In Grok 2 mini’s fun mode I ask where a marble ends up after a glass containing it is inverted, placed on a table, lifted and put in a microwave. The model gives a lengthy answer saying the marble is most likely inside the glass. I judge it wrong: under the ordinary interpretation the marble falls out or stays on the table rather than traveling inside an inverted glass.

My first requests for a llama image repeatedly produce an error, which I suspect may be due to load; the video does not establish the cause. I show other users’ synthetic images of public figures. Their appearance in a feed is not evidence that the depicted events occurred.

About an hour later image generation works, and a simple llama prompt produces an image. I then try a fictional character and several public-figure prompts, including a deliberately absurd image of Elon Musk. Some outputs are recognizable, while others miss parts of the request; they are generated images, not photographs of real events.

I ask for Mark Zuckerberg with different styling and for a Breaking Bad character, with mixed results. A more complex prompt involving Zuckerberg, Musk and Donald Trump also fails to represent the requested scene cleanly. The images look polished to me, but the complex prompt following is unreliable in this small test.

Finally, I ask for the day’s headlines and see a list of current events. Grok’s access to X posts can provide recent information, but a generated headline still needs verification. I close by observing how quickly striking synthetic images of public figures are spreading across the platform.