Strawberry Rumors and Two Anonymous Chatbot Reasoning Tests

On August 7, 2024, I’m looking at the rumors building around OpenAI’s supposed next model. Sam Altman posted a picture of strawberries in a garden with the caption “I love summer in the garden.” The timing excited people following the rumored Strawberry or Q* project, though a photograph is not a product announcement.

Two anonymous model names have appeared in Chatbot Arena, bringing back memories of earlier anonymous model tests. I initially couldn’t find them myself, so I’m looking at reports and screenshots from other users. One listing is called anonymous-chatbot, while another has a different name; they should not automatically be treated as the same model.

One user asked anonymous-chatbot what it was and received a claim that it was based on GPT-4 architecture. A model’s own description is not proof of its identity. The same user reported very limited personal testing, with no obvious reasoning improvement and perhaps some improvement in math.

A social media thread collected the claims about Q* and Strawberry, including a possible new frontier model and a connection between the two project names. These were reports and speculation, not confirmed details about a release. The proposed improvement is better planning and reasoning, potentially helping with math and logic.

Other rumored capabilities include navigating the web to perform deeper research and completing long sequences of actions. Berman also discusses post-training and the possibility of continued learning after pretraining. Those are distinct concepts: post-training can change a model without establishing that a deployed system continuously learns from every interaction.

People also circulated an alleged jailbreak of anonymous-chatbot. The example does not establish which company made it. Bindu Reddy offered a different competitive prediction: other labs were making progress in reasoning and synthetic data, so she questioned how much advantage Strawberry might have over future competing models.

I point to Google DeepMind’s recent Olympiad work as evidence of progress. My suggestion that math reasoning was nearly solved goes too far: [DeepMind reported](https://deepmind.google/blog/ai-solves-imo-problems-at-silver-medal-level/) that AlphaProof and AlphaGeometry 2 solved four of six 2024 IMO problems, scoring 28 out of 42 with problems translated into formal language. That silver-medal-level result did not establish general mathematical reasoning as solved.

The other model was called sus-column-r. A screenshot shows it answering which is larger, 9.11 or 9.9: it writes 9.9 as 9.90, compares the tenths digits and correctly chooses 9.9. Some users speculated that the explanation revealed a new reasoning method or an OpenAI project, but a visible explanation alone does not establish either.

We can now add an important historical correction: on August 13, 2024, [xAI said](https://x.ai/news/grok-2) sus-column-r was an early version of its Grok-2 model. That confirmation does not identify the separately named anonymous-chatbot. OpenAI later introduced [o1-preview on September 12](https://openai.com/index/introducing-openai-o1-preview/), but that release does not prove an anonymous Arena identity.

Another social media account was enthusiastically predicting an imminent Strawberry launch. I don’t know who operates it or whether it has inside information, and its claims should be treated accordingly. At the time I felt competitive pressure from releases such as Llama 3.1 405B could push OpenAI to make an announcement.

After recording the first part, Shaun Ralston explained why I couldn’t locate the models directly. They appeared in Arena’s blind battle mode: users submit prompts to two anonymous models and learn their names after voting. Getting a particular model therefore required repeated trials.

For my first test, three killers are in a room and someone enters and kills one of them. Nobody leaves: how many killers remain? The GPT-4-0613 response says three, because two original killers are alive and the person who entered is also a killer. The ordinary answer counts living people and assumes the entrant is a distinct person.

The sus-column-r response breaks the situation into initial state, action, result and final count. It gives three when counting the entrant and two when counting only the original killers. The explanation is more elaborate in this instance, but one sample cannot establish broad superiority or expose a model’s hidden internal reasoning.

My harder test puts a marble in a glass, turns the glass upside down on a table, and then moves the glass to a microwave. Where is the marble? Under the ordinary interpretation, it stays on the table when the inverted glass is lifted. Two initial responses put it in the microwave; neither was one of the mystery models.

A second trial reveals anonymous-chatbot, and it too says the marble ends up in the glass in the microwave. I consider that wrong under the intended physical setup. As with many short puzzles, the answer depends on assumptions about how the glass is turned and lifted.

Ralston also shared a sus-column-r response to the same prompt. Its final answer places the marble on the table, outside the microwave, which matches the intended outcome. Some of its intermediate physical explanation is awkward, so a correct final answer should not be mistaken for a fully coherent explanation.

These are a few demonstrations rather than a full evaluation. I plan to run a broader test and ask viewers whether the rumored release is close or whether the strawberry post was simply teasing. The later identification of sus-column-r shows why an anonymous model’s name and output are weak evidence of its provider.