Reflection 70B: Self-Correction Claims and the Limits of a 2024 Demo

In September 2024, I look at Reflection 70B, introduced by Matt Shumer as a fine-tuned Llama model that can use “reflection” to identify and correct mistakes in a response. The release was accompanied by extraordinary performance claims. This video was sponsored by Vultr.

Shumer describes the 70-billion-parameter model as trained with reflection tuning, using examples of reasoning and correction. He claims it competes with leading proprietary models and predicts an even stronger 405B version. These are release claims and a forecast, not independently established results.

The benchmark table shown in the video compares the model with Claude, GPT-4o, Gemini and Llama on tests including MMLU, GPQA, HumanEval and GSM8K. It reports roughly 89.9% on MMLU and 99.2% on GSM8K; I describe the latter as essentially perfect. The figures come from the release presentation and should not be treated as independently verified.

After the September 6 video, independent evaluators reported being unable to reproduce the advertised results. A subsequent [Hugging Face model-card change](https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B/discussions/154) removed the benchmark claims, and [third-party testing discussions](https://huggingface.co/mattshumer/Reflection-Llama-3.1-70B/discussions/42) describe substantially weaker results under their conditions. The original claim that the model beat frontier competitors is therefore not established.

The online demonstration is heavily loaded, so I cannot run my full test at the time. The model weights were made available under the Llama 3.1 license; availability of weights does not mean an unrestricted open-source license. A model card described an initial upload issue, adding uncertainty about which version a test used.

One displayed example asks for the first sentence of the Declaration of Independence in mirrored writing. The response produces a plan inside a thinking tag, a reflection about possible meanings of mirrored writing, and an output. Its quoted text begins “We hold these truths,” which is not the document’s first sentence, so the confident reflection does not establish that the answer is correct.

The example shows that the model can generate text that revisits or qualifies an earlier step before a final answer. I caution that it is a single generated sequence, not evidence of a separate multi-agent system or an external verification loop. A model can condition later tokens on earlier generated text, but that does not guarantee reliable self-correction.

Shumer illustrates reflection with a count of the letter r in “strawberry”: the displayed response first says two, then corrects itself to three. I question how to interpret that correction and note that making an error and then revising it consumes additional tokens. One example cannot demonstrate a general reduction in hallucinations.

Another example asks whether 9.11 or 9.9 is larger. The response lays out a plan, compares decimal parts, reviews the reasoning and correctly selects 9.9. I point out that a much shorter answer would suffice for this easy case; extra reasoning has a cost even when the result is right.

I compare the method with prompting a model to solve a problem step by step and then check its work. Fine-tuning and prompting are different interventions, and the video does not provide a controlled experiment showing they produce the same performance. My interpretation is that the release encourages a structured reasoning format rather than proving a leap in underlying intelligence.

A Hacker News commenter offers a system prompt that asks for thinking, reflection and output sections. I read this as another way to elicit a similar style of answer. Similar-looking text alone does not show that the prompted model and the fine-tuned model behave identically or that their self-checks are reliable.

Shumer says the team checked for benchmark contamination and announced model weights, an API and a future 405B release. Berman repeats those statements as they stood at launch. A decontamination claim is not independent validation, and the future release and its predicted performance were not demonstrated in this video.

A shared example counts nine instances of r in a nonsense word, which I count manually and accept. I then manage to get one response from the busy service: it correctly says “strawberry” contains three r’s. This is a single informal test, not the full evaluation I intended.

I conclude that the reflection format is interesting and say I would like to test the model more thoroughly. The video’s broad enthusiasm for its benchmark standing reflects the release claims available at the time; later reproducibility concerns materially limit that conclusion.