Reflection 70B: Conflicting Results and Unanswered Questions

In this September 9, 2024 video, I revisit my coverage of Reflection 70B after questions arose about its claimed performance and released weights. Some online critics alleged fraud, but I say the evidence available to me at the time is not definitive. I want to explain what I saw, distinguish independent results from accusations, and reflect on my own reporting.

On September 5, Matt Shumer announced Reflection 70B and claimed it led open models and exceeded some closed models on certain benchmarks. He described “reflection tuning,” published a Hugging Face release and credited Sahil from Glaive for help. He also discussed a possible 405B model and a forthcoming report; those were plans, not results established in this video.

Some people immediately questioned the reported benchmark figures, including an unusually high GSM8K score given known concerns about mislabeled answers. I initially reported the announcement enthusiastically. The demo was overloaded, so I could run only one prompt at first, which appeared to show the advertised thinking and reflection tags.

Shumer messaged me to say that if I tested the model I should use the newest upload because an earlier version was broken. I invited him onto a livestream instead, and he joined with Sahil. I say this was my first interaction with Shumer and that we discussed the release and audience questions for about half an hour.

Later on September 6, I recorded a longer test using a Hyperbolic-hosted demo, but my microphone was disconnected and the recording had no audio. I show the files and explain why I had planned to redo the video. The failed recording became awkward in light of the later controversy, but it was a recording mistake, as I describe it.

In the screen recording the model produced thinking and reflection tags but failed a word-count question. The tags show the format of the response; by themselves they do not establish which underlying model generated it or whether the announced benchmark claims were correct.

I review my informal test notes. Code-generation examples could not be evaluated because I encountered a local execution error and skipped them. Other simple questions produced a mixture of passes and failures; I did not find performance that clearly supported the exceptional benchmark claims. These examples are not a controlled benchmark.

Over the weekend, people raised concerns about the model's weights, performance and distribution. I describe the negative signals but say I still want to hear Shumer's response before making a definitive accusation. I then examine several public claims and independent evaluations as they stood on September 9.

Artificial Analysis reported that it could not reproduce the initial results and that the first released model scored below Meta's Llama 3.1 70B in its tests. A Reddit user separately argued, based on weight differences, that the uploaded model was based on Llama 3 rather than 3.1. These were outside parties' findings and claims, not a determination of intent.

Shumer said the Hugging Face weights had been mixed up during upload and promised a fix. I say I do not understand how that happened and want a fuller explanation. A problem with released weights and the reason for it are separate questions; the video does not establish the cause.

A private API subsequently produced stronger results for some testers. Critics questioned what model actually powered it because they could not inspect the backend; some alleged that it wrapped a different provider's model based on outputs they observed. Such outputs are circumstantial evidence, not conclusive verification of backend identity.

A questioner also asked Shumer to disclose his investment in Glaive. Shumer replied that he had previously disclosed a small investment, which he estimated at around $1,000. I think the relationship should also have been clear in the model release or accompanying praise so readers could evaluate the context.

Asked why the released model appeared to use Llama 3 instead of 3.1, Shumer said he did not yet know and that the team was retraining. He said the team was investigating discrepancies and planned to release data. I criticize one answer as insufficiently clear, but the discussion at this point remains unresolved.

Artificial Analysis later summarized its timeline: the first release underperformed Llama 3.1 70B, a private API performed impressively but below the original claims, and another public weights release again performed worse than the private endpoint. The evaluator said it could not independently verify exactly what powered that private API and planned to retest released weights.

I turn to a broader discussion by researcher Jim Fan about benchmark gaming and contamination. Training on closely related benchmark material or tuning heavily for public tests can inflate measured scores without proving comparable real-world performance. His general warning is not, by itself, evidence that any particular team used those methods.

Fan also discusses extra inference-time computation, including reflection and ensembles. Berman notes that spending more computation at test time can improve a result, so comparing systems without accounting for that budget can be misleading. Whether a particular technique constitutes cheating depends on the evaluation rules and disclosures.

For stronger evidence of model quality, Fan suggests independent human comparisons such as Chatbot Arena and evaluations using carefully curated private test sets. These approaches have their own limitations, but they address some risks of relying only on public benchmarks and unverified endpoints.

I say I still want Shumer to have a chance to respond even though I view the available signals as concerning. I also ask how I should improve my coverage: I was excited by the launch, relayed the provided claims and welcomed an interview out of curiosity. I consider applying more skepticism while still covering new developments promptly.

I describe the episode as a learning opportunity and invite feedback. At the time of recording, key questions about the released weights and private API had not been resolved in the evidence I reviewed; I expect to revisit the story as more information becomes available.