An AI-Generated Kimi Benchmark Chart Used Invented Scores

SUMMARY

In 30 seconds

In this short video, Matthew Berman shows an AI-generated benchmark graphic and questions its broad categories and numbers. The generating assistant says the numbers are invented placeholders rather than verified results. The incident concerns that generated graphic; it does not establish that Moonshot AI fabricated its official benchmark results or identify the system used to generate the chart.

1A benchmark graphicBerman asks for an image displaying Kimi K3.1 Pro benchmarks; the generated chart uses broad categories.
2Vague categoriesThe chart groups scores under coding, reasoning, mathematics and agentic tasks rather than identifying specific tests.
3Scores described as placeholdersAsked whether the numbers are accurate, the assistant says they are invented placeholders, not verified results.

Takeaways

A generated graph presented broad benchmark categories and numerical scores.
The chart labeled its scores as illustrative, and the assistant later described them as invented placeholders.
The example does not establish that the model developer fabricated official benchmark results.

TopicsBenchmarks Evals Generative media

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

A generated chart and a follow-up

Berman shows the chart, questions its broad categories and asks whether its scores are accurate. The assistant says the figures are illustrative placeholders.

SLIDE 1 / 2The generated benchmark graphicThe chart displays broad categories and explicitly labels its normalized scores as illustrative, not verified benchmark results.

I asked an AI assistant to create an image of a graph showing Kimi K3.1 Pro benchmarks. The resulting graphic uses broad categories such as coding, reasoning, mathematics and agentic tasks, with numerical scores. Its small subtitle describes the figures as illustrative normalized scores, rather than naming actual tests or presenting verified measurements.

SLIDE 2 / 2The assistant’s clarificationThe displayed response says the numbers are invented placeholders and should not be used to compare models.

The numbers did not look accurate, so I asked whether they were real. The assistant replied that they were invented placeholders, not verified benchmark results. This is an example of a generated chart containing illustrative data; it does not establish that the model maker fabricated its official benchmark figures.