DeepSeek R1 0528: Benchmark Gains and Two Coding Tests

요약

30초 요약

In May 2025, Matthew Berman reviews DeepSeek-R1-0528, an open-weight update that reported stronger reasoning and coding scores than the original R1. He contrasts DeepSeek’s published benchmarks and Artificial Analysis’s contemporary rankings with two informal programming tests. The generated cube appeared after a Three.js initialization fix but did not rotate correctly; the Snake game errored and then ended immediately after a repair. Berman finds those results disappointing despite the reported benchmark improvements. Rankings, provider speeds and product details reflect the time of the video.

1A post-training updateDeepSeek describes the release as a minor version upgrade using more computational resources and post-training optimizations. Berman describes it as a substantial leap for an open-weight model.
2Improved reported benchmarksDeepSeek reports AIME 2025 pass@1 rising from 70.0 to 87.5 and GPQA Diamond from 71.5 to 81.0. These are vendor-reported results under its own evaluation protocol.
3Independent index comparisonArtificial Analysis reported a rise from 60 to 68 on its then seven-evaluation intelligence index, tying the specified Gemini 2.5 Pro preview and trailing o3.
4Rubik’s Cube testThe generated program initially showed controls without a cube. After Berman fixed a Three.js initialization issue, the cube rendered but scrambling and rotation did not work correctly.
5Snake test falls shortThe 1,117-line program first produced a name error. After Berman asked for a fix, the game appeared but immediately ended.

핵심 정리

DeepSeek reported that R1 0528 improved on its predecessor across several reasoning and coding benchmarks, under specified evaluation settings.
Artificial Analysis’s 2025 seven-evaluation index placed the updated R1 at 68, up from 60 for the original.
More reasoning tokens can accompany improved results, but they also increase inference work.
Berman’s two informal coding tests exposed failures despite the reported benchmark gains.
The published model weights do not make hosted inference or the hardware needed to run the model free.

주제모델과 학습 Benchmarks Open-weight models Reasoning & test-time compute Software engineering

전문

챕터별로 다듬은 전문 읽기

원본 자막과 대조

원본 자막만 보기

The R1 update and benchmark claims

Berman introduces the May 2025 release and compares DeepSeek’s reported benchmark results with other models. The model card describes post-training improvements and publishes evaluation settings.

DeepSeek has released DeepSeek-R1-0528, an update to its reasoning model. The weights appeared first, followed by more information about what changed. DeepSeek calls it a minor version upgrade, but the reported improvements look substantial.

DeepSeek attributes the update to increased computational resources and algorithmic optimizations during post-training. It says the model improved in mathematics, programming and general logic and is approaching o3 and Gemini 2.5 Pro overall. Its weights are available under the MIT license; running the model still requires compute, and hosted services may charge for inference.

슬라이드 1 / 8DeepSeek’s benchmark comparisonThe release chart compares R1 0528 with earlier R1 and selected proprietary models across reasoning and coding benchmarks.

The benchmark chart compares the previous R1 with the new release. In [DeepSeek’s model card](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528), AIME 2024 pass@1 rises from 79.8 to 91.4, AIME 2025 from 70.0 to 87.5, GPQA Diamond from 71.5 to 81.0, and LiveCodeBench from 63.5 to 73.3. Humanity’s Last Exam rises from 8.5 to 17.7. For sampled benchmarks the card says it used 16 responses per query to estimate pass@1; its HLE evaluation used text-only prompts.

The chart places the new R1 close to o3 on AIME 2024, somewhat behind on AIME 2025, GPQA Diamond and LiveCodeBench, and further behind on Aider. The exact comparison depends on the evaluations and configurations shown; these figures do not establish that the models are interchangeable across all uses. I am surprised that Gemini 2.5 Pro, which I rate highly for coding, trails o3 on most of this chart.

Independent evaluation and reasoning costs

He reviews Artificial Analysis’s then-current index and token consumption. He argues the gap between open-weight and proprietary models is narrowing.

슬라이드 2 / 8Artificial Analysis evaluation summaryThe independent evaluator reports the updated model’s historical index change and comparison.

Artificial Analysis also evaluated the release independently. Its May 2025 intelligence index, then made up of seven evaluations, put R1 0528 at 68 versus 60 for the original, level with the Gemini 2.5 Pro preview shown and behind o3. It described this as an eight-point jump, the same size as the difference between its o1 and o3 scores. These are rankings under that index at that time.

I understand this as an R1 update rather than a new architecture: the model has 671 billion parameters, with 37 billion active. Artificial Analysis also reported a major improvement in its coding index. In my early tests, the volume of code the model was willing to produce for tasks such as a cube and a Snake game was substantial, comparable to Gemini 2.5 Pro.

The new version also uses more reasoning tokens. Artificial Analysis reported 99 million tokens for its intelligence-index evaluations, about 40% more than the original R1; Gemini 2.5 Pro used still more in that comparison. In effect, the new model spends more tokens thinking, which matters when considering the work and cost behind the result.

To me, the broader story is that the gap between open-weight and proprietary models was narrowing. Artificial Analysis also saw Chinese labs approaching US labs on its index. DeepSeek says post-training optimization and more compute helped improve reasoning; I interpret the release as getting more from the existing pretrained model. On the chart, the January R1 scores 60 and the May update 68; I am disappointed by where Llama 4 Maverick appears.

슬라이드 3 / 8May 2025 intelligence indexThe chart shows the seven evaluations and the scores used in the historical comparison.

Testing an interactive cube

Berman asks for a Three.js Rubik’s Cube simulator and fixes a library initialization issue. The rendered cube still fails to scramble and rotate correctly.

For a hands-on test I turn on DeepThink and use the same prompt I give other models: write a complete HTML and JavaScript program with Three.js for an interactive Rubik’s Cube up to 20 × 20 × 20, with dynamic size, colored faces, camera controls and layer rotation. In my prior tests, both Gemini 2.5 Pro versions handled this particularly well.

The new R1 spends 328 seconds thinking before finishing its response. Its visible reasoning considers sticker positions, meshes and performance. Running the result initially shows settings but no cube, including when I try it locally. I find that Three.js was not initialized and fix that issue; the cube then appears.

슬라이드 4 / 8Cube simulator before initialization fixThe generated interface shows controls but no cube in the initial run.

슬라이드 5 / 8Cube after the initialization fixThe cube renders, but its pieces rotate incorrectly during the test.

The initial appearance is promising, but the scramble and solve behavior disappoint me. The cube does not scramble properly and seems to solve itself; the rotation and physics do not work as intended. In this particular test, I still prefer the result I saw from Gemini 2.5 Pro.

Provider comparison and Snake game

While a second program generates, Berman reviews historical model progress and provider speed. The Snake game errors and then ends immediately, leaving him unimpressed by his limited hands-on tests.

Next I request an advanced Snake game with different foods, power-ups and teleportation. While it generates, I look at Artificial Analysis’s historical chart: it shows a major jump around the arrival of reasoning models, another when the original R1 appeared, and a further gain for the update.

슬라이드 6 / 8Historical inference provider speedsArtificial Analysis compares output tokens per second across providers in May 2025.

The provider chart shown in the video puts Fireworks first at 253 output tokens per second for this R1 variant. I also discuss provider-dependent context limits and prices. Those figures describe the services and measurements available in May 2025, not a current ranking or guarantee.

The Snake request takes 22 seconds of thinking, followed by a lengthy output of 1,117 lines. The first run fails with a “player_snake is not defined” error. I return the error to DeepSeek and try its revision; the game appears but ends immediately. These are only a few informal tests, but despite the promising benchmark jump, I am unimpressed by what I saw the model accomplish here.