Claude Opus 4.1: Anthropic’s August 2025 Benchmark Update

SUMMARY

In 30 seconds

In an August 2025 video, Matthew Berman reviews Anthropic’s Claude Opus 4.1 announcement and its selected benchmark comparison. He sees gains in several coding tests but notes a lower airline tool-use score and says practical testing matters.

1Coding resultsThe displayed SWE-bench Verified result rose from 72.5% to 74.5%, a two-percentage-point increase.
2Mixed tool-use scoresTerminal-Bench increased, but the airline portion of TAU-bench fell from 59.6% to 56.0%.
3Compare by taskThe table shows other models ahead on GPQA Diamond and AIME 2025; Berman plans practical testing.

Takeaways

Anthropic reported 74.5% on SWE-bench Verified for Opus 4.1 versus 72.5% for Opus 4 in its displayed comparison.
Terminal-Bench rose in the table, while the airline portion of TAU-bench declined.
Other models led the displayed GPQA and AIME comparisons; results depend on evaluation settings and task.
Berman treats real-world usage as an important complement to benchmarks.

TopicsModels & training Evals Coding agents

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

The Opus 4.1 release

Berman introduces Anthropic’s update and its stated goals.

Anthropic announced Claude Opus 4.1 in August 2025 as an update to Opus 4 for agentic work, coding and reasoning. I review the company’s reported benchmark table, which reflects its selected tests and settings at the time.

Benchmarks and real-world use

He reviews coding, tool-use and reasoning scores and emphasizes practical evaluation.

On SWE-bench Verified, the displayed scores rise from 72.5% for Opus 4 to 74.5% for Opus 4.1: an increase of two percentage points. Anthropic also describes improvements in research and analysis, including tracking details and agentic search.

SLIDE 1 / 1Anthropic’s selected benchmark tableThe table compares coding, tool use and reasoning under the reported evaluation settings; the airline score declines.

The table shows Terminal-Bench increasing from 39.2% to 43.3% and GPQA Diamond from 79.6% to 80.9%. TAU-bench retail rises from 81.4% to 82.4%, while airline falls from 59.6% to 56.0%. Other displayed categories show smaller gains.

In the displayed comparison, Opus 4.1 leads the included o3 and Gemini 2.5 Pro results on the two coding benchmarks, while those models score higher on GPQA Diamond and AIME 2025. These selected numbers do not establish a universal ranking.

I think the important question is how the model works on actual tasks, and I plan to test it. My view that Claude was particularly strong for agentic coding is an assessment from 2025, not a current leaderboard or a guarantee for every workflow.