Claude Opus 5: Benchmarks, Cost per Task, and Early Impressions

Before the model announcement, Berman discusses a letter supporting open-weight AI. He argues that people can inspect, adapt, fine-tune, host, and serve models through different providers, creating more options and competitive pressure. His principal concern is the concentration of AI power among a small number of frontier companies, though he acknowledges environmental, censorship, and employment concerns as well.

Competition, in his view, can lower inference prices and make AI available for more uses. Berman also recognizes that NVIDIA has a commercial interest in this argument: more people running AI can mean more demand for its chips. His support for open weights is a judgment about the ecosystem, not proof that every open model or deployment will be inexpensive or safe.

The stream changes direction when Anthropic announces Claude Opus 5. Berman says he has had only brief access to the model and is seeing much of the announcement for the first time. He places it in the Claude family alongside Haiku, Sonnet, and Fable, and begins with the company's reported evaluations rather than a completed independent test.

The displayed table reports 43.3% for Opus 5 versus 33.7% for Fable 5 on Frontier-Bench's terminal-coding evaluation, and 70.6% versus 66.1% on OSWorld 2.0 computer use. It also shows differences in the other direction: Fable scores slightly higher on DeepSWE and the legal benchmark. These figures are tied to the particular evaluations and settings in the announcement.

Berman is especially interested in practical coding and agent work. He notes that DeepSWE is nearly flat relative to Fable in the chart even while some other scores rise. His initial reaction is enthusiastic, but he has not yet established how these differences will translate to his own work.

Anthropic also plots performance against the cost of each task. Berman argues that token price alone is insufficient: a model charging less per token can consume more tokens or require more attempts to produce an acceptable result. The useful comparison combines quality or success rate with the full cost of the task, under comparable conditions.

On the OSWorld chart shown in the stream, Opus 5's points sit above several alternatives at lower displayed costs. Berman reads this as evidence of improved efficiency in that test, and makes a similar observation about AutomationBench. These are launch charts, so their costs and rankings should be interpreted in the context of their configurations and workloads.

The Frontier-Bench chart also illustrates a tradeoff between effort and price. Berman observes that one higher-effort point costs more yet scores lower than the preceding point. More computation does not automatically improve every measured result; users need to test the setting that works for their own tasks.

ARC-AGI-3 presents interactive games without instructions, requiring a model to discover the rules and solve novel problems. The launch table shows Opus 5 at 30.2%, compared with 7.8% for GPT-5.6 Sol and 1.5% for Opus 4.8; Fable has no score in that table. Berman calls the increase striking while awaiting the benchmark organization's verified results.

The announcement also discusses alignment and cybersecurity. Berman notes that Opus 5 is reported to be stronger than Opus 4.8 on cybersecurity tasks but behind Mythos 5 at developing exploits. He speculates about training and safeguards; those guesses, including whether capabilities were deliberately removed, are not established by the chart.

Berman reads launch pricing of $5 per million input tokens and $25 per million output tokens, the same as Opus 4.8 and half the stated Fable price. The page also describes a faster mode at a higher rate. These are prices at the time of the video; actual task cost depends on usage, settings, and any subsequent pricing changes.

An optional API fallback can route requests flagged by safety classifiers to another model. Berman dislikes disruptive fallbacks and warns that the fallback model's pricing applies. The feature is a user choice described in the launch material, not evidence that every flagged request will have the same outcome.

Berman starts a live coding test: he asks the model to create a visually polished Rubik's Cube simulation with settings, scrambling, and step-by-step solving. The stream's screen capture freezes and crashes, so the audience cannot observe the test reliably. He later reports that the run has been working for nine minutes and used about 46,000 tokens, but does not show a finished result.

Asked about model loyalty, Berman says he currently uses GPT-5.6 Sol Medium and would switch if another model worked better, faster, and more cheaply for him. He says he is not paid by either of those model companies. His preference is presented as personal experience, not a controlled comparison.

Box sponsors a section of the stream and shares its own evaluation of document-grounded enterprise work across 12 industries. The examples include due diligence, drafting reports from data, reviewing expert work, and data analysis. Berman points out that the comparison is against Opus 4.8 and says he would also like to see GPT-5.6 Sol included.

Box's displayed chart shows full-dataset accuracy of 63% for Opus 4.8 and 68% for Opus 5. Due diligence rises from 65% to 76%, report drafting from 67% to 69%, expert review from 66% to 67%, and data analysis from 54% to 60%. Berman says 78% for the overall result, but the visible chart reads 68%. These are Box's results on its own evaluation.

With the live screen share still broken, Berman can only give a progress update on the Rubik's Cube task. He wonders whether Opus 5 was distilled from Fable 5, while explicitly labeling that as speculation. Neither the incomplete demo nor the benchmark graphs establishes the model's training method.

Berman relays a message from ARC Prize's Greg, who says the team is preparing verified scores and describes the model as impressive. The launch chart's 30.2% result is much higher than the earlier displayed comparison. Berman also discusses concerns about benchmark cheating; his suggestion that cheating is always obvious is not a reliable assurance, and independent verification remains important.

The conversation returns to safeguards and whether government review might affect availability. Berman speculates about the review process and the reasons for the cybersecurity results, without presenting confirmation. He also mentions an Anthropic claim that classifiers would intervene less often than for Fable 5, a claim that depends on the company's testing.

Berman checks other public comparisons, but Opus 5's Arena score is not yet available during the stream. He also reads an index combining finance and coding evaluations, where the model debuts near the top with a lower displayed cost than the leading entry. Different leaderboards measure different tasks and can produce different rankings.

His closing point is to judge both quality and the cost of completing useful work. A lower headline price is not enough on its own, and a strong benchmark is not a substitute for testing a representative workflow. The live demo remained incomplete from the audience's perspective, so the stream's conclusions are early impressions of the published results.