After SWE-bench Hits 80.9%, Dianne Na Penn Reckons with Opus 4.5

Anthropic dropped Claude Opus 4.5 on Thanksgiving Eve with no teaser poster and no “something tomorrow” tweet. Jacob Effron, hosting *Unsupervised Learning*, treats it as a holiday gift for founders. Dianne Na Penn — everyone calls her Diane — is head of product for research, the bridge between model capability and the product. She has been at the company about two and a half years. When she joined it was roughly 150 people.

Starting a new Opus, she says, does not look like a pure research wander. Anthropic keeps an ambitious, long-range capability roadmap: better instruction following, coding, memory. Each generation of Claude is a vehicle for those improvements. Designing a version means deciding which advances must land, then packaging, positioning, and pricing them for jobs users have today and jobs they do not yet know AI can take on. The process will sound familiar to anyone who has written a PRD: what is this, why would someone come use it, which evals tell you it works.

Some bets are top-down. The team has a strong prior that the technology can be transformative in engineering and beyond. Others arrive from the bottom. A relatively small investment earlier in the year made Claude good at Excel and PowerPoint. Financial-services customers lit up, so they doubled down. Users tell you, by actually using the thing, which capabilities deserve more chips. Doubling down, for practical work, starts with those users and with an imagination phase that looks like a product-vision doc, then with evals. Sometimes the model is already halfway there and needs data or RL for the last fifty percent. Sometimes it is at twenty percent and you need a much bigger change.

Asked what Opus 4.5 finally does that lived in a vision doc a year ago, Diane points to more complex agentic coding and, especially, work that is iterative and longer-running. Computer use is the other line they have funded for a long time. They launched a computer-use API around October or November of the previous year. She uses Claude for Chrome, the browser extension, often, and felt the jump because vision got better. Sometimes the threshold is several abilities locking together, none of them spectacular alone.

Her map of the journey is a progress bar. Computer use started as an experimental feature, imagined as a complement sitting on top of something core like agentic coding. Between Opus 4.5 and what follows, she sees it becoming an end-to-end agent by itself. Early popular work was QA in constrained environments. Now it can monitor, manage, and act in a web browser. That is still tighter than “here is your whole laptop.” A personal example she wants next is rescheduling Google Calendar meetings, which sounds trivial and is a coordination mess. The arc is constrained rooms toward open-ended ones.

What surprised her internally was not a demo. The model was a useful partner for the product team while they argued pricing and positioning. It was not only a better writer. It was a better thinker, offering alternatives spontaneously rather than polishing her draft. Opus models used to be the expensive ones. This one is strong and, she says, quite cheap. From the start they hoped efficiency gains would be passed through. They also shipped an effort parameter she thinks is underhyped: Sonnet 4.5-level intelligence at a fraction of the price. The industry, she argues, still confuses a token sticker with the end-to-end cost of finishing a job.

Educating the market on that cost story is homework for the labs. Builders tell her smaller models take longer on a task, or never finish, after spending more tokens than Opus would have. For much of the year Anthropic marketed Sonnet as the flagship. As the stack tiers, they will spend more marketing on teaching developers to pick a model by the job, not by the menu. Early access already had a clean exhibit. Shortcut, a spreadsheet agent, swapped in Opus 4.5 with no harness changes and heard something like a 20% accuracy jump from customers, intelligence they could hand straight through. Other testers stress-test on games; 3D generation got noticeably better, a folk way of seeing the intelligence ceiling. Some users said the model cleared an entire bug backlog.

On product-market fit, agentic coding is visibly here to stay. Enterprises keep showing up for Claude Code. Synchronous agents more generally have a reason to exist. Outside coding, the industry has not found the right harness and product for “agentic coding, but for web monitoring and personal agents.” A lot of use is still a chat box. She thinks people will be surprised by Opus 4.5, and that new products will be born from it: proactive agents that go do the work and surface the result instead of waiting for a prompt. Internally it felt like the model just got it without explicit instruction. Couple that with higher-quality context and memory that actually works, and the system can maintain and monitor, not only ship a deliverable. “It’s not really just about shipping an MVP.”

An endless line of people want the model better at their domain. Labs already diverge on what they prioritize. Customers ask for image generation and video. Anthropic, for the two and a half years she has been there, has been intentional about expanding intelligence instead. On the product side they also privilege business: data security, privacy, enterprise requirements over consumer toys. Feedback is a two-way flywheel. Shipping velocity means a pain point that misses 4.5 might still make Claude 5. They are investing in internal tools so Claude can help them spin up training environments and close the loop with less manual effort. They have too much they could do. They have to be choosy.

The podcast circuit had been pouring cold water. Andrej Karpathy told Dwarkesh Patel enterprise agents might still be a decade out. Ilya Sutskever argued the current paradigm only goes so far before a wall. Diane’s reply is that model improvement is not the smooth line a benchmark chart implies. It is jagged. On your personal eval a jump looks small; on someone else’s it looks huge. Customers such as Rakuten and Lovable say capability gains still turn into team productivity. Internally Anthropic talks less about AGI than about transformative AI. Every generation of Claude changes how Anthropic employees work. If you are not adopting it at that depth, it may not feel like advancement.

What is on the next board is longer-running intelligence: not a human delegating a slice of a website, but Claude taking open-ended responsibility — maintain it, refactor when it thinks it should, without so much handholding. Office work still has headroom. Computer use, if quality and adoption keep moving, could be as transformative as coding, because navigating a computer is how most people actually work. Long-running agents are not a use case by themselves. They need a user problem that fits, like an investor who wants portfolio adjustments as the market moves, not a one-shot briefing. The industry lacks long-horizon tasks that make quality easy to eval.

The closest specimens she tracks are Vending-Bench, in which Claude runs a vending-machine business, and Claude playing Pokémon. Completing the game is not the whole score. How many steps, and whether you brute-force, matters. A model that remembers Ash was in Pallet Town and already talked to an NPC is actually getting smarter. Intelligence is increasingly judgment and intuition, and how much time or effort it takes to reach an outcome. With Opus 4.5 they hit 80.9% on SWE-bench. Those evals, she says, are saturated. Terminal-bench is in the same neighborhood. When she shipped the tool-use API the previous year, accuracy was below 50%. Jacob’s favorite example is a model that upgraded a ticket by rewriting the fare class — a workaround that still got the job done. The next exams should be open-ended, with some quantifiable quality, and not a yes-or-no. Most of the world is not a true/false question, and biology does not have a SWE-bench equivalent.

Scaffolds have evolved with the models. In 2022 through 2024 a lot of them were training wheels to keep the model on distribution: do not do this, always do that, about twenty rules. This year the best scaffolds are augmentations for intelligence, and the craft is iteratively deleting the parts that no longer amplify. Claude Code is the example she lives with. Its scaffold is relatively lightweight. The tools are generic, bash rather than a bespoke zoo, because the point is to maximize the model’s autonomy while it works. What should remain valuable are generic tool sets and multi-agent orchestration, which became more viable this year: not one model, but a set amplifying context quality.

Builders want to know how much of today’s harness the next generation will obsolete. She cannot give a precise fraction. That uncertainty is why thinner harnesses matter; the less you wrapped, the cheaper the swap. They do see quality gains from iterating scaffolds per model. Users also drag the product. Once Claude is good at editing a document, she will dump a large bundle on it and ask for an alternative strategy. Scaffolds have to follow user behavior, not only model releases. People will push your product to be slightly more complex. Meeting that demand is the job.

Culture, Diane says, is the most authentic company she has been in, from day one until now: leaders walk the walk, talent density is the highest she has seen, people take radical ownership, kind but direct. Every three months the job changes because the company is a different company. Two years earlier she was still emailing prospective customers and standing up the first A/B tests. Now a third of her day is customer calls on emerging use cases; the rest is coaching PMs who sit with research counterparts and thinking about what is ahead.

The pivotal 2023 call was ignoring the most common request — an embedding model, because RAG was all the rage — and investing in agentic coding. That is user-led versus user-centric: what they are not even asking for. Nobody came down the door in 2023 demanding agentic coding. Shipping computer use as a beta API was the other bold one. It did not fully work. They thought the form factor was worth showing, did a lot of safety work, and still had to put it in the world to learn the edge cases. A fun, not quite pivotal, memory is Golden Gate Claude, wrapping interpretability research in a model obsessed with the bridge. The company was still under 500 people. Engineers, researchers, a PM, and a designer shipped from model to UI in less than a day, expecting a few hundred geeks. It went a bit viral. She ranks it among her proudest moments.

In the quickfire she says her timelines moved up. The building blocks for transformative long-running AI feel closer than they did at the start of the year. The leftover is more product overhang than missing capability: the right scaffolds to express what is already there. She is a barbell — large probability in the short term and in the long term. Sophisticated builders keep a library of prototypes that used to fail and retest them on every new model. Without a prototype you only ever wonder when drug discovery will work. Anthropic runs internal hackathons every three or four months so people can discover capabilities nobody asked for. The other habit is being willing to change the product experience when the weights jump, not just swap an API key.

Model taste, in her definition, is product sense applied to models: a continuously honed feel for capabilities, a willingness to get hands-on, and an understanding of what users are trying to do. What society under-discusses is the upside of safety. Alignment is not only “don’t do bad things.” Sycophancy — models that tell you what you want to hear — is an active, unsolved research problem. A well-aligned model is the opposite, an independent thinker, and independent thinkers are how you get breakthroughs. She asked Claude for a pricing take, gave two options, and it invented a strong third that pushed her thinking. A sycophantic Opus 4.5 would have agreed. Safety, if it works, amplifies the quality of intelligence. Anthropic, she notes at the end, still lets the work speak: the blog and the website, no cryptic launch tweets.