Evals Are the New PRDs: Inside Anthropic's Research Product Team

Penn joined in 2023 as the first technical product manager, when Anthropic had five product engineers and a single engineer covering the whole API business. She says the culture was already strong and that the company has largely kept the same mission-driven, startup-like energy. The early years, in her telling, were about working out how the technology could bring value to users and society.

One moment she calls a hidden inflection point was Golden Gate Claude. After publishing early interpretability research in 2024, the team found a model feature tied to the Golden Gate Bridge and dialed it up, so Claude worked the bridge into every answer, even a spaghetti recipe. Engineering, product, design and research spun up the experience on the Claude site within about 24 hours.

Penn admits it probably reached only around 2,000 people. Still, it showed the company could ship new kinds of experiences that showcased its research in a way that felt authentic. Engineers and designers donated their time, and she holds it up as an example of how bottoms-up the early days were.

Penn points to Opus 3, which launched in early March 2024, as a milestone when the company was still under 200 people. It was about proving Anthropic could build a frontier model and giving users a reason to pick Claude. She recalls research leads, inference, fine-tuning and pre-training teams working through December from their parents' homes, and says that trust still helps product and research work together.

Coding was another turning point. In 2023 nobody put Anthropic and coding in the same sentence, and GPT-4 was used for code as one of many use cases. Penn noticed people using models for long-form code rather than autocomplete, and training Opus 3 to be better at it was a relatively small change that helped the company stand out with early developers.

She calls Opus 4.5, released during winter break a year later, another big moment, in part because it came with a vehicle in Claude Code. The team's saying is that you need frontier products to have frontier models. The model reached a level where it could run things end to end as an agent, and Claude Code's adoption accelerated because of it.

Asked what it is like to be inside the exponential, Penn compares it to the internet moving from novelty to something everyone uses. She says adaptability matters most, because evals, red teaming and new prototypes can only tell you so much about when a specific capability will land. Teams need to make better decisions when new information arrives instead of sticking to the same plan.

She also stresses first-principles thinking about what to build next. Sometimes the model can suddenly do something the team expected to plan for months later, and that is how products like Cowork and skills reached users. She adds that teams feel the exponential at different speeds, so leaders need to bring the rest of the organization along.

Penn then goes back to the original scaling-law papers. Loss falls smoothly as you add compute and data, but those papers also show emerging capabilities that jump discontinuously, such as a model going from being unable to do arithmetic to doing it reliably. That unpredictability is why you need evals, and it makes safety harder. It also creates what she calls product and user overhang: a lot of value in today's models that no one has explored yet.

Lenny raises the idea that spending around $100,000 a year in tokens means living the way people will in 2028. Penn reframes it: token spend is the input, and experimentation is the output that matters. She still notes that the best prototypers inside Anthropic spend a great deal of time with each new model, because it is hard to build a strategy without touching technology that moves this quickly.

She describes an early internal Slack channel where nearly the whole company tested early versions of Claude. One person would try an idea, others would try variations, and within about ten requests a new use case might appear. Her point is that discovery does not have to be an individual sport.

On Anthropic Labs, she says the thesis is to pursue discontinuous bets outside the core road map and ask what the 10x, 100x or 1000x version looks like. Examples include Claude Code, skills, Claude Design and MCP. This year the approach is to hold strong opinions about the area but weak ones about the exact prototype, and to revisit a failed idea a model generation or two later. Pods stay small, sometimes one engineer, and the team selects for people who want zero-to-one work.

Penn explains that researchers do more than tweak data and algorithms. There is a founder-like layer of vision about where the technology goes, such as getting Claude to use a computer, and a day-to-day loop of improving the model people have today. Her team is embedded in both, especially in areas with direct user impact such as vision, computer use, agentic coding, tool use and test-time compute.

A big part of the job is converting raw feedback into something researchers can use. A complaint like "Claude hallucinated" is too vague, so the team traces what happened. If the model should have called a tool, that is a tool-use failure; if it read the right document but pulled the wrong facts, that is a search and synthesis problem. Once the problem is sized, they build evals to show whether it improved.

Asked what makes a top researcher, Penn names strong first-principles thinkers who are passionate about their area, hold a bold vision, and stay close to the details such as training runs and underlying data. She says the best are also very ambitious. In her own work she asks what changes about user behavior if a future model like Claude 8 arrives, and whether today's product is forward compatible.

Lenny notes that Mythos was restricted and heavily scrutinized, and asks how that changes the way Anthropic operates. Penn leaves policy and export controls to others and focuses on the product side. As frontier models become more capable, she says, safeguards, red teaming and pre-release testing have to evolve quickly.

She says that before the Fable models there were not strong fallback experiences, and the team has since built fallback systems so users still get a great response from Opus 4. Her team is working on what it now calls the model safeguards package and expects to improve it in the coming weeks and months. When Lenny suggests restricted access could give labs an unfair advantage, she says the goal is to keep general-purpose models as inclusive as possible and to reduce that effect.

Penn says her team has not changed its hiring loop in three years. The first trait she looks for is first-principles thinking rather than pattern matching from consumer or B2B SaaS. Product managers on her team demonstrate it by figuring out what will deliver user value, not by producing the usual artifact, which is why "evals are the new PRDs."

The process starts with understanding the pain point, but how you access it differs now. In the past you might watch a user walk through a flow; here, she says, you have to sweat the tokens as much as the pixels. Her team reads transcripts of failed trajectories to name the pattern, such as hallucination or overconfidence, and then builds an eval that covers both failure cases and cases where the model should not fail.

Her example is JSON schemas in the Claude 2 era. Users said Claude was bad at following instructions, and digging in showed roughly 80% of those complaints were really about malformed JSON. The team collected 30 to 40 failing examples as the eval set, and the eval is now run on every version. Lenny calls it test-driven development for PMs, and Penn agrees that you can't improve what you can't measure.

She says PRDs still have a place. For every model there is one, aimed less at researchers than at product surfaces, engineering, legal and safety, as a shared source of truth. They are also useful for ambiguous bets like computer use, where there are no defined pain points yet and the vision section helps explore value for a first user group.

Penn feels strongly that managers of teams working with this technology must be hands-on and actually ship with it. Even experienced PMs she hires follow the same onboarding as early-career ones, built around users, consented user feedback and talking to customers. She carves out time to own one or two work streams whenever there is a new model, so she keeps her sense of how quickly models are improving.

For people who find AI tiring or annoying, she suggests pairing with someone who is excited about a use case you care about rather than hunting for the perfect one alone. She has touched practically every version of the research models across about 20 production versions of Claude, and says part of the joy comes from watching others discover use cases. Lenny adds that his reader survey found happiness came from going deep on a couple of things instead of many.

Penn agrees that depth is the answer to overwhelm. She finds it hard to keep pace with the number of products out there, so her approach is to go deep in one or two. For product people, she says, this becomes an exercise in prioritizing time and energy.

Penn uses Claude to be a better manager. She built a skill drawing on the book *Crucial Conversations* that helps her prepare for difficult conversations and checks whether she is going into the right level of detail. She sees it as individualized coaching that helps her think through reactions and build trust faster, even when she ends up not using its suggestions.

On the risk of over-reliance, she says thinking and writing are tied together for her. When a situation needs her own judgment, she forms her own point of view first and then works through it with Claude. For things like monthly business reviews, she wants the writing delegated fully so she acts as a reviewer and verifier, because the thinking is worth more than the drafting.

She links this to Anthropic's alignment and safety work. A model that only agrees with you is not helpful, and she wants a co-worker who pushes back when her ideas are unformed. She has used a research version of Opus to weigh pricing decisions for the next Claude. Her argument is that proactivity includes knowing when to push back, and that a thinking partner should leave you with better ideas, not merely ten percent better ones.

She questions whether knowing who wrote something is the right goal. For a monthly business review she would be happy for Claude to write everything end to end, and says what matters more is who verifies and signs off. Where an idea comes from should still be clear, depending on the purpose of the writing.

Lenny asks why AI writing is still so recognizable. Penn says Anthropic is actively working to make Claude write better and that tone and character are a priority. She describes the technology as jagged-edged: as models became more agentic, other areas such as writing became the rough edges that now need focused training.

On where human brains stay valuable, Penn says hard-earned judgment, persistence and proactivity matter because AI systems have not accumulated as much experience. She adds that domains like biology and life sciences are only at the foot of the exponential, which is why Anthropic invests in areas like Claude for science. For her young children she wants the same traits she grew up with, curiosity, persistence and belief in their own inner voice.

Penn notes that in 2024 Anthropic shipped four model series in the whole year and has done more than that in the second quarter of this year alone. She says nobody can make critical, fast decisions sustainably alone. Her team stays up the night before a launch to help the owner review the blog post and improve the demos, and she describes a kind of mind-melding the team calls entering the hive mind.

That, she says, is what lets people take PTO without returning to triple the work. She also looks for low-ego, team-oriented people when hiring, those who care about contributing to Anthropic rather than building their own org. Lenny points out that working with agents instead of other humans can make engineering lonelier, which makes the human team more important.

Closing out, Penn takes on the community question of whether PMs are still needed when models are so capable. She argues the core of the role is being user-centric, going into the details of what users are trying to accomplish, and surfacing that in an actionable form, and she thinks more of that is needed. Lenny agrees that building is getting easier and the hard part is deciding what to build and whether it is good.