Most Productivity Metrics Are a Lie: Nicole Forsgren on Measuring Developer Experience in the AI Era

Forsgren defines developer experience (DevEx) as what it is like to build software day to day: the friction, workflows and support developers face. She says that when DevEx is poor, good processes and tools will not help. Output alone is a weak lens, because high-toil or high-friction ways of reaching it lead to burnout, and high cognitive load leaves no brain space for innovative solutions.

She describes DevEx through three reinforcing ideas: flow state, cognitive load and feedback loops. Lenny, a former engineer, worries that agents and generated code make flow harder. Forsgren admits that the field is only a few years in and that interruptions are now frequent as people prompt, review and integrate rather than type for long stretches.

She has also seen senior engineers build toolchains that keep them in flow at a higher level. They describe the architecture, stack and workflow up front, have agents work on pieces in parallel, tell them to use appropriate APIs and conventions, and then review the result. She says the output is closer to production code than vibe coding because of the systematic setup, and Lenny summarises it as planning more up front.

Asked how teams get AI productivity measurement wrong, Forsgren says lines of code were always a bad metric, but AI blows it out of the water. Agents are verbose by nature, so more code can mean more complexity and technical debt. It becomes more relevant, though, if you can separate AI-written code from human code, since you can then ask about survivability, quality and how much AI-generated code is feeding back into training and fine-tuning.

On DORA, she explains that the four key metrics (deployment frequency, lead time, MTTR and change fail rate) still assess the speed and stability of the pipeline. But they are prescriptive and must be used for that purpose. AI introduces feedback loops earlier, even in the middle of the pipeline, so applying existing metrics blindly would miss important changes in how people work.

SPACE, by contrast, is a framework and does not dictate metrics, which frustrates some people but makes it adaptable. It still covers satisfaction, performance, activity, communication and collaboration, and efficiency and flow. Forsgren suggests adding trust, since LLMs are nondeterministic and teams must evaluate hallucinations, reliability and style rather than accept output as they would a successful compile.

Lenny recalls that in their first conversation Forsgren predicted trust would be a major AI issue and that more time would go to reviewing code than writing it. She says it will be interesting to see how this changes the structure of work, given that attention now has to focus at certain moments and is broken up differently.

Citing Gloria Mark's work on attention, she notes that people get about four hours of good deep work a day, usually needing blocks of at least two hours to get into flow. Writing code with AI is interrupt-driven, so one open question is whether a 4-hour block is still useful and whether a 45-minute one now can be, because the machine helps with re-entry through context reminders and system diagrams. She stresses this is purely speculative and an area ripe for research.

Lenny describes engineers as becoming managers of junior AI engineers, able to unblock agents even in a short window and then jump into code with context restored. Forsgren agrees this is an interesting direction. She makes the business case for DevEx as a way to speed up all software creation, problem solving and customer experimentation, as prototypes that once took a while can now be tested in hours.

Forsgren's top advice for this week is to talk to people and listen, starting with listening rather than tools and automation. Teams often build what is easy to automate or what they personally struggled with. She suggests asking developers to walk through yesterday, what was delightful, where they were frustrated and where friction slowed them. A handful of conversations often reveals low-lift fixes or an unnecessarily slow process.

Her example is a company with a painful process tied to an old mainframe that everyone hated because it caused delays and would require replatforming. Someone had to print a document, walk it down three or four flights for approval and have another person walk it back up. They did not replatform anything. They changed the process to send an email.

Most companies have process steps that exist only because they always have, and small companies can be too ad hoc and benefit from a lightweight process. For business leaders, she says the job is to provide structure and support: communicate priorities and why this matters, and celebrate wins, since an isolated side project struggles to gain momentum. Technical changes to build, test and feedback systems matter, especially with AI, but are necessary and not sufficient, and need not be the starting point.

Asked how a founder can tell whether a team could move faster, Forsgren says most teams can, though gains have diminishing returns and not all speed is good given cognitive load. Signs of friction include constant broken builds, flaky tests, overly long processes, slow requests for new systems or environments, and difficulty switching tasks. A particularly good smell is when people do not move to other parts of the org for unclear, non-political reasons. She knew companies where switching orgs meant paying the same tax as a new hire.

She stresses that the question is faster for what. Teams can ship trash faster every day, and strategy and good decisions about what to ship, what to experiment with and in what order are the core piece. Lenny notes this is where PMs come in. Without those pieces, she says, it is garbage in, garbage out.

AI can also help with strategy, messaging, targets of experimentation and market sizing, but teams still need an aligned plan or two or three alternatives to test. Prototyping and customer-facing A/B tests that once took months can take a day or two, provided infrastructure exists. She advises checking with data science on instrumentation and analysis so a big test does not disrupt users, breach privacy or security, or produce unusable data.

On AI gains, Forsgren says they are real but there are no great measures yet. The best candidate is velocity through the whole system, from idea to a feature or experiment, although this is hard to attribute to a particular tool. She saw one study across companies where regular users of AI coding environments produced more code, and the engineers' own increase was double what the agent generated. She also mentions AI unblocking starts, writing unit tests and documentation, and notes that agents depend on good documentation and comments.

Her new book *Frictionless*, co-written with the founder of DX, is aimed at technology leaders, anyone kicking off a DevEx program and PMs. Lenny notes that DX was sold to Atlassian for a billion dollars, which he takes as evidence of how much companies value measuring developer productivity. The seven steps are: start the journey with a listening tour, get a quick win, use data to optimize, decide strategy and priority, sell your strategy, drive change at your scale, and evaluate progress and show value. Readers can join the book at any step.

The book also recommends practices for resourcing, change management, making technology sustainable and bringing a product lens to DevEx. There is a workbook of almost 100 pages, with the book due by the end of the year at developerexperiencebook.com. Forsgren says the term developer experience is deliberate: it is about unlocking value and giving engineers freedom to solve problems creatively, not just lines of code or PR counts.

For a pilot team, Forsgren suggests a couple of engineers plus a PM, PgM or TPM to handle communications, which she says matter greatly. Early on, look for small paper cuts that help developers feel the benefit, such as cleaning up a test suite. With a top-down remit, quick wins can be global, such as changing a cumbersome organization-wide process or cleaning up provisioning environments.

She has seen impact in the hundreds of thousands of dollars for smaller companies and billions for large ones. Results tend to follow a J curve: early quick wins look big, then comes a dip once the obvious fixes are done and teams must build infrastructure and telemetry, after which benefits compound. Quantifying value means speaking to each audience: developers care about time savings, reduced toil and focus, while leadership cares about revenue, time to value, velocity and cost.

One example is cleaning up a test and build suite, which gives developers fewer reruns and less toil, and gives the business cloud cost savings and recovered capacity. She says correlation with business outcomes such as time to value and market share is usually the best you can do, though some correlations are compelling.

On measuring the impact of AI tools, Forsgren says it depends on what the leadership chain cares about. If they talk about market share, focus on speed from feature to production, customer or experiment. If they talk about margin, look for money saved and recovered headcount cost, or lower vendor spend. She suggests reusing leadership's own language, whether productivity, velocity, transformation or disruption.

For velocity, she recommends as broad a span as possible, such as idea to customer, and tracking how long it takes with improved AI use and less friction. She says to be open about attribution: say that AI tools and a DevEx effort were rolled out together and both contributed, since AI tools without DevEx improvements would likely have helped much less.

If you are starting from nothing, she says, run a survey after talking to people, since you may lack instrumentation or have metrics designed for other purposes. Ask about satisfaction, the biggest barriers and let people choose just three, ask how often each affects them, and add open text. Avoid compound questions, such as asking if builds and tests were slow or complicated, and use experts or an AI assistant to refine questions. She dislikes happiness surveys because too much contributes to happiness, and prefers satisfaction with tools, processes and the team.

Asked what else matters, Forsgren says to bring a product mindset to DevEx improvements and to the metrics themselves. That means identifying the problem for a set of users, building MVPs and experiments, having a strategy and a go-to-market function with comms, collecting continuous feedback and eventually sunsetting things.

This matters more now, she says, because AI is changing things so rapidly that it is worth taking half a beat to ask whether a metric used for the last ten years is still important or should be sunset. A metric that no longer drives the decisions and actions you need is a candidate to retire.