Spotify lets agents change code at scale by investing in tests

Spotify engineering leader Niklas Gustavsson recalls being skeptical when Boris Cherny predicted that developers would stop using an IDE within months. Two months later, Gustavsson says, he had stopped using one himself; the way he worked changed more quickly than at any other point in his roughly 30-year career.

Gustavsson trained as a molecular biologist and learned more programming while working with genome-sequencing data during his doctorate. A planned year away from biology became a long career in software. Spotify later began experimenting with language models to automate code changes, initially with difficulty.

The early experiments did not solve every maintenance problem, but using models and judges gradually suggested a direction. For Gustavsson personally, a later Claude Opus release was a turning point: instead of treating the model as autocomplete and manually editing the last portion of its code, he could give it real problems without doing that final IDE pass himself.

Today Gustavsson describes a relatively simple setup with multiple terminal sessions and agents working in the background. He typically has five to ten tabs, matching terminals and worktrees. Spotify has a few large monorepositories alongside thousands of smaller repositories, so he opens temporary sessions when he needs the latter.

He initially worried about using agents on a very large monorepository because earlier tools struggled with indexing. Spotify’s backend monorepo contains more than 20 million lines of code, but he says Claude has worked well there, including by finding other code in the repository to use as inspiration.

Cherny asks how Spotify came to build Honk, its background-agent tool. Gustavsson says that five or six years earlier the company found its codebase growing about seven times faster than its engineering team, increasing the amount of code each engineer would eventually have to maintain.

Spotify wanted to keep shipping ideas rather than spend all its time on maintenance. Previously, migrations such as a Java upgrade or API change meant sending instructions to hundreds of teams, each of which made similar changes manually across thousands of components. A migration could take months, and the company could complete only around ten a year.

Fleet-management infrastructure automated many of those changes and merged millions of pull requests. But deterministic scripts hit a complexity ceiling: even changing an API became difficult when code could call it in many ways. Each script accumulated thousands of lines handling edge cases.

Spotify tried early language models on the problem, but simply presenting code and asking for a one-shot edit did not work well. Models improved and the team learned to break the task apart and use a model as a judge of the result. Those iterations eventually became Honk, though its original version was not built on Claude.

Honk began by scheduling and orchestrating code changes across repositories. Engineers soon wanted to invoke it for other tasks, including from Slack, so it became a more general tool. Gustavsson says the released version called V2 is really the result of many more internal iterations.

Gustavsson says early versions of Honk benefited substantially from a separate judge: he recalls pull-request success rising from roughly 20–30% to 80%. The company later removed the judge because it found that the newer model and agent infrastructure were good enough without it.

The current architecture is relatively simple, he says: the Claude Agent SDK runs in a Kubernetes pod with access to tools. Earlier versions used a predefined allowlist; V2 lets users add their own tools. One important capability is running verification builds on Linux and macOS, the latter needed for iOS development.

Cherny asks whether verification can go beyond building to operating an iOS simulator. Gustavsson says it can: Spotify has integrated the simulator and Claude to help turn Figma designs into interface implementations and to port features from iOS apps to TV apps.

Cherny argues that verification is especially important when an agent decomposes a task and works without a person at each step. Gustavsson agrees and says Spotify strengthened test automation as part of its fleet-management work.

Spotify divides its codebase into thousands of components, each owned and operated by a particular team. Previously those teams could review every change to their own code, sometimes compensating for weaker tests. Automated pull requests changed that expectation: teams might never see a change before it was auto-merged, so tests had to improve.

Gustavsson says the earlier investment now helps Spotify verify agent-generated work too. Cherny argues that automating quality practices makes faster work possible. Gustavsson adds that speed and quality must both receive attention: Spotify’s quality metrics have remained neutral while speed increased, but the testing investment was necessary and reliability work must continue.

Spotify makes roughly 4,500 production deployments a day, Gustavsson says, creating many opportunities for something to go wrong. Frequent deployment is a long-standing effort to get feedback on ideas quickly. Some ideas can reach production in about an hour, though others require extensive exploration and do not ship that fast.

The company has about 2,900 engineers. Asked how it measures return on AI tools, Gustavsson reports more than 75% improvement in pull-request frequency that Spotify attributes to AI, and says approximately 73% of pull requests are AI-authored. Those output measures are not the end of the measurement problem.

Spotify wants to connect pull requests and deployments to planned work items, then to experiments, rollouts, user value and revenue. Gustavsson says the company is still building those connections. As adoption matures, he expects more precise return-on-investment estimates to become important, including the cost of producing an improvement.

For other engineering leaders, Gustavsson recommends not neglecting foundational investments such as test automation and standardization. Spotify made code, tools and frameworks more consistent to help people, and he says the same consistency helps agents find examples without encountering ten different ways of doing the same thing.

Gustavsson says he had worried that delegating coding might take away the mental challenge he enjoys, including in competitive programming. With several agents working in the background, he found that solving problems still mattered to him even if the way he solved them changed. He emphasizes that the transition will feel different for different people.

He says the tools let him contribute in unfamiliar codebases that previously would have taken days or weeks to understand. Cherny describes a similar shift away from implementation time toward thinking about what comes next, talking with customers and building prototypes.

Spotify has invested in prototyping for engineers and non-engineers. Gustavsson says employees can express an idea in natural language and have Claude implement an end-to-end prototype using the company’s mobile apps and backend. An internal app store lets colleagues share and try these prototypes.

Gustavsson says a working prototype can sometimes be ready to share in an hour or two, using real application infrastructure and data. He says people across the company, including a co-CEO and other senior executives, have built prototypes; an idea can be tried in a day instead of waiting weeks or months for an available team.