Sam Newman on Resilient Distributed Systems

Gergely Orosz interviews author and architect Sam Newman about microservices, resilient distributed systems and AI-assisted development. Newman discusses his own book and experience; the episode has sponsors.

Newman recounts his time at Thoughtworks and the influence of extreme programming and test-driven development. He describes the practices as part of a longer evolution in software delivery.

The conversation traces the emergence of microservices. Newman emphasizes independent deployment and team autonomy, while cautioning that distributed services introduce operational costs and should not be the default for every problem.

They discuss whether specifications may become more prominent as agents write code. Newman treats the relationship among intent, implementation and verification as an evolving question rather than declaring code irrelevant.

Newman introduces his work on resilience engineering and explains why ideas from other safety-critical fields can broaden software teams’ thinking about failure and recovery.

He reduces distributed-systems challenges to three constraints: communication takes time, a dependency can be unavailable, and resources are finite. Timeouts, failures and saturation can interact, such as when retries consume scarce connections.

Observability is the ability to understand system behavior from its outputs. In a distributed system, teams must connect signals across components and consider user-facing outcomes, not just individual service health.

Resilience is a matter of degree and involves cost. A stripped-down standby service can protect essential functions, but the right investment depends on the consequences of failure and the business context.

Idempotency helps with retries: repeating an operation should have the same intended effect as applying it once. Newman uses an elevator call button as an illustration and contrasts it with a toggle.

A thundering herd occurs when demand overwhelms available capacity. Recovery traffic, cache misses or synchronized retries can create self-inflicted overload; capacity and retry behavior must be designed together.

The team’s choice to fail open or closed depends on the business and the harm of each error. Newman argues that formal concepts are useful but cannot replace understanding the consequences for customers.

He discusses resilience as more than absorbing known disruptions: added robustness can itself increase complexity and create new failure modes. Teams also need to adapt and learn.

Newman sees possible uses for AI in correlation and incident analysis, while expressing concerns about model limitations. His claims about how models reason are his interpretation, not a settled scientific conclusion.

He uses “cognitive debt” to describe a risk that a team loses its shared understanding when it delegates too much thinking. He recommends keeping people engaged with architectural boundaries and reviewing generated work.

Modular architecture can give AI tools bounded areas to work in while people focus on the relationships between modules. Newman frames this as a way to experiment without surrendering the system’s mental model.

The interview closes with learning resources and Newman’s book. He discloses personal relationships when recommending some authors.