Sean Allen: Make software correctness part of the design

Tests are a means to an end, says Sean Allen: they can do remarkable things, but they arrive relatively late in the process. No set of tools gives a complete guarantee, so teams are also testing in production whether they acknowledge it or not.

Allen initially cared more about music than a technology career. After college he moved to New York, found a job fixing computers at Scholastic and learned Perl after a friend suggested a startup opportunity. The startup folded, but he found he enjoyed programming. As a teenager he had also tried to learn it from a Commodore 64 machine-language book, hoping to make games.

At Antithesis, Allen works on finding failures in software systems. He describes an internal red team that uses the product and pushes against its assumptions, giving the builders direct feedback on the problems their own users may encounter.

Allen values OCaml and related languages because their type systems let developers describe the possible shapes and states of data. The compiler can then flag cases a program has failed to handle before it runs, rather than relying on someone to remember every case in a test.

A sum type can say that a value is one of several alternatives. When code handles the alternatives, the compiler can warn if one is missing. Adding a new alternative can therefore reveal the places that need to change, instead of leaving those discoveries entirely to runtime.

Allen contrasts C with Ada: the Ada compiler rejected programs that he could persuade C to compile. His initial frustration changed as he understood that the rejected code could otherwise have produced unsafe memory access, crashes or other failures.

F# brought similar functional-programming ideas into the .NET world. The attraction for Allen is not a particular fashion in languages, but the ability to have the compiler help preserve the properties the program is supposed to have.

Pony combines a type system designed around safe concurrent work with the actor model. Instead of sharing arbitrary mutable state among threads, independent actors communicate through messages; the language makes a number of the concurrency decisions for the programmer.

Concurrency bugs are difficult because the order in which work happens can change from run to run. Allen values language-level guarantees that rule out classes of data races instead of asking developers to discover every bad interleaving after the fact.

The actor model is familiar from Erlang and Elixir. Allen says Pony integrates asynchronous programming with its type system and chooses a coherent set of mechanisms, rather than leaving developers to assemble every part of the concurrency model themselves.

The host asks why garbage collection has a reputation for being slow. Allen distinguishes freeing unused memory from the specific implementation people may know: stopping an entire program can damage tail latency, but that is not the only way to reclaim memory.

Even applications that manage memory themselves often reserve large blocks and subdivide them, using strategies such as arenas or slab allocators. They still need a rule for knowing when memory is no longer needed; memory management involves choices about who makes that decision and when.

As a useful simplification, Allen says to imagine each Pony actor having its own heap and taking responsibility for reclaiming its memory. Work can alternate with local collection rather than stopping the whole program. He presents the separate-heap description as an approximation, not an exact implementation account.

That approach can produce steadier latency, although the trade-off may be lower peak throughput than a system that postpones collection. The important distinction is between different collection strategies, not between garbage collection and no memory-management cost at all.

Allen defines correctness loosely as a program doing what its authors expect. He often talks about invariants: properties that should continue to hold. That requires the team to say what the software must do, including under conditions where it is not working normally.

Testing is useful as verification, but Allen argues that the first opportunity to improve correctness is earlier. A language that can express allowed states and transitions can make some invalid states impossible to express, narrowing what later tests need to investigate.

Allen likes test-driven development as a way to design APIs: writing a test first makes a developer use the interface. But an example-based test can be satisfied by an implementation that simply returns the hard-coded answer to that example.

In a paired exercise, Allen says he deliberately hard-coded answers whenever that was enough to pass the current test. His partners eventually had to vary or randomize inputs, moving toward property-based testing: checking a rule across many generated cases instead of a few chosen examples.

Earlier in his career Allen enjoyed dynamic languages and thought he did not need much help from a compiler. As systems and teams grew, he came to value even basic compiler feedback, such as identifying all the places affected when a class is removed.

For a larger system, Allen says the test workload includes the inputs, the operations applied and the checks used to decide whether the result is valid. A key-value store, for example, needs choices about reads, writes and deletes as well as key and value sizes.

Randomness alone does not guarantee a useful workload. If reads, writes and deletes are equally likely, a long run of writes that fills storage and triggers eviction or resharding may be so improbable that the test never reaches those behaviors.

A workload should deliberately drive the system toward situations its designers care about. The generator and the rule used to judge results both need thought: generating activity is not the same thing as knowing whether the observed behavior is correct.

Allen rejects the idea that quality is a gate at the end of development. Review, automated checks and staging environments can each add confidence, but passing them does not prove that the code will work under every condition it will meet.

The host describes a layered model in which different checks catch different problems. Allen agrees checks help but says none closes every hole: production can contain unusual data or combinations that the test generator never created. Everyone is learning something in production.

Teams that deliver reliably, in Allen’s experience, ask how a system can be known to work when they begin designing it. For a distributed database, that might mean checking a consensus design or exercising an early implementation, when mistakes are still comparatively cheap to fix.

The host challenges the idea with search ranking: reasonable people can disagree about which result should be first. Allen separates relevance from simpler properties, such as a query returning results and one user not receiving another user’s response.

Those properties can be tested under load and concurrent requests even when the best ranking is subjective. Markers associated with each query can help check that a server returns the right response to the right requester.

Supply-chain security concerns the components a system relies on, not only the code its own developers wrote. Allen says an attacker can target a weak dependency or build system; code executing in a continuous-integration environment may have access to secrets used to publish software.

The host lists possible mitigations: delaying package updates, removing dependencies or maintaining private forks. Each carries a cost; delaying an update can also delay a security fix, and if everyone waits, someone still has to discover the malicious update.

Allen says most commonly used languages give imported code broad capabilities similar to those of the application itself. A small package intended to manipulate text may also be able to open network connections or run other computations, which makes a large dependency tree hard to vet.

He points to object capabilities as a research direction: code should need an explicitly granted capability before it can perform an action such as opening a network connection. But a route into unrestricted native code can undermine the boundary, so the surrounding system matters too.

Allen connects security to processor side channels. Speculative execution can make processors faster, but information about work performed along speculative paths can sometimes leak. He says the confidential-computing work he did at Microsoft included a qualification for side-channel attacks.

Confidential computing aims to run software without requiring complete trust in the operators of the underlying environment. Allen describes the risk not just as a cloud provider acting maliciously, but also as an employee being compromised or bribed.

He describes the security and performance tension in speculative execution: disabling or mitigating features can have a cost, while new side channels continue to be found. Allen also uses processor behavior to make his own code faster, illustrating the trade-off he sees.

On technical debt, Allen stands by an older argument: not every shortcut must eventually be repaid. A product may be retired, the business may change, or an alternative technology may make the original limitation irrelevant.

He gives an example of database scaling constraints that were sidestepped by paying for a distributed database. The new system cost money and had unusual edge cases compared with a single-node setup; his team judged those trade-offs acceptable for its circumstances.

The host recalls friction between a mature Uber rides team and an early product team whose immediate priorities differed. Allen agrees that startups sometimes must move quickly, but says an expedient that can be removed is different from one that traps the team as other work accumulates around it.

Going slow to go fast means spending time on a plan, agreeing what the team is building, deciding how to test it and exploring unknowns before committing to a full implementation. Allen also includes stopping when exhausted: tired developers can spend their time finding mistakes instead of shipping useful work.

A common source of wasted effort is people using the same words to mean different things. Allen recalls arguments over “exactly once” in distributed systems: one side meant a message having an effect once, while the other meant the message literally arriving only once.

A few days spent writing down what a complicated system should do can expose disagreements while changes are still inexpensive. The same planning should ask how the system will be observed in production, including partial failures that leave it technically up while users have a poor experience.