Doug Patti on State Machine Replication at Jane Street

Ron Minsky introduces Doug Patti, a Jane Street developer working on distributed systems. Patti previously built web frameworks and APIs in Node.js and Ruby; his work at Jane Street has involved different technologies and systems often organized around a trading day.

Patti describes Concord, built to support direct trading relationships with clients. Jane Street provides both liquidity and infrastructure resembling an exchange. That makes availability, fast responses and a regulatory audit trail important responsibilities.

A client connects and sends an order; the system attempts to match it with an order from a trading desk. On a match, copies of the execution go to the client, the desk and a regulatory reporting agency. This is a simplified account of the end-to-end service.

A client disconnected after sending an order must be able to discover whether it was received and filled. The system must preserve that information even if a machine crashes, and be able to move processes to another machine in minutes rather than waiting hours for repair.

Patti explains that FIX messages carry sequence numbers. After reconnecting, the parties can identify and exchange missed messages. Minsky contrasts this with TCP: TCP orders data within a connection, but a lost connection or crashed host requires another layer to recover the session’s state.

The system must also cope with unusually busy trading days, sometimes orders of magnitude above normal. In addition to client orders, regulatory needs bring in market data from public exchanges, creating particularly hot paths that must be handled efficiently.

Concord uses replicated state machines. Here a state machine is a module that processes one message at a time, updates encapsulated state and may produce an action. Each message is processed synchronously and deterministically, without blocking on a network or disk request.

Given the same code and the same ordered messages, a restarted process can replay the transaction log from the beginning of the day and reconstruct its prior state. Minsky emphasizes both ingredients: deterministic applications and a reliable common ordering of messages.

To disseminate the log, Concord uses UDP multicast. A packet sent to a multicast address is copied by network switches to subscribers, efficiently distributing one message to many processes. UDP can lose or reorder packets, so Concord adds its own reliability layer.

Sequence numbers reveal gaps: a process receiving 45, 47 and 48 can ask a retransmission service for 46. Patti says this works even when drops are common, including when a process cannot keep up. Retransmitters listen to the sequenced stream, store it and can request missing packets from peers or the sequencer.

Any process can propose a message via multicast. One sequencer receives these proposals, assigns a sequence number and timestamp, and republishes them to listeners. That single point turns incoming messages into one canonical order.

If Concord’s sequencer fails, new messages stop. A backup can be promoted manually; Patti says the team prefers a short pause to inspect the system and establish confidence before resuming trading. The interview does not explore consensus algorithms in detail.

A single sequencer is a potential bottleneck, but Patti says the design has worked for Concord’s workload. Limits include a 10-gigabit network, packet-processing capacity and disk writes. He reports a trading-day log exceeding 400 GB, a baseline above 100,000 messages per second and several times that on busier days.

Minsky argues that scalability needs to be judged against the actual finite workload. Compact encoding, especially for frequent market data, helps fit the system within its limits. Patti reports approximately 8–12 microseconds round trip from an application’s proposal through the sequencer and back.

Concord is also an application framework. Deterministic, synchronous state machines make tests powerful: a test can inject messages in a chosen order, pause at specific points and reproduce race conditions that would be difficult to trigger in production.

The framework inverts control. An application processes messages and records what it wants to send, but waits to be asked for the next proposal instead of sending whenever it wishes. This forces pending intentions into explicit state rather than hiding them in a suspended control flow, making them inspectable in tests.

Minsky relates this style to event-driven programming with select. Threads can make sequential code easier to express, but they also hide state in stacks and program counters and require coordination. Concord processes each message completely before the next, exposing state in ordinary data structures.

Patti says state machines can be composed from smaller state machines. A process that is too slow can be split, or related processes combined. Because separate processes see the same ordered messages and use deterministic code, they can include the same component and independently arrive at the same view without directly querying one another.

There are costs. Most processes must examine every message, even if they quickly discard those meant for another subsystem. They must keep up with the stream, tune data structures and allocations, and sometimes split work across processes to gain CPU capacity.

Concord’s large shared message type also makes versioning brittle. If a new message format is written to the log and the team rolls back to code that cannot read it, recovery fails. A digest causes incompatible applications to stop rather than misinterpret the stream, and releases need careful coordination.

Patti and Minsky distinguish this particular implementation from a broader constraint: any persisted message must remain readable, and a broadcast stream cannot negotiate a protocol separately with each consumer. Supporting multiple versions is possible, but the log and recovery model impose real compatibility requirements.

Each Concord instance has its own sequencer, retransmitters, configuration and hardware. Patti says this overhead makes deploying a new instance expensive to manage, even if worthwhile for a large application. It is a poor fit for a small service.

Aria is intended as a more general shared service. Like Concord it combines transport with an inverted-control application framework, but the Aria team manages the core infrastructure. A user runs an application on a machine and connects through a proxy instead of provisioning an entire Concord instance.

The proxy filters messages so a small application need not process a high-volume stream it does not use. Core infrastructure still handles the full traffic, but it treats message payloads as opaque bytes; clients own the payload formats and versioning. Core scaling is largely a matter of moving and storing bytes.

Aria organizes data in hierarchical topics, and clients can subscribe to a subtree. Crucially, messages across topics retain one canonical ordering. Patti gives the example of relating an order to the market data at that moment: after a crash, replay must preserve their relative order.

Minsky contrasts this with his understanding of Kafka, where ordering is guaranteed within a shard rather than across topics. When an application combines several streams, a different interleaving on replay can change decisions. A shared total order allows deterministic replication across those topics.

Aria lets users publish different protocol versions on separate topics so older clients can continue consuming an older version. Minsky points out a caveat: even if each version’s messages maintain their own order, they can interleave differently with unrelated messages, so different versions do not necessarily yield identical cross-topic behavior.

Patti adds another versioning issue: the state-machine code itself can change. A new binary replaying identical messages may interpret them differently and reach a different state. Aria users must manage configuration and changes to semantics carefully; a common log alone does not guarantee equivalent replicas running different code.

Aria restricts who may write to topic subtrees. Read permissions are harder because multicast packets can be received by anyone able to listen on the address. At the time of the discussion, read controls were being added as a best-effort, trust-based check in the client library, not a hard multicast security boundary.

Minsky notes that read restrictions also protect abstractions: users should not accidentally depend on internal topics that a team expects to change. Patti says Aria stores some of its own session metadata on Aria and has the same concern. Sequencing configuration changes in the log also gives consumers a consistent point at which new configuration applies.

Aria was in production in several regions with a deliberately limited number of teams. Patti says new clients bring new requirements and design decisions have lasting effects. Some needed features were still missing, including rate limiting.

Without rate limiting, a faulty client could repeatedly inject messages as fast as possible. Topic filtering prevents unrelated subscribers from receiving those messages, but sequencers, proxies and other shared infrastructure still handle them; disk usage is also a concern because every byte is stored for replay.

Snapshots are a harder open problem. At the time, a restarting application had to replay its subscribed topics from the start of the day. Extending a session to a week would make a late restart potentially replay hundreds of gigabytes; a snapshot could supply state at a point in time and allow replay from there.

Minsky compares this with exchange market-data feeds, which provide a snapshot as well as a reliable stream and gap recovery. But the right snapshot depends on the computation. A current order book can recover a count of open orders, for example, but cannot recover the total number of orders seen during the day.

Patti describes giving users the ability to build snapshot servers tailored to their applications and combine state from multiple topics. Minsky notes that providers can still offer reusable snapshot components for common cases. Patti sees this as composability: clients can reuse, extend or compact components to fit their needs.

Early Aria users largely came from client-facing technology, including systems for high-touch workflows that mix automation and human decisions. Organizational proximity made coordination easier while the core was evolving. Minsky notes that the programming model takes time to learn, even if its benefits make that effort worthwhile.

Patti wants to document recurring Concord patterns and provide examples or a boot camp with immediate test feedback. One common pattern is an injector that translates external data into stream messages. A subtle bug can cause it to propose the same message forever if its explicit state is not updated after sequencing.

Minsky describes the application as a problem solver: it proposes a message while an external fact is missing from the stream and stops when the problem has been resolved, regardless of which process resolved it. Patti says abstractions can prevent common errors; a pipe-backed injector, for example, provides a familiar interface that sequences each item once.

Looking ahead, Patti hopes more teams will use Aria and expose higher-bandwidth needs. The team expects to improve fairness and scale shared infrastructure when needed, balancing implementation ahead of demand against time spent building other features.