Brian Nigito on Multicast and the Architecture of Markets

Ron Minsky introduces Brian Nigito, whose career spans exchanges and trading firms. They will discuss the infrastructure behind financial markets, particularly networking and IP multicast, and how the constraints differ from those of the open internet and cloud systems.

Nigito explains that a securities exchange is a formal marketplace operating under regulatory rules. Its participants include firms trading for themselves or clients, clearing institutions and market-data providers. Other exchanges also consume its information because their quotations interact with one another under the applicable rules.

Minsky notes that physical trading floors have given way to highly electronic exchanges. Nigito traces electronic communication back decades, while observing that rules and practices gradually shifted toward continuously accessible electronic quotations.

At the message level, an order expresses interest in buying or selling a security up to a quantity and at no worse than a price. It may execute immediately, rest in the order book or be canceled. When compatible interests meet, execution messages report the trade and its terms to both sides.

Minsky describes the order book as the core data structure, surrounded by many consumers needing different information. Traders need private reports about their own activity and public, anonymized market data. Reliability matters because missing or inconsistent trade records can have serious consequences.

Nigito adds fairness: competing participants study how quickly each receives information and seek every permitted advantage. Delivering a stream to many people at nearly the same time is a systems problem as well as a market-design problem.

At Island, an early approach used similar machines and sent each participant through a comparable amount of work. It was fair in some respects but inefficient: thousands of machines were mostly idle. Sharing a machine among clients exposed a new problem when messages were sent serially in connection order.

Participants rushed to establish the first connection. Round-robin ordering reduced that advantage, but clients then opened many connections and took the fastest copy. Giving every connection a dedicated machine would waste resources, so the architecture needed a different distribution mechanism.

TCP was a reasonable choice over the open internet, and early systems delivered some market data that way. As firms colocated equipment near exchange infrastructure on controlled private networks, UDP and multicast became more practical: a sender could transmit one copy and let network hardware replicate it.

Nigito sketches the layers. Basic packet delivery is best effort; TCP builds reliable, ordered point-to-point streams using sequencing, retransmission and other mechanisms. A separate TCP connection requires a separate stream for each receiver. UDP is lighter and can address a multicast group that multiple machines join.

Minsky asks why replication in the network helps. Nigito says switches can repeat bits to many ports quickly and regularly, while a general-purpose server writing individual copies incurs repeated work. Multicast trees distribute replication across the network, so copying can happen near the recipients.

Early deployments sometimes used broadcast, repeating everything everywhere. Nigito recalls kernel developers noting that TCP makes trade-offs for network health, which are important on the internet but can create latency variation. A controlled, low-loss network makes UDP a more viable choice for a shared feed.

The speakers distinguish two flows. Order entry, cancellations and private responses generally use TCP connections; shared anonymized market data is generally multicast. Colocation and cross-connects put trading firms close to exchange networks and provide a much more controlled environment than the public internet.

They caution that the picture is especially representative of equities. Currency and fixed-income markets can have bilateral prices and request-for-quote workflows. Different products, participants and histories mean that not every market converges on the same model.

Nigito recalls an attempt to use UDP for order entry. Adoption was limited when firms already needed to manage TCP connections for many other exchanges. By contrast, multicast offered such a large advantage for a common public feed that it became broadly attractive.

Minsky observes that internet video did not become the multicast use case once imagined: viewers often watch different things at different times. Market participants, however, need the same live exchange feed. Multicast also has an internal role beyond publishing market data to customers.

Inside an exchange, the order of events is fundamental. Nigito describes keeping the central sequencing work small and distributing the ordered stream to other applications, which can generate execution reports, clearing information and anonymized public data.

Minsky frames this as a transaction-processing architecture: a sequencer chooses an order for requests and publishes them, while downstream components perform the specific computations and reporting. Nigito notes that some exchanges separate sequencing from matching; others organize the work differently.

Multicast alone is not a reliable ordered stream. Sequence numbers let receivers detect missing or out-of-order messages; separate recovery services can retransmit gaps. This combines fast dissemination with application-specific mechanisms for restoring the sequence.

Nigito says general-purpose reliable multicast protocols often allowed a slow receiver to slow publication. At large scale, someone is almost always having trouble. The systems he worked on instead used sequencing and specialized recovery so the main feed could continue for participants keeping up.

Domain knowledge offers another shortcut: a receiver starting late may be given a compact snapshot of current state rather than replaying every event since the beginning. Similarly, knowing that a queued order and cancellation cancel one another can suggest optimizations unavailable to a generic byte stream.

Minsky summarizes the trade-off: exchanges value fast and fair dissemination to the great majority, while lagging recipients recover separately. Nigito says a major design simplification comes from identifying which problems an application does not need to solve.

Fragmented markets make speed and predictable response relevant to routing decisions. A participant that waits on one venue may miss opportunities elsewhere. Nigito argues that faster and more consistent responses can help participants quote and route more effectively.

Minsky gives an example: a firm may be willing to buy a quantity at any one of several venues, but not at all of them simultaneously. Without a single cross-market instruction expressing that preference, it may post orders in several places and cancel the others after a fill. Speed changes the risks and trade-offs in that strategy.

Minsky recalls being surprised that a Nasdaq architecture could process transactions through a single machine and core running a largely ordinary Java program. The design seemed both impossibly fast and strangely restrictive. Nigito emphasizes that the work was done by teams, and that concentrating a critical sequence can simplify the system.

Rather than assuming work must be distributed, one can ask whether the entire critical path fits on one core when carefully optimized. Hardware capabilities and theoretical bounds provide a baseline. Not every exchange uses the same architecture, but the approach can remove substantial coordination complexity.

The implementation techniques have evolved. Interrupt-driven networking trades CPU overhead against latency by batching work before notifying a core. With many cores available, a program can instead poll a network card and dedicate a core to noticing data as soon as it arrives.

A conventional socket interface can involve copying data into application-owned buffers. A lower-level design can let the network device place data in memory, have the application poll for it, and release it for reuse after processing. Reducing copies and making per-event service times predictable helps smooth queues.

Nigito recalls a design principle from Island: if a component is so complicated it cannot be rewritten correctly in a weekend, question it. Minsky says their current systems do not all meet that standard, but both value the underlying question: can this be simpler while still doing its job?

Nigito calls the hardware-aware mindset mechanical sympathy: understand the physical machine well enough to design near its capabilities. A back-of-the-envelope bound helps determine whether a proposed system can work and how far an implementation is from the practical limit.

Developing that intuition takes curiosity, experiments and a grounding in computer architecture. Modern processors anticipate and optimize work, so simple benchmarks can mislead. The useful skill is choosing measurements that actually correlate with the question being investigated and keeping current with the hardware.

Minsky distinguishes scalable from efficient: a general distributed design is not always simpler or faster than a concrete solution sized for the actual workload. Nigito adds that plentiful hardware improvements historically allowed software layers to grow; slower improvement in some areas may make efficiency more important.

Nigito says custom hardware is not equally relevant to every Jane Street business. For extreme message rates, programmable network hardware can filter by information deeper in a packet, such as a security identifier, and help consolidate data from many venues when network and processing capacity become constraints.

The speakers compare network devices. A traditional store-and-forward switch receives and examines a packet; a cut-through switch begins forwarding as bytes arrive; a layer-one crosspoint can electrically replicate signals according to fixed configuration. The latter is highly specialized but useful for some market-data distribution.

Nigito offers illustrative, hardware-dependent figures: roughly 7–10 microseconds for an older store-and-forward switch, 300–500 nanoseconds for cut-through, and 3–5 nanoseconds for a crosspoint itself. Wires and other system costs must be added; these are not universal end-to-end guarantees.

Very fast networks change the boundary between one computer and several. Nigito estimates inter-core communication around 100 nanoseconds and optimized network-card paths at hundreds of nanoseconds, depending on the machines. A minimally processed frame may travel through user space and back out in under a microsecond in the systems he describes.

Physical links matter too. Serialization delay is the time required to place the packet’s bits on a wire at a given line rate. Different input and output speeds may require buffering; a packet cannot be emitted at a faster rate before enough data is available.

At 10 Gb/s, the speakers use roughly one nanosecond per byte as an approximation. Sending a 1,000-byte message individually to 600 recipients consumes about a microsecond per copy, so later recipients wait much longer. Sending one copy and replicating it in hardware improves both distribution latency and the sender’s available throughput.

InfiniBand contributed ideas including hardware reliability, zero-copy interfaces and remote direct memory access. RDMA extends direct memory access across machines, allowing data to be placed in remote memory without the receiving CPU copying every byte, but it requires different APIs.

Nigito recalls deployments where hardware flow control and oversubscribed receivers could push back on senders, slowing market data in ways that were hard to diagnose. Best-effort Ethernet with explicit recovery can be easier to reason about in some workloads. Reliability and flow control introduce trade-offs as well as benefits.

The conversation turns to RoCE, an Ethernet-based approach to RDMA, and the broader choice between specialization and commodity technology. Nigito recalls dedicated telecommunications hardware being displaced by software running on increasingly powerful commodity processors.

Commodity hardware has a large ecosystem improving its price and performance. Yet specialization can again be worthwhile where processor and memory trends or a particular workload justify it. Minsky notes that even FPGA-based custom systems build on programmable hardware produced at scale by major vendors.

Looking ahead from the time of the conversation, Nigito expects more standardized user-space polling and zero-copy I/O, including similar ideas appearing in Linux io_uring. He also expects higher network rates to become more relevant, though a faster link is not automatically lower latency.

Higher signaling rates can bring additional fixed latency, including error-correction costs, depending on the hardware and link. For sufficiently large messages, reduced serialization time may compensate. The speakers stress that the trade-off depends on the actual devices, workload and market requirements.