Sebastian Funk on a Fast, Safe and Maintainable Packet Processor

I want to share techniques Jane Street has used to build high-performance, low-latency applications close to machine capabilities. This is a survey of work by several people, not a claim that I am an expert in every detail. Speed is not the firm’s sole value, but increasing data volumes and shrinking time windows make performance necessary.

I will not concentrate on algorithm selection, hardware choice, parallelism or a language contest, although each matters in real systems. The constraint for this case study is a single-core packet processor written in OCaml. The goal is to show that this can work in OCaml, not that everyone must choose it for performance-sensitive code.

Packet processing means receiving network packets, doing nontrivial but not necessarily highly complex work, and sending data onward. This is a specialized case where speed takes priority; for many other systems, including those at Jane Street, maximizing performance above all else would be the wrong tradeoff.

The motivating application is market-data distribution. An exchange maintains an order book containing offers to buy and sell a security. Participants submit orders, and the exchange broadcasts updates and trades; the market-data stream reports activity without identifying the participants involved.

Trading systems use those updates to make automated decisions, while risk, monitoring and historical systems also depend on an accurate feed. A slow distribution system can make a trading system act on stale information. The problem is to handle the incoming firehose and distribute it efficiently to internal clients.

Nasdaq produces roughly a billion market-data updates a day, with millions around the 4 p.m. close. There are many other exchanges and asset classes. To understand the requirements, I examine how a strictly sequential processor would queue messages over a ten-second window at different fixed processing times.

At 750 nanoseconds per message, one sample window reaches a maximum queue of 22 messages. Across the close, queue growth becomes nonlinear as processing slows. At ten microseconds per message, the queue can reach roughly a million messages, leaving the system processing the start of the burst while later updates pile up.

Falling behind can continue to hurt after the burst ends: even when new traffic is quiet, a backlog delays subsequent messages. Looking at maximum message delay supports the same conclusion for this analysis: below roughly 750 nanoseconds is workable, while over a microsecond is unacceptable. This gives us a target rather than a vague desire to be fast.

Speed is not the only requirement. Exchanges use different and frequently changing protocols, and new exchanges or regulations introduce more variation. We want exchange-specific complexity at the edges, shared code in the middle, and a design that permits small, safe changes without rewriting the whole system.

An audience member asks whether the model assumed constant processing time; it did. Another asks whether messages can be filtered. Clients may subscribe to subsets, but some component near the input generally has to handle the entire feed before distributing it.

OCaml combines a strong static type system and functional tools such as higher-order functions with imperative features. Its runtime distinguishes immediate values, often integers, from boxed heap values. It uses generational, incremental garbage collection that pauses program execution during collection, and it can call C through a low-overhead foreign-function interface.

Consider a function that maps over a list twice. It creates copies of the list, and a closure that captures a multiplier may allocate too. Floating-point values can also require boxed representation. For ordinary applications this code is fine, but on a very latency-sensitive path it is not always obvious where temporary allocations occur.

The allocation itself is often cheap: in the common OCaml case it can be only a few machine instructions. Garbage collection creates pauses and worsens tail latency, but that is not the whole story. Allocation and collection can evict useful cache lines, so a stream of short-lived objects can also worsen median performance.

A zero-allocation approach avoids allocating on the critical path while allowing allocations at startup, on errors and elsewhere. The code often uses flat representations, reusable I/O buffers and synchronous processing. Explicit pools or preallocation are alternatives when objects are needed but garbage-collector activity on the hot path is undesirable.

The foreign-function interface can support memory managed outside the OCaml garbage collector, with careful layout allowing the runtime to work with those values. Funk characterizes this as low-level and potentially awkward, but emphasizes that only a small part of the program needs such treatment. Elsewhere, the language’s type system and memory-safety benefits remain useful.

A function call has obvious costs such as register movement or instruction-cache effects, although processor optimizations often make those less significant than expected. Two larger concerns in this setting are allocating a closure and limiting the compiler’s ability to optimize across call boundaries.

Inlining substitutes a function body at its call site. In the example, it eliminates a closure and exposes constant values that the compiler can simplify. But inlining everything can make a program slower, so the hard part is deciding when it helps; Jane Street’s compiler work includes developing those heuristics.

An option type expresses whether a value may be absent and forces the caller to handle that case. For example, a market book may have no current price. Matching on `Some` and `None` makes the missing case explicit, rather than relying on a developer to remember a null check.

The downside in the example is that a normal option representation allocates even when the price itself is an immediate integer. Funk’s benchmark shows four bytes allocated per operation. At millions of messages per second, this can trigger frequent collections with a modest minor heap.

An immediate option can reserve one integer value to mean absence and use the other values directly, avoiding allocation. But a manual unchecked extraction gives back the safety problem: reviewers must verify that every use checked for absence. Eliminating a box is not enough if the abstraction becomes unsafe.

The solution shown uses a preprocessor. It expands the source into a branch containing normal option matching for the type checker and another branch containing the efficient immediate representation. Type checking happens before dead-code elimination, so missing cases are reported while the code that runs avoids the allocation.

The second example reads market messages with identifiers and a list of book entries, then totals the size on one side of the book. A straightforward representation allocates a complete record and eagerly parses fields such as timestamps even if this particular computation never uses them.

A faster approach reads directly from an I/O buffer and accesses only the required offsets. Hand-written offsets are difficult to understand and maintain when the protocol changes. Jane Street instead uses a small DSL and a code generator, called protogen, to produce those accessors from a message-layout specification.

In a laptop benchmark, the generated efficient version processes the example in about 700 nanoseconds, versus roughly five microseconds for the eager implementation, and avoids allocation. The important point is not merely an eightfold improvement: the earlier queueing analysis tells us why five microseconds is unacceptable for this particular workload.

Even optimized application code can be slow in production. The operating-system networking stack may dominate, and cold-cache behavior can make a lightly loaded system surprisingly slow. Network topology and core placement also matter; these deployment questions are important but outside the main case study.

Define what fast enough means before optimizing, and understand the language, runtime, machine and measurement tools. It is easy to benchmark the wrong thing and draw confident but false conclusions. Start with a minimal system that meets the target, add features incrementally and measure after each change.

These optimizations can be fragile across compiler or runtime changes, so continuous benchmarks and regression checks matter. The result, Funk argues, is that a high-level, relatively safe language can meet sub-microsecond targets for this case without giving up every useful abstraction.

In questions, Funk explains that synchronous processing permits a buffer to be reused safely once a message is fully processed. It also requires the processor to keep up with the line rate: a slow step slows everything. Asynchronous processing requires extra care because another operation may still be using mutable state.

A participant suggests fan-out. Funk agrees that parallel distribution can be useful later, but an exchange may provide one firehose that an initial component must process. The ability to handle the full rate on one core is therefore an architectural option even if the rest of the system fans out.

Asked about generated representations across processes, he says the team has experimented with shared memory and device-driver techniques, although he does not claim detailed knowledge of every implementation. A later question about variable message layouts leads him to acknowledge that richer type-system techniques are needed beyond his simplified product-type example.

The examples shown use the ordinary operating-system path, but the team has also experimented with bypassing parts of the networking stack. FPGAs can provide still lower latency for specialized tasks; Funk says the firm uses them in some places, while this talk focuses on what a general single-core software approach can achieve.

The queue study can be reproduced by subscribing to a real feed and simulating processors with different service times, although Funk does not offer a universal tool for every system. He also acknowledges the frustration of using preprocessing to work around runtime representation choices, which are difficult to change for compatibility and garbage-collector reasons.

Why use OCaml rather than another language without a garbage collector? Funk says a common language throughout the workflow has substantial value: developers know it, use it daily and have tools built around it. The team initially considered alternatives if necessary, but found it could meet the goal in OCaml for this case.

Inlining decisions are largely empirical: benchmark the alternatives and derive heuristics from function size or hot-path annotations. Funk says the compiler can also be instructed to inline at a call site and report an error if it cannot. The team uses that guarantee on some critical paths.