Andrew Hunter on Performance Engineering for Low-Latency Systems

Ron Minsky introduces Andrew Hunter, a Jane Street market data engineer whose background is in performance engineering. Hunter says that, however many practical reasons he can give for his work, he finds understanding systems and making them faster addictive.

Minsky observes that expertise requires fascination with details of CPUs, interconnects and compilers. Hunter recalls studying operating systems by reading Linux's implementation of each topic, including virtual memory, and following the questions that arose instead of deciding they were outside the course's scope.

In graduate school, Hunter worked on a graph traversal system that reproduced specialized hardware in software. The interesting challenge was not just algorithms: it was understanding memory controllers, concurrent accesses, stalls and the number of operations that could be kept in flight.

Summer internships at Google showed Hunter that large-scale systems work could be deployed and used in production. He stresses the difference between turning in a university project and maintaining software after it meets real data: users return with unexpected behavior, and that feedback drives another cycle of improvement.

Hunter calls optimization at a hyperscaler 'easy mode' because the scale creates many valuable targets, not because fixes are easy. When enough machines run shared infrastructure, even a small reduction in the cost of allocation, serialization or compiler-generated code can pay for substantial engineering effort.

He cites the idea of a 'datacenter tax': diverse applications share infrastructure that consumes a significant fraction of their cycles. Improving that common layer can benefit many services at once, even though an application's own business logic may offer its biggest individual opportunity.

Minsky connects this to cost structure: hardware, power and queries make small fleet-wide improvements valuable. Hunter says he does not need to know whether the application serves search or cat videos when he is optimizing a component used by everyone.

Jane Street does not have the same scale, so an average CPU-cycle saving may not justify a project. Low-latency trading systems often poll for network packets in a tight loop, spending most of their time intentionally idle; a conventional profiler may mostly report that spinning.

Minsky qualifies the picture: Jane Street has many kinds of software, and this description applies especially to trading systems. They are designed to respond quickly and handle bursts rather than maximize utilization during quiet periods. Historical simulations, by contrast, are throughput workloads where faster processing lets researchers get answers sooner.

Small improvements can still accumulate. Minsky says a one-percent compiler improvement matters alongside other gains, while Hunter describes gradually removing slack from a system. Individual changes may be below the noise floor, so sometimes engineers must test them in isolation or rely on their understanding of the mechanism.

Hunter challenges the rule that every performance change must first be justified by a profile. In performance-sensitive code, efficient defaults can be sensible if they do not add disproportionate complexity or risk. Experience and 'mechanical sympathy' help an engineer recognize what the machine will actually do.

Mechanical sympathy, Hunter explains, is a learned intuition for the relationship between high-level code and physical execution. Languages expose different abstractions, but memory layout, loads, arithmetic and compiled branches have concrete costs. An expert develops an instinct for those consequences rather than reconstructing the entire model from scratch each time.

Hard-mode optimization often concerns latency at a particular important moment, not average resource use. A profiler can say where a program spends 20 percent of its time, but a trading engineer wants to know what happened while a specific market event was being analyzed or an order was sent.

Minsky notes that this puts the investigation close to business logic: the relevant path is the one that makes the decision. Hunter agrees that infrastructure can occasionally intrude, but the significant code is often the trading system's own decision-making path.

A sampling profiler periodically interrupts a program, records its instruction location and often its stack, and then lets it continue. Hunter likes Linux perf, which uses hardware facilities to collect those samples efficiently; a profile is essentially a collection of stacks that can be aggregated to show hotspots.

Samples need not be triggered only by elapsed CPU cycles. Hardware events such as cache misses or branch mispredictions can focus the profile on where those costly events occur. An allocation profile can similarly reveal where memory is being allocated, even though it measures a resource other than time.

Minsky says high-performance OCaml code often tries to bring heap allocations on hot paths close to zero. Hunter notes that allocations can also be significant in optimized C++ systems: even when some are necessary, a profile can reveal that the program allocates more than it needs.

Sampling trades detail for lower overhead, which creates tension when the question is what happened at one critical instant. Hunter recalls a system where deciding to send an order and physically sending it were each cheap, yet the order went out about 200 microseconds after the decision.

The cause was a low-priority work queue and an older API that assumed outgoing traffic could wait until an idle moment. Switching to an eager API fixed the delay. A profile of CPU cost could not reveal that the important problem was the time between two individually inexpensive operations.

Magic Trace, a tool built at Jane Street, provides a retrospective view of a short window of execution. Instead of an aggregate sample, it shows the sequence of function calls over time. It has more overhead, but the timeline directly exposes relationships that an ordinary profile can miss.

The speakers caution that profiling and tracing terminology is inconsistent. RPC tracing may capture the full chain of messages for a request; Memtrace samples allocations but follows the lifetime of each selected allocation. Capturing complete lifecycles for a sample of objects or requests can answer different questions from sampling isolated moments.

Magic Trace uses Intel Processor Trace, a hardware facility that records compressed execution information in a ring buffer. The buffer holds a short history, often a few milliseconds, which can be captured when an interesting event occurs. The approach relies on support in silicon and on perf's integration with it.

Minsky wishes the hardware capability came with more accessible tooling. Hunter agrees that performance analysis often demands an encyclopedia of obscure flags and processor behavior. Better defaults and interfaces can help people get useful answers without first becoming specialists in every tool.

Minsky has also seen people use Magic Trace for ordinary throughput problems: even a trace taken at an arbitrary moment can make execution easier to interpret. Hunter gives the example of a hotspot that might represent one slow call or a thousand unexpectedly repeated cheap calls; a timeline can make the loop obvious.

Visualization is a large part of Hunter's work. Flame graphs improved on earlier views by using width to show the share of sampled stacks, but they can hide a function reached through many different call paths: each occurrence looks small even if the combined cost is large.

Hunter describes pprof's graph view, with functions as nodes and weighted arrows connecting callers and callees. Multiple small paths can visibly converge on one function. Allocation is a common example: many different pieces of business logic may all call malloc.

Flipping a flame graph can help in some cases, but convergence in the middle of a stack remains awkward unless the analyst already knows which function to focus on. Minsky finds graph views less immediately intuitive, and Hunter agrees that visualization could still improve.

Tail latency matters when a system normally responds in five microseconds but sometimes takes a millisecond, particularly because the busy moments may be the moments when speed matters most. A trace can show an exceptional event such as a major garbage collection, but the tail is not always caused by anything exotic.

A burst may simply create a queue of ordinary work. If each packet takes one microsecond but the arrival rate effectively gives the system only 800 nanoseconds per packet, it falls behind. Batching can sometimes help; otherwise, the repeated operation itself may need to be faster.

Hunter would like to select a range in a trace and view its calls as an aggregated profile. Minsky distinguishes that wish for a profile visualization from the separate question of whether sampling is preferable to exhaustive tracing.

Sampling is useful for throughput problems such as historical research and machine-learning training, and hardware counters can target particular events. It is also cheaper: Hunter estimates that Intel Processor Trace can impose roughly five to fifteen percent overhead, so Jane Street turns it on selectively rather than everywhere all the time.

Hunter would still value a broad sampled view of trading systems excluding their idle spinning, so he could compare opportunities such as order sending and market-data parsing across the fleet.

Asked about moving from C++ to OCaml, the speakers reject the assumption that C++ automatically makes software fast. Architecture, process boundaries, data structures and the work a system chooses to do can dominate language choice.

Hunter says the first job is often to avoid unnecessary work: do not call the function, consume the data source or perform the operation if the system can be structured without it. A two-line change may deliver a large gain, but only after weeks of investigation have established why it will work.

He separates OCaml's performance challenges into three groups. Less mature code generation can produce extra branches or register spills, which is irritating but often not the main constraint. Garbage collection also places requirements on memory initialization and references that can impose costs in some situations.

The challenge Hunter considers most important is representation. OCaml data structures often use boxed values and pointers where a C++ representation might put data inline. These choices support clean and safe programming, but they can increase memory footprint and make physical layout harder to control.

Jane Street uses domain-specific languages to describe packet layouts and generate usable, type-safe interfaces over compact low-level memory representations. The same approach can represent an in-memory market book in a flat, packed form, though the result is less pleasant to use than ordinary OCaml and is reserved for places where it matters.

The team also uses a zero-allocation dialect of OCaml and language features that give more control over layout and garbage-collector interaction. C stubs provide another escape hatch: Hunter says calls across that boundary can be cheap, while noting his rough comparison with Java's JNI.

Work on the compiler aims to offer more control over layout in ordinary OCaml types. Until then, teams can compensate through architecture, preprocessing and distributing work across processes. Minsky notes the practical cost: when an optimization is harder to express, people will not always do it.

Hunter says discipline includes leaving slow code alone when it does not matter. Performance specialists may find an unnecessary allocation personally offensive, but there are more important problems competing for their time.

Minsky asks why, if latency is paramount, a system should use a CPU at all. He estimates about 400 nanoseconds in each direction across PCI Express in an ordinary packet-processing path, while an FPGA attached to a network card can turn around a packet in under 100 nanoseconds.

Hunter says hardware changes architectural choices, but hardware development is difficult and complex trading logic may be impractical to implement there. A system can offload a small, simple, speed-critical task to hardware while feeding it from a more flexible software system.

The right latency target depends on the task. Hunter describes work spanning sub-100-nanosecond trades to transactions involving email and a next-day response. One system aimed to reduce tail latency from 50 milliseconds to one; simpler, more speed-sensitive systems might target 20, 10 or five microseconds.

Human-facing latency matters too. A frozen editor or a slow price display disrupts work, and a research system that gives feedback on a trading idea in a minute enables a different pace of experimentation from one that takes a day.

Minsky observes that faster feedback makes creative iteration possible. Hunter invokes John Boyd's OODA loop—observe, orient, decide, act—and compares it with accelerated spirit aging: even if the process is controversial, shorter feedback cycles allow more experiments in the same time.

The same principle applies to engineering experiments. A realistic test bed that answers whether a change helped in ten minutes lets an engineer try and revise ideas much faster than waiting several days for production data. Minsky says the habits of attention to detail transfer across environments, including browser performance.

Hunter jokes that performance work requires finding supposedly boring details fascinating. Minsky agrees that the appeal is hard to explain to people outside the field, and the conversation closes with Hunter's story of his father's concern about whether computer science would offer a reliable career.