Ron Minsky on the Saga of Multicore OCaml at Jane Street

Ron Minsky describes Jane Street's experience bringing the multicore OCaml garbage collector into production. The talk is both about technical problems and about technology transfer: the work needed to turn ideas from research into infrastructure that an established organization can adopt.

For many years OCaml lacked shared-memory parallelism, and Jane Street's ecosystem grew around that assumption. A 2020 ICFP paper described a multicore collector intended to be high performance, maintainable and easy to adopt, with one runtime for both sequential and multicore workloads.

The design aimed to preserve sequential throughput, low pause times and the foreign-function interface so existing C extensions would not all need rewriting. The paper reported roughly a 3% sequential performance loss, similar memory use and pause times, and good parallel speedups on its benchmarks.

Minsky cautions that garbage-collector performance is a curve rather than one number: tuning changes the balance between collection work and memory use. Upstreaming took about nine years from the original multicore project, and even after the collector landed in 2022, Jane Street needed roughly another two and a half years to use it internally.

Some work was expected: improvements made to the old collector during multicore development, including prefetching, had to be brought forward. The team also made both runtimes available with the same language version, enabling direct comparisons. Once it tried real workloads, the performance was unacceptable and roughly a year and a half went into diagnosing and fixing regressions.

The old runtime 4 collector is sequential and generational. The generational hypothesis says most objects die young. New objects are allocated very cheaply in a minor heap by advancing a pointer; when it fills, the collector finds survivors and promotes them to the major heap, while the dead objects can be discarded together.

A write barrier records pointers from the major heap back into the minor heap. Without it, a minor collection might need to scan the entire major heap to discover whether young objects are still reachable. In a mostly functional program these backward pointers arise through mutation, so bookkeeping can be attached to writes.

The major collector marks reachable objects starting from roots such as stack and register values, then sweeps memory to reclaim unmarked objects. Minsky's animation illustrates marked and unmarked colors and how sweeping resets survivors for the next cycle while freeing unreachable objects.

The old collector is incremental: marking and sweeping happen in slices rather than one long pause. This trades some throughput for responsiveness. A snapshot-at-the-beginning rule means a cycle need not reclaim new objects that become unreachable after the snapshot, helping ensure the mark phase eventually finishes.

Runtime 5 preserves much of the old design but adds parallelism and synchronization points. The compiler inserts safe points so domains can stop and coordinate. Each core has its own minor heap to keep allocation cheap and reduce contention, while one major heap is shared.

When a minor heap fills, domains synchronize for a minor collection. A domain that finishes early can do major-collection work while waiting for the others. The shared major collector combines marking and sweeping so the two can proceed without an extra synchronization between separate phases.

The old collector's colors move between marked, unmarked and free. Runtime 5 adds a garbage state: marking changes unmarked to marked, while sweeping changes garbage to free, allowing the operations to work on disjoint states. At cycle end, domains synchronize and rotate the interpretation of the color bits rather than walking the heap to rewrite them.

Real tests found some programs 10–20% slower and some using 10–20% more memory, with worse regressions uncovered later. Minsky lists several causes: collector pacing, transparent huge pages, slow legacy thread switching, stack checks associated with effects, and code that had not yet received as much optimization as the old runtime.

He concentrates on pacing: how much collector work happens relative to application work. Rather than a fixed heap-size ceiling, OCaml exposes a space-overhead target, expressing how much extra space the collector may use. That choice suits a fleet of programs with different and changing memory requirements.

For ordinary heap allocations, the original analysis assumes a steady state in which a long-running program eventually collects roughly what it allocates. This yields a constant amount of marking and sweeping work per word promoted to the major heap. Minor-heap allocations are paced separately.

Off-heap allocations pose another problem. An OCaml heap object may hold a handle to a large external allocation, such as a bigstring or memory managed by a foreign library. The on-heap collector must do enough work to discover dead handles and allow the external resources to be reclaimed.

The old off-heap scheme accumulated extra-heap work using a formula proportional to the size of the external allocation and inversely proportional to current heap size. When the accumulated resource value reached a threshold, it triggered extra collection. Minsky says the heuristic seemed plausible but behaved poorly.

He frames the distinction using control theory. On-heap pacing is open loop: its work rate is based on allocations rather than a quantity the controller itself changes. The off-heap formula is closed loop because heap size is affected by collection; feedback can make the system difficult to tune and can cause oscillation.

Benchmarks show ordinary allocation workloads following the requested space-overhead target more closely than an off-heap-heavy workload, whose behavior is nearly flat. Minsky says aggressive collection of external allocations was a serious longstanding problem for Jane Street programs and required awkward application-level adjustments.

Runtime 5 retained the basic on-heap approach but fixed several off-heap implementation bugs. For example, runtime 4 could count an off-heap handle allocated on the minor heap even when it died before promotion, making pacing overly aggressive. Other bugs pushed in different directions, and applications had adapted to their combined behavior.

After those fixes, one off-heap benchmark improved, but ordinary heap workloads could use substantially more memory. The team found that the unified mark-and-sweep design was less aggressive at reclaiming floating garbage: objects that are dead but cannot yet be collected under the current snapshot and phase ordering.

Minsky compares the timing of a new allocation in the two designs. In his simplified analysis, objects allocated during the sweeping part of runtime 4 take about one cycle on average to become reclaimable; in the merged runtime 5 cycle that case can take about two. The average across the cases increases from roughly 1.25 to 1.75 cycles.

A mark-delay patch adds another synchronization and recovers reclamation behavior closer to the old collector. Experiments showed that the synchronization feared by the original design was cheaper in practice than expected. It improved ordinary heap behavior, but the more aggressive collection made the off-heap workload worse again.

The team first tried changing the off-heap denominator to include both on-heap and off-heap memory. That helped some pathological cases where the ordinary heap was tiny, but the closed-loop scheme still suffered oscillation and tuning problems. They returned to first principles rather than continuing to patch the heuristic.

They redid the steady-state analysis with both kinds of allocation in the same equations. Heap size canceled out, leaving a constant amount of collection work per allocated word without reading the current heap size. On-heap allocation requires an extra unit of sweeping work because its own memory must be swept; the off-heap allocation itself does not.

The resulting pacing curves were substantially closer to the desired behavior and easier to reason about, Minsky says. He also mentions ongoing experiments for startup, when a program may allocate mostly long-lived data and aggressive collection is unlikely to recover much. Those ideas were still being explored at the time of the talk.

Minsky draws lessons from the migration. Benchmarking is hard: the original evaluation leaned toward convenient, relatively small programs and did not cover enough behaviors to expose the problems in large production applications. He believes making both runtimes available under the same language earlier would have enabled more comparable tests.

Debugging was also difficult because regressions masked one another. The team sometimes built a polished solution to the largest apparent problem before establishing how important it really was. Minsky says quick, even imperfect prototypes and small targeted benchmarks would have helped test hypotheses faster.

Sometimes incremental repair is not enough: the pacing work benefited from returning to the underlying equations. Backwards compatibility also includes accidental behavior. A bug can become load-bearing when applications are tuned around it, so fixing it can disrupt production even if the new behavior is more principled.

Minsky then discusses a second stage of the parallelism challenge. A runtime can run shared-memory code in parallel, but programmers still need a usable model. Work described in a 2025 paper on data-race freedom adds type-level guarantees intended to rule out races, including annotations for functions that can move to another core.

Propagating annotations through libraries is one task, but fixing races in old code is harder. The type discipline exposes mutable state and tricks that developers may have forgotten were present. Minsky says the team found real problems even in a codebase associated with functional programming.

Other difficulties include choosing the right granularity and amount of parallelism. Tasks that are too small lose time to overhead; tasks that are too large leave cores idle. The team is exploring scheduling ideas, including heartbeat scheduling, to adapt the amount of parallel work dynamically.

Parallel programs are also difficult to profile: even shared-memory code is affected by cache-line traffic and contention. The team wants tools that explain poor performance and abstractions that combine parallelism and concurrency, along with teaching materials so developers can use the new capabilities.

Minsky announces the planned public availability of OxCaml, the name for Jane Street's OCaml work with extensions. At the time of the talk it was the firm's production compiler, but he cautions outside users that experimental extensions could change incompatibly. He invites researchers and enthusiasts to experiment rather than treating it as a stable external production platform.

In questions, an audience member asks why a garbage-collected language was chosen and how it compares with Rust. Minsky notes that Rust did not exist when Jane Street chose OCaml. He argues that OxCaml's combination of garbage collection and optional controls such as local allocation and ownership-related modes suits the firm's mix of workloads.

He contrasts paying the cost of explicit memory discipline everywhere with applying more precise control where needed. For extremely latency-sensitive loops, the firm aims to avoid heap allocation in the critical path; at the same time, garbage collection can be convenient for other applications and shared structures.

Asked about rollout, Minsky says runtime 5 had general availability internally but had not been adopted by every application; some regressions remained. After testing, teams were told to watch performance and could roll back. An earlier rollout had gone badly and was reversed, so he does not portray the process as foolproof.

An audience member asks about contention in the shared major heap. Minsky distinguishes collector contention from application-level cache-line sharing and says predicting the latter remains hard. Static resource analysis is limited, in his view, so programmers still need to understand and measure their parallel workloads.

On formal verification, Minsky says Jane Street uses relatively little full theorem proving in production. Hardware design can make some static analysis and solver techniques practical; C bindings and compiler optimizations also benefit from targeted tools. Elsewhere the firm relies heavily on testing and, for some important risk systems, independent reference implementations.

Asked about predicting allocation lifetimes at startup, he describes ongoing work using the proportion of minor-heap objects promoted as a heuristic. A high promotion rate can suggest many allocations will live longer, allowing pacing to be reduced. The goal is to avoid forcing users to tune collection manually during startup.

A question contrasts parallel and concurrent collectors for low-latency finance. Minsky says compatibility and the broader range of workloads shaped this design; some major collection can occur while another domain does application work. For the tightest trading loops, he says, the firm's approach is to avoid allocation rather than rely on a collector to provide deterministic latency.

He adds that a trading firm solves far more than packet turnaround. Different tasks require general-purpose systems, while extremely fast packet processing may use specialized hardware. Optimizing the collector for one narrow picture of trading would miss much of the firm's computing work.

Asked why the company persisted after the regressions, Minsky points to a very large existing OCaml codebase. Introducing another language carries its own cost, even when justified as Python was for machine learning. He argues that improving an established ecosystem is often cheaper than replacing the software built on it.

Another audience member questions whether researchers should have compared both runtimes earlier. Minsky praises the original work and notes that some people worked on both stages. He considers one runtime a sound long-term maintenance goal but believes a temporary period with both would have made evaluation and migration easier; most fixes had since been upstreamed.

Asked why Jane Street gives technical talks, Minsky corrects a suggestion that he is the company's CTO, saying it has no such role. He says sharing technical work lets people understand the problems the firm solves and helps recruiting. The migration work described was done internally, including by some people who had contributed to the original research.

Internal development can allow rapid iteration on ambitious language changes with feedback from a large, engaged user base. Minsky contrasts that with the justified caution of an upstream community, which must maintain public APIs and avoid leaving users dependent on mistakes.

On whether the firm serves its technology rather than its business, Minsky acknowledges the cost of using a less popular language, including reduced access to external ecosystem tools. He says early trading applications were relatively self-contained and that OCaml proved useful, fast and safe for the actual business rather than being adopted out of purity.

Asked for the biggest immediate multicore benefit, Minsky says adoption is still early. Some already-parallel systems showed substantial speedups when redesigned around shared-memory scheduling; he also expects simpler tasks to gain parallelism without building a distributed system or serializing data between processes.

His main concern is the language's complexity budget: more power also adds conceptual weight. The team is aiming for a pay-as-you-go design and thinking about how to teach the new model, but he says it is hard to know in advance whether the balance is right.

A question about organizing the migration prompts Minsky to describe a focused effort by the roughly 20-person language team, with GC and runtime experts running experiments and using profiling tools. He highlights Magic Trace and Intel Processor Trace as useful for examining exact execution over short intervals in addition to sampled profiles.

Finally, asked whether a new firm should choose an obscure language to gain control over it, Minsky cautions that building a language ecosystem takes time and substantial effort. Jane Street did not gain control early. He says a niche language's ability to attract talented people was an important benefit, but hesitates to prescribe the firm's path to a small new company.