Mark Doss on Production Engineering for High-Stakes Trading

I’m Mark Doss. I’ve spent much of my nearly ten years at Jane Street in what we now call production engineering. I want to explain the challenges that arise when operating software in a trading environment, especially when things go wrong.

Production engineering overlaps with SRE and DevOps. Our software runs on computers that fail, in data centers with cooling problems, over networks whose cables can be cut, and against exchanges with their own issues. We also deploy bugs. The question is how to manage all of that technically and operationally.

A simple model of trading starts with a human or automated strategy receiving market data: recent orders and trades from an exchange. The strategy decides what to do and sends orders through an order-entry port, or order engine, connected to the exchange. The exchange acknowledges or rejects orders and reports fills.

After a trade, the firm must track its position and feed that information back to the strategy. It must also report trades to regulators, clearing firms and other parties. A simple order contains a side, quantity, instrument, price and currency, though real orders can have more complex conventions.

The first important feature of this environment is that every order matters. In a typical web service, one slow or failed request may be inconvenient. In trading, getting the side, quantity, instrument or price wrong on even one order can be catastrophic.

Consider reversing the quantity and price: instead of selling one share for about 610,000 yen, a trader submits 610,000 shares at one yen. This was not merely hypothetical. In 2005 a Mizuho trader made that error, and the exchange did not accept the attempted cancellation. Doss says the firm lost 27 billion yen and recovered only part of its requested losses in litigation.

Compliance matters too. A regulator may require every order to be reported; a 99.99% success rate can still leave thousands of missing reports. Automated trading is particularly daunting because software with access to money operates in a tight loop, and a small error repeated many times can become a large one.

Adverse selection makes mistakes more dangerous: as soon as you offer a bad trade, other market participants have an incentive to take the other side quickly. Knight Capital’s 2012 incident illustrates how quickly a technical failure can become existential.

Knight had old code that would lose money if a particular configuration flag were sent, but the flag had not been used. The firm repurposed it for a new feature and began sending it. Seven of eight instances upgraded; one failed, and its notification emails were overlooked.

When losses began at the open, engineers identified the system and rolled back, assuming the new deployment was responsible. The unupgraded instance was actually the source, so rolling back the other instances accelerated losses. Doss says roughly half a billion dollars was lost in 45 minutes; the details deserve a full postmortem of their own.

The next feature is timing. Trading volume has large spikes at the open and close, and scheduled events such as Federal Reserve announcements can matter greatly. Unexpected news can also drive volatility. The same failure has a very different impact depending on when it occurs.

External dependencies follow a clock too. A firm might download metadata at 7:30 a.m., load it into one system, then another, alongside other feeds. There may be only a couple of hours to get everything ready before the open, so a delay forces people to scramble.

Our users are mostly inside the firm. That allows unusually direct, high-bandwidth communication: an engineer can explain exactly what is wrong, and a trader can describe the real impact without the constraints of an external customer relationship. That changes how we respond to incidents.

Consider a conventional SLO alert that fires when the error rate exceeds 0.01%. We use relatively little of that in live trading systems. If the one order you cannot cancel is the order that will bankrupt you, an aggregate success rate offers little comfort. Every economic detail of every order matters.

Instead, we rely heavily on event-based monitoring. At a particular point in code, a condition can raise an error for a human to investigate. That forces us to enumerate edge cases and decide deliberately whether each should alert, fail safely, or be handled some other way.

The benefits include specificity: an alert can point to the exact code path and provide detailed context. The cost is substantial engineering effort. Missed edge cases can create noise and tribal knowledge, and changing an alert may require understanding and modifying the underlying logic rather than simply changing an SLO threshold.

We still use metrics-based alerts, especially where there is no direct risk concern and for long-term system health. But for many systems we prefer alerting on symptoms rather than presumed causes. Consider a web application with a database: an outage of that database may seem like the obvious alert.

You also need to detect user-facing HTTP 500 errors, because a database outage is not their only possible cause. Alerting on both can create duplicates. Over time the assumption that a database outage always affects users may become false: failover or maintenance can make it harmless, and silencing exceptions adds layers of duct tape.

Users care about the failed requests, not the database in isolation. To aid debugging, attach the database state to the symptom alert or put it on a dashboard. That preserves the useful diagnostic information without waking someone for routine maintenance.

Some of our favorite alerts look at what I call epistemic health: they do not identify a particular broken service, but tell us that our picture of the world is inconsistent. A fill-too-good alert means a trade appears to have made too much money.

That signal can catch a market-data problem, a bad instrument definition, an order-engine error, a strategy bug or a human mistake. Another example is suddenly representing 60% of market volume when we normally represent 3%. The unusual outcome tells us something may be wrong even if we do not yet know where.

An alert also needs a recipient. If fill-too-good can indicate a failure anywhere in the stack, which team should receive it? There is no universally correct answer, but the question matters because someone must be able to investigate and act.

We also obsess over signal-to-noise ratio. Traders, developers and production engineers all need to understand that noisy alerts are worse than useless. We spend time adjusting logic, thresholds and edge cases because human attention is limited; event-based alerting depends on that cultural discipline.

Defense in depth addresses the possibility that any one alert or check is wrong. Trading systems, order-entry ports and external risk enforcers can run related checks built by different teams. We try not to share too much implementation or metadata, so the failures are less likely to be correlated.

Common dependencies are the weakness in this approach and deserve careful examination; sometimes they need independent versions too. Technical staff monitor software, hardware and connectivity, while traders monitor market impact, profit and loss, and whether intended orders were sent. Either perspective can reveal problems the other misses.

Monitoring is not optional for us. We generally do not trade without it, so monitoring systems are among our most robust, tested and redundant systems, with failover. We would rather have a trading system go down than lose the ability to monitor it.

Business context is essential for technical responders, and traders also need technical context. Direct communication helps only if we can understand one another. Knowing the language of the business lets us bypass a chain of lookups in an incident.

For example, if a trader says ‘Spoos market data looks stale,’ an engineer who knows the S&P 500 futures and where they trade can find the relevant feed in seconds. A person without that context first has to translate the term, find the exchange and map it to the right system.

Engineers also need to know which scheduled or unexpected events matter to a desk. A sudden policy announcement may explain abnormal market data, and a deployment immediately before important economic news can be a bad decision. Context changes both diagnosis and risk decisions.

The following sample incident is a composite based loosely on several incidents, not a literal account of one event. At 7:15 a.m., an order engine is routinely upgraded with bug fixes and a small price-serialization refactor; the tests pass.

At 9:31, traders receive individual fill-too-good alerts on several symbols. At 9:32, the order-engine team receives an aggregate alert and the engine automatically halts. Individual alerts can go to the affected trading strategies, while the broader alert goes to the team responsible for the last safety point before the exchange.

At 9:33, the team opens an incident and uses a ‘what changed’ tool to inspect code, configuration and metadata changes. The price-serialization refactor looks suspicious. Within a minute, responders see that the change was deployed to other order engines that have not alerted.

If the alerting is perfect, perhaps the refactor is a red herring, or it affects only one protocol. If the alerting is imperfect, the other engines might also be losing money without detection. The team discusses whether to halt them preemptively, balancing uncertainty and risk.

Traders then join the call and report that market data looks stale: prices that should update every second or faster have not changed. This shifts the investigation away from price serialization. The market-data team joins and confirms that some symbols have not updated since the previous close, yet are not marked stale.

A search reveals that the exchange had announced a feed split two weeks earlier. It was moving some symbols from four existing partitions to a fifth. The firm had missed the change and was not reading the new partition; symbols still associated with an old, otherwise current partition could appear live despite being stale.

The market-data team recommends keeping the affected engine halted and rolling a configuration change. After the change, trading resumes and the fill-too-good alerts stop. Further investigation reveals two failures: the exchange-driven change was missed, and the code meant to mark moved symbols stale had a bug in a path that had never been exercised.

The order-engine alert caught a market-data issue without knowing its cause. Other independent signals might also have caught it, such as a trade not appearing in the market-data feed. This illustrates why an outcome-level alert can protect against many different failures.

The timeline also shows the importance of timing. Even though responders acted within minutes, the open had already been missed. A postmortem should ask whether pre-open trading or another check could have exposed the issue 20 minutes earlier. A scheduled event such as Fed minutes would also affect urgency and risk decisions.

The turning point was a trader saying the data looked wrong. Technically, the systems appeared healthy; business-level observation brought in the right team. Clear communication across technical and trading roles can make an incident room much more effective.

To summarize, we use event-based alerts and independent outcome-level alerts for live trading, make monitoring especially robust and practice defense in depth. Technical responders need business knowledge. That said, I am emphasizing differences: plenty of production engineering at Jane Street resembles work elsewhere, especially outside the tight trading loop.

An audience member asks why the exchange email was missed. Doss says exchange-driven changes are common and most do not require action, so people must sift through a lot of notices. The firm uses processes and tools, including LLMs, to help improve signal-to-noise, but people have sometimes read a notice and incorrectly decided it did not affect them.

Another question asks how yesterday’s data could appear live. Doss says trading venues vary widely: on a normally illiquid exchange, a symbol without a new update can be plausible. In the incident that informed the example, an ordinarily quiet market became unusually active after news, making the failure consequential.

The order engine did not directly identify stale market data; it detected a trade that seemed too good. Asked who integrates event alerts into software, Doss says teams generally monitor the systems they develop. Engineers must think through failure modes and how the receiving human will respond.

That effort is expensive, he says, but dependable alerting can also make it possible to move faster elsewhere because teams have greater confidence that problems will be detected. The questioner notes that deep event integration can be difficult to implement without generating excessive noise.

An audience member asks whether teams share logging and alerting standards, since inconsistent terms can slow cross-system diagnosis. Doss says common libraries and log levels exist, but there are few top-down mandates; specific terminology and conventions can differ between teams. Much infrastructure is shared across desks, which also provides some commonality.

Asked whether aggregate firm risk is checked synchronously at a single gate or monitored asynchronously, Doss describes a mix. Automated systems may stop activity when particular limits are exceeded, while traders monitor broader firm-wide risk and make decisions over longer time horizons.

A final question asks about cascades of thousands of alerts and what happens to orders while an engine is halted. Doss says a halted order engine rejects the trading system’s order; the trading system must decide whether it wants to resend it.

On alert multiplication, Doss candidly says the firm has the same problem and has not solved it especially well. In the composite incident, no individual service appeared broken, so the outcome alert did not produce a service-failure cascade. Where downstream services do fail, symptom-based alerts can still be useful because they identify affected users and necessary changes in behavior.