Liora Friedberg on Solving Puzzles in Production

Ron Minsky welcomes Liora Friedberg, a Jane Street production engineer, to discuss how her team supports live systems. Friedberg explains that the firm runs software trading billions of dollars a day, so people need to notice unexpected behavior quickly and respond while production is running.

Support is a first-class part of the production engineering role. During the trading day, an engineer on rotation responds to alerts and reports from people who notice unusual behavior. This is distinct from overnight or weekend on-call work, although software engineers also participate in support.

The other part of the job is longer-term work that improves the response to issues or makes intervention less necessary. That might mean building an OCaml application for self-service requests, or improving processes for responding to major incidents. The mix depends on the person and team.

Minsky compares the role to site reliability engineering: live support and projects that improve support belong together. Friedberg estimates that production engineers spend roughly one-quarter to one-third of their time on rotation, with the rest available for longer-term work.

The boundary with software engineering is not sharp. Friedberg says software engineers tend to know a smaller set of systems deeply, while production engineers usually develop a strong working model of a broader set and their connections. A team benefits from combining different perspectives around the same goal.

Friedberg studied computer science and economics, then entered consulting because she wanted to try different kinds of work. She found the problems interesting but wanted a different motivation and better work-life balance, so she looked for a return to technology.

She knew of Jane Street from an OCaml class and applied for software engineering. The firm suggested production engineering as a possible fit for her interdisciplinary background. After learning what the role involved, she joined and stayed in it for five years.

What she enjoys most is support: it feels like a puzzle hunt. She gathers evidence across systems, forms hypotheses and sometimes gets a satisfying moment of recognition when the explanation becomes clear. There is also the immediate reward of helping colleagues resolve problems throughout the day.

Minsky adds that the puzzles extend beyond a single program: they may involve many technical components, people and organizations, including exchanges and clearing firms. The breadth makes debugging rich, but it also requires an understanding of how the business fits together.

Friedberg’s team owns applications that ingest, parse, normalize and group the firm’s trading activity before sending it downstream. The resulting records go into a historical database, a Kafka topic used by subscribers, and systems that communicate with banks.

The bank-facing flow matters because banks need the information to carry out the relevant real-world processing. Downstream systems also need a consistent central source. Traders use the data for live monitoring and calculations such as the current profit or loss of a strategy.

A routine support issue occurs when a new kind of trade reaches the pipeline. Friedberg describes a type of trade as a particular combination of fields and values, such as booking date, counterparty and settlement system. A previously unseen combination may have no configured path through the system.

Minsky points out that configuring a new settlement flow can require actual business decisions. Friedberg’s team works with operations, which has the relevant context, to translate those decisions into configuration the systems understand. The team is also working to make parts of that process self-service.

Production engineers acquire knowledge of the financial system as well as its technology. Friedberg likes the variety: she cannot always predict what will appear on a support day or what she will learn about while investigating it.

At the more serious end of the spectrum are incidents affecting multiple teams. Friedberg recalls trades whose identifiers had been manually modified. The application tried to raise errors for them, but crashed while creating the errors, preventing trades from being booked downstream.

The team found the crash and restored the system. Fortunately, a tool already existed to remove specific trades from the live stream, so they filtered out the malformed trades and then reflected on what had happened. It was an unusually severe example of data the pipeline did not expect.

On her team there is generally time to investigate, find a root cause and consult people. Other teams face more immediate pressure. Order engines connect internal trading intentions to exchange or broker protocols, so an outage can stop trading in the moment; Friedberg gives a hypothetical loss figure, not a measured incident.

Minsky clarifies that the immediate financial impact often means opportunity cost: trades cannot be made while a system is down. Post-trade processing can sometimes be repaired on a longer timescale, but monitoring is an exception. If essential monitoring is unavailable, trading cannot safely continue.

Friedberg previously worked on monitoring software whose redundancy, review and rollout processes were particularly rigorous. Her team used the system daily and had strong views on features, so she joined the team building it to represent those needs.

One feature she added was the ability to snooze pages, which she still uses. Minsky explains that Oculus is an alerting and workflow system: when something goes wrong, it presents a discrete alert to the relevant people, who must coordinate and keep track of it during potentially hectic periods.

Support engineers are often power users of these tools and can change them. Friedberg describes engineers building a stateful layer over automatic page actions to make automation more expressive. Sometimes production teams also create tools specifically for their own needs.

One such tool analyzes a team’s alert history. It helps answer what proportion of alerts are acted on, forwarded, snoozed or closed automatically, and which systems generate them. That evidence lets a team improve the health of its support rotation.

A common pattern is flickering: a page opens and closes without anyone acting. It still consumes attention, and an engineer may start investigating before it disappears. The team can review the code that raises the alert, change a threshold or fix the underlying alert so it does not create unnecessary noise.

Minsky distinguishes alerts raised by application code from monitoring that examines exported metrics. Historically, a system might notice an unexpected internal condition and raise a page itself. Fixing noisy alerts can therefore require tracing the problem back into the application, not merely changing a central alerting system.

A newer feature can take an engineer directly from a page to the call site in the codebase. Friedberg says alert statistics can also show transient pages that close without intervention, along with actions such as ownership, forwarding and snoozing.

Repeated snoozing is another signal to investigate. When the feature was introduced, the team considered whether it might encourage people to ignore problems, but chose to trust colleagues to use it thoughtfully. A forwarded alert might belong with another team from the start; a routinely snoozed alert might be raised too early.

Minsky worries about treating alerts like a video game: pressing buttons until the screen is clear without understanding the problem. Friedberg says good production engineering calls for reflection, not merely clearing a queue. Ask whether the alert needed to fire, whether its impact can be reduced, whether its message can be clearer and whether part of the response can be automated.

Automation and curation can raise the signal-to-noise ratio so people take meaningful alerts seriously. But Minsky notes a risk in the other direction: complicated filtering can contain bugs or suppress real issues. When the failure is that nothing happens, an over-silenced alert can be difficult to notice.

Friedberg says applications often expose internal state through command-line tools, making it possible for expert users to inspect and automate around them. She does not claim an easy answer to over-automation. Code review helps, and teams also discuss triage and prioritization.

On her team, engineers write a summary at the end of each support day. The issues they encountered often prompt discussion of possible improvements, and ideas that gain the most traction get picked up. Minsky stresses that support automation is real software whose correctness matters.

Under-filtering can overwhelm people until they stop paying attention; over-filtering can hide important information. Both failures can undermine monitoring. Tools that shape an engineer’s view of production deserve serious engineering and review.

Beyond alerts, support depends on debugging and introspection. Friedberg points to a relatively recent push toward centralized log management. Previously, engineers often connected to production machines with SSH and read or parsed log files on each host.

Jane Street also stores much data as S-expressions, so people build shared tools to extract information from them. Minsky describes the internal environment as sometimes better, sometimes worse and often different from the tooling found at other companies.

Centralized logs are part of a broader observability effort, including distributed tracing. At the same time, some specialized tools have existed for years: highly detailed packet captures with accurate timestamps help reconstruct events across trading systems at extremely fine time resolution.

New tools still have to be adopted. Friedberg notes that engineers may be excited about them, but migration competes with many other priorities. Minsky warns that a long tail of older systems can increase complexity and risk once most people are accustomed to the new approach.

Making time to retire old approaches helps engineers navigate unfamiliar parts of Jane Street’s large monorepo. Production engineers often read the code that generated a page; shared, current tools make that investigation easier for them and for software engineers.

Not every production engineering project is primarily coding. Friedberg gives the example of preparing a new kind of trading: an engineer might talk to the desk, read a specification, write some code and coordinate with downstream clients. That person connects teams and systems to make the new flow work.

Minsky sees a connection between that operational work and support. Understanding a new counterparty, the surrounding systems and the details of setup prepares engineers to respond when the first day of a new flow produces unexpected failures.

Culture is just as important as tooling. Friedberg says people should feel comfortable admitting mistakes, including production mistakes, and raise them promptly so the team can mitigate the impact. Postmortems describe what happened and help people learn rather than assigning blame.

Minsky says an individual may sometimes need to change their own behavior, but recurring errors also call for changes to the system: make mistakes less likely or reduce their blast radius. A postmortem records the sequence, causes and resolution, then identifies concrete improvements.

Those improvements might include reaching affected users sooner or making technical changes. Writing honestly can still be uncomfortable, especially when several people made different mistakes along a chain of failures. The person involved should have an opportunity to explain what happened.

Friedberg says newcomers learn by reading postmortems and watching others acknowledge errors without losing respect. Teams should also support someone directly when an incident involves them, making sure they feel comfortable and that the discussion remains thoughtful.

Minsky has mixed feelings about postmortem templates. A structure for timelines, what went well, what went badly and takeaways can help, but it may encourage authors to fill boxes without stepping back to ask how serious the incident was or what the organization should learn.

Friedberg agrees that templates can constrain thinking, but finds them useful for someone writing a first postmortem after an intimidating event. With experience, people may need more room. Team discussions and conversations after incidents are also important; a document should not be the only reflection tool.

Training becomes harder as systems become more reliable. Minsky recalls earlier periods with frequent production failures, which gave engineers many opportunities to learn. Today, teams must prepare people to debug serious incidents even when those incidents occur less often.

One approach is a tabletop incident simulation. A trainer presents a scenario, the trainee chooses actions, and the trainer describes what happens next. The conversation covers technical response, mitigation, stakeholder updates and how to think through the developing situation.

Some teams also use games such as Overcooked or Keep Talking and Nobody Explodes to practice communication under manufactured pressure. They are fun, but they also require teammates to stay calm, think clearly and coordinate while the situation feels stressful.

Another option is to deliberately break a nonproduction version of a system and let trainees investigate with real tools. Minsky calls this close to ideal, but it requires a realistic parallel environment and people to design interesting failures. Friedberg has seen it used, though less widely than she would like.

She recalls a firmwide exercise called Training Wheels: a small system would break and engineers could explore its manageable codebase and filesystem to fix it, even without prior knowledge. Such exercises have appeared over the years, but they are not available on every team.

New production engineers attend OCaml boot camp as software engineers do, plus production-specific training and classes. Much of the remaining training is team-specific: an experienced colleague sits with a newcomer for weeks, supports issues together and gives feedback. Minsky describes this as an apprenticeship.

Some teams use a more structured training model, particularly where support is high-stakes. The needed qualities include strong debugging, carefulness, calm and clear communication, but the actual work differs substantially between teams.

Recruiting reflects that variety. Students often apply for software engineering without knowing production engineering exists, so Jane Street may raise it as an option and offer production-focused interviews. Experienced candidates may already know what kind of operational work they prefer.

Friedberg emphasizes communication: production engineers connect teams whose members may have different mental models and use different language for the same trade. Investigative debugging, caution in production and a level head under pressure are also especially valuable. Different people thrive at different levels of urgency.

Minsky says some people find putting out fires energizing, while others find it unpleasant; Friedberg adds that resolving a difficult incident can be especially rewarding. Neither treats creating an emergency as desirable—the appeal is in responding well when one occurs.

Candidates receive software engineering questions as well as production-oriented exercises. In one example, they investigate a failure using code, command-line tools and data. It is based on work Friedberg’s team used to perform repeatedly before building a self-service website.

The exercise is synthetic, but interviewers can observe how a candidate forms hypotheses, chooses steps and explains their reasoning. Friedberg says the process matters: merely looking at the final code is not enough to form a strong view of how the person investigates.

Minsky agrees that interviews reveal more when they show what it is like to work together, not simply whether a puzzle was solved. Production work is collaborative and connected. Friedberg adds that interviews should also let candidates shine when they fit roles with relatively little coding.