Ella Ehrlich on Modernizing a Critical Trading System

Ron Minsky introduces Ella Ehrlich, a Jane Street software engineer who has spent years working on Gord, a system older than her roughly decade-long tenure. Ehrlich came for an internship despite having little interest in finance and expecting to pursue game design. The experience of learning and growing led her to start her career at Jane Street.

Ehrlich still plays and follows games, but is happy to keep them as a hobby. From friends in game development she has heard about a highly competitive industry with long hours; she compares game developers to the musicians of software. Minsky agrees that passion draws many people to difficult, competitive creative work.

Gord receives trading data from platforms and exchanges around the world, normalizes it and distributes it inside Jane Street. Consumers use it to calculate positions, upload trades to clearing firms, settle transactions and support other workflows.

Ehrlich explains a position with an example: buying 100 Apple shares creates a position of 100 shares; selling 20 leaves 80. Minsky adds that this information supports real-time risk and profit analysis, regulatory distinctions such as short sales, and end-of-day reconciliation with other market participants.

Performance matters most when trading is busiest. Ehrlich says the team plans for at least twice, and ideally two to three times, the most recent record day because new peaks can exceed previous ones by large margins. At the time, she says a busiest day could reasonably be five times a normal day, though comparisons vary.

If Gord goes down, Jane Street automatically stops trading because it cannot safely make decisions without knowing its activity. The system is designed to isolate failures, for example at one exchange, and changes undergo extensive testing. A production standby processes the same activity in a separate data center so the firm can fail over after machine or network problems.

Minsky says Gord was built when the firm had less experience writing software. The team has had to grow and harden it while working around decisions embedded in the original design. Ehrlich describes an early pipeline that used SSH to follow files from sources, with pipe-separated tag-value data.

An early promise was that Gord would deliver an order before a fill, even if the source sent a fill without an observed order. To preserve that invariant it synthesizes an order. Ehrlich now considers this a mistake, but consumers have come to depend on it, making the assumption hard to remove.

Gord also normalizes symbols. The same instrument can have different identifiers from exchanges and data providers such as Reuters or Bloomberg; different regions and asset classes add complications. Usernames have similar problems when one platform provides an email address and another a specialized login.

The team has gradually moved some normalization closer to the sources, where the relevant knowledge naturally lives. Minsky notes that the original system consumed raw files downloaded from exchanges or brokers, leaving Gord as the only place to normalize. More activity now passes through internally built systems that understand individual venues.

Gord also imposed protocol-level invariants. Its old core message types were orders, outs for closed orders, and fills. To represent a rejection, the system synthesized an order and a close, adding a field to say it came from a rejection; consumers had to inspect that field rather than receive a first-class rejection.

Clients originally connected directly over TCP and decoded binary messages. Because the protocol had no general versioning strategy, new information was often added to a string-to-string map rather than the top-level type. This was convenient for optional fields but made the meaning and use of fields harder to understand.

Corrections and allocations were represented by canceling and rebooking an entire fill rather than making smaller adjustments. Minsky observes that a flexible format can still become rigid: client code silently depends on the structure inside it. With hundreds of clients, many critical to trading, breaking those assumptions is not an option.

For a fundamental protocol change the old system had to upgrade or downgrade messages for each client version, potentially serializing the same event several times. Optional fields remained easy to add, Minsky notes, but new transaction types were much harder. New high-volume trading also required a smaller representation than the large existing types.

Ehrlich describes the architecture: parsers read sources, a process called the DB collects data and may synthesize missing events, and normalizers in each office adjust symbols and users before distributing data to clients. Sending the full stream directly to hundreds of consumers created an expensive fan-out problem.

The implicit contract with sources was essentially that Gord would always keep up and never fail. Throughput limits were not documented or enforced. Ehrlich built performance tests to find where a source needed sharding and to measure the capacity of a single pipeline.

At the time of the interview, a parser–DB–normalizer pipeline could process roughly 50,000 messages per second, but the team used about 25,000 as an allowable rate because of additional downstream work. Independent pipelines can be spread across boxes, providing horizontal scaling rather than forcing all activity through one thread or process.

The system guarantees ordering within each upstream source, not one shared global order across sources. Events at separate exchanges do not have an easily meaningful universal arrival order anyway. Minsky notes that this weaker guarantee also allows offices to continue operating with degraded cross-office connectivity.

The team moved away from each client connecting directly to Gord by using Kafka as a distributed message queue. Previously, a slow client accumulated per-connection buffers and could eventually be disconnected. Some consumers, such as end-of-day reports, do not need to keep pace at every market peak as long as they process the data by the deadline.

With Kafka, consumers request messages when ready and can fall behind the tip of the stream without immediately failing. The speakers clarify that Kafka also uses TCP; the improvement comes from the engineered queue and its resource management, not from avoiding TCP itself. Messages are retained on disk instead of accumulating as separate active buffers for every slow client.

Kafka also improved snapshots. Previously Gord generated a snapshot of all prior activity for each client that connected. A separate process can now consume the stream and periodically create shared, ready-to-use snapshots; clients start with a snapshot and then continue from the corresponding point in the stream.

Another change addresses producer-side data quality. The old FIX-style tag-value files had many possible fields but few explicit constraints. Producers had to ask Gord developers which fields were required—for example, a bond execution needs a settlement date even when other executions may not.

A new library gives producers typed records for particular transaction types and tells them what they must supply. It handles translation into Gord’s existing representation, reducing coordination and making it easier to change the underlying format later. At the time, the library was still being developed and refined.

Minsky sees the typed interface as a way for producer teams and Gord developers to collaborate on structured transactions. In the longer term, types can extend through the pipeline: changes become visible across applications, aided by a shared monorepo in which current versions compile together.

Consumers originally chose the full stream or fills only; a monitoring stream later accommodated activity that did not affect bookings or positions. As the firm grew, consumers wanted subsets such as options or client-facing trading data. The team began providing relatively static substreams, while dynamic queries by symbol remained a future goal.

When Ehrlich joined, the system had almost no in-code tests. The team compared development output with production over several days and followed extensive manual procedures before a rollout. Releases happened a few times a year; by the interview they were about monthly, still slower than the team wanted.

Ehrlich added performance tests to detect regressions at each rollout. The team also built in-code tests spanning input through output. Previously a developer had to start a full development system and manually construct activity; automated tests could demonstrate the intended behavioral change and make reviews and changes more confident.

A former deployment process relied on manually changing a symlink to the production binary at different times for Hong Kong, London and New York. The regional trading days overlapped, so missing a narrow window could leave offices disagreeing about which instance was primary.

Ehrlich describes forgetting the London symlink during her second rollout and discovering it after that region had started. The team had to roll back and deal with a production incident. She notes that ADHD makes remembering tasks in narrow time windows difficult; the deeper problem was a process dependent on repeated human vigilance.

The team replaced the manual switches with a configuration that can be staged days in advance. A process tracks which Gord should be primary for a given trading day. Minsky notes that the complexity arose partly from historical regional trading-day boundaries and overlapping global data sharing.

The firm was moving toward more continuous operation, and Ehrlich wanted Gord to become more available around the clock. Shutdown windows are still useful for rebooting and archiving, so dynamically adding and removing machines was part of a longer, gradual transition rather than a completed change.

Turning to recruiting, Ehrlich says she was the second female developer at Jane Street when she joined. She had not expected to work in finance and might never have considered the firm without her internship. Outreach therefore tries to introduce potential candidates to the work, people and culture before they decide where to apply.

Women in STEM brought students between high school and college to meet employees and learn about the firm. Minsky describes it as a long-term investment: participants might not apply for internships until years later, but some eventually did. Ehrlich says early positive contact can counter negative perceptions of finance.

Insight brought sophomore women to Jane Street for a week of finance and OCaml instruction and time with employees. Ehrlich says the program helped attract internship applicants and was expanded. The number of women developers grew enough that she no longer knew them all individually.

In the summer discussed, about 25% of developer interns were women. Ehrlich calls that a large improvement from the starting point but says it is still below her goal. The team continued to seek ways to recruit and retain women and maintain an excellent working environment.

The recruiting team built campus relationships with women-in-computer-science groups and broader organizations, including through the Grace Hopper conference. Once a firm establishes a reputation at a school, word can spread. Minsky asks about reaching talented people at schools where Jane Street has no established relationship; they do not give a specific Insight answer.

Ehrlich says experienced women hires remained a harder problem: only about half of developer hiring came through internships, with the rest from industry, and the percentage of women among lateral hires was substantially lower. The team was experimenting with several approaches and had not found a single solution.

One favorite experiment was brunch for women in technology, conceived as a community event rather than a hard recruiting pitch. Ehrlich and a colleague chose something they would themselves want to attend. Participants could meet peers even if they were not looking for a job, with possible recruiting benefits emerging over time.

Minsky connects this to a broader long-term approach: an event or technical blog can support recruiting while providing genuine value to participants or readers. Ehrlich enjoyed meeting other women developers and hoped the brunches could return as pandemic restrictions eased.

Ehrlich also describes a new educational program for students from groups underrepresented in technology, between their first and second college years. It aimed to reach people without an established recruiting pipeline or peers who could help them prepare for interviews.

The program planned several weeks of OCaml instruction followed by open-source work. Participants could gain education and a public project to discuss on a résumé, useful when applying for future internships whether at Jane Street or elsewhere. Open source made the work visible without putting participants inside proprietary systems.

Asked about results, Ehrlich estimates that around 20% of participants in Insight ultimately entered a Jane Street internship, while noting uncertainty about current aggregate numbers. In the first group she recalls roughly five of 25 students joining internships over the following years.

She recalls about three of 15 students from the first In Focus program joining Jane Street the next summer. These were early results, and the team wanted to improve them. Many participants told her they would not have considered applying without the programs; she sees meeting employees as one of the strongest ways to convey the firm’s culture.

Minsky and Ehrlich close the conversation after discussing how long-term engineering and recruiting work both depend on gradual, sustained improvement.