The 95% problem: AI, chat trading and human review

Chloe Casale arrives wearing a boot after the Mongol Derby, a ten-day, 1,000-kilometer endurance horse race. Riders navigate across the steppe using GPS points and change randomly assigned horses roughly every 35 to 40 kilometers.

She had grown up around horses and worked on a ranch, but was not a competitive rider. She entered an academy expecting an adventure rather than a place in the race; once accepted, she decided to do it.

Casale says a stirrup repeatedly struck her shin. Trying to cushion it with socks restricted her ankle movement and contributed to a stress fracture. On the last day she could not put weight on her leg, but managed to mount and ride the remaining 60 kilometers.

She says 23 of 43 people who started finished, with heat exhaustion and injuries among the reasons others dropped out. She was proud to finish but thinks once was enough.

Her interest in engineering began with wanting to know how cars worked. She took an automotive technology class in high school and initially thought she might become a mechanic or mechanical engineer.

At university, friends bet her $20 that she could not pass an introductory computer science class. Writing simple Java programs felt like magic: a few instructions could produce a game. She fell in love with programming and switched to applied computational mathematics so she could graduate in four years.

After university she joined Palantir as a forward-deployed engineer. Rather than building one product for many customers, her job involved working with an individual customer, understanding its problems, adapting existing software and building what was missing.

The role also meant telling product teams when a customer’s need might be useful elsewhere. Casale likes talking to users and says the easiest solution is sometimes an email, not more code.

After nearly five years at Palantir she wanted a new challenge. She applied to Jane Street without knowing trading, OCaml or functional programming, and was impressed by the care invested in its interview process.

Two interviewers attended most of her Jane Street interviews. Minsky says a second observer helps interviewers compare impressions and improve their technique; Casale says the extra engineering time is expensive but she continues to find the second perspective valuable.

Although based in New York at Palantir, she spent much of her time at customer sites in several countries. A period in London convinced her to return there for her next job: she liked the city but found it less intense than New York.

At Jane Street, Casale builds tools for sales traders. They maintain relationships with clients such as funds, bring requests into the firm and work with traders responsible for pricing and managing the resulting positions.

Minsky describes Jane Street’s broader role as providing liquidity: buying from one party and selling to another. Unlike a sales trader, a proprietary trader may focus on pricing, strategies, managing risk and assessing whether trading is profitable.

A client might seek a large trade that is difficult to execute at a good price on an exchange. A pension fund rebalancing a portfolio, for example, may want more of an ETF than the available orders at an attractive price can supply.

Minsky explains that market makers post prices knowing some counterparties may have better information. That risk can lead them to post smaller sizes or wider spreads. Casale says knowing a client and why it wants a trade can change that assessment.

For an ETF with little liquidity, Minsky says a firm may price the fund using the underlying securities and manage its resulting exposure in those securities. The client gets a simpler way to execute a large trade without building that capability itself.

Casale was surprised to find clients still using chat and sometimes the phone. A request for quote asks, for example, what price a firm would offer for 100 Apple shares; a client can ask several firms and choose among their prices.

Dedicated quote platforms provide structured messages but can require integration and fees. For a client trading only a few times a year, a message through a system it already uses may be more practical.

Structured forms can also be too rigid. A client may not know the size of a trade yet, or may not want to disclose it, while a platform requires that field. Chat accommodates those variations.

Before large language models, some regular chat messages could be parsed with rules tailored to the counterparty. Casale says a sales trader still watched the chat and might send the resulting quote, even when the request entered internal systems automatically.

Automatic extraction can improve speed and reduce manual typing errors. A human comparing a parsed request with a message may be faster, and perhaps less error-prone, than retyping all the details.

Different clients have different timing expectations. Casale says some requests need to reach the internal system in under 20 seconds, some in under five, while other clients expect a price within an hour.

Speed can reflect a genuinely time-sensitive trade or a person’s workflow: someone may simply want to complete one request before starting another. Casale works with sales traders to help them ask and answer questions about what clients need.

Language models make it possible to consider parsing less structured messages, including inconsistent formats and typos. But Casale worries that their non-deterministic behavior makes it hard to know when the output will be wrong.

She gives a hypothetical 95% accuracy rate: if a human sees a long stream of correct results, they may habitually approve them and miss the remaining errors. A parser that is wrong most of the time has a different problem: traders may stop using it.

Minsky suggests deliberately inserting errors to test whether reviewers catch them. Casale says she raised the idea but traders were not enthusiastic. She would rather explore ways to catch mistakes without depending entirely on a person, such as comparing two independently prompted parses.

Latency creates another constraint. Some counterparties need a response too quickly for a model whose completion time is hard to guarantee; others may allow more reasoning time. Adding conversation history could improve context but may also make inference slower.

Casale says their model-based parsing is still early: the present priority is getting prompts to produce the right result, even when the model takes time, rather than optimizing specialized inference hardware.

She gives a hypothetical failure: a request for Apple shares is parsed as Tesla, the wrong price is sent without context, and both sides say the trade is done. A counterparty might notice an implausible quote, but the error could also create a loss or a difficult conversation about reversing the trade.

Minsky says an obvious mistake can sometimes be corrected even after both sides say done. Casale notes that the price may have moved by the time an error is found, and the trader must weigh obligations and the client relationship.

One safeguard is an explicit confirmation spelling out the trade: security, size, price, settlement date and other relevant terms. Even if the initial parse is wrong, the counterparty has another opportunity to see the discrepancy.

Minsky distinguishes that confirmation from free-form model output: a deterministic system can present the actual terms used for pricing. Casale works with traders to learn what information is needed to make a confirmation unambiguous, especially for complex options.

For interfaces, Casale distinguishes Bonsai, an OCaml web framework suited to substantial applications, from Webs, a tool used for more lightweight data views and dashboards. A temporary view does not always justify building a full custom interface.

She describes Webs as a shared computation graph and a way to build interfaces around tabular data. Minsky explains the graph: incoming data changes incrementally, and only dependent calculations need to be recomputed rather than rerunning everything.

Webs grew out of a desk’s pricing spreadsheet. Minsky says the later system scaled the same basic idea and made computation collaborative: users can access data and calculations that others have already contributed and build new views on top.

Casale says Webs becomes more useful as users contribute their own views. Recruiting has begun using it, and an integrated AI coding assistant helps recruiters create views and inspect data without asking a developer for every query.

Minsky adds that a Webs view can be deployed immediately into the shared system. Casale contrasts that with a quick AI-generated HTML page a trader can make to test an idea before deciding whether the tool merits a more durable implementation.

A prototype might stay a simple read-only page, move into Webs so others can reuse it, or become a custom Bonsai application if it is important to trading and needs a specialized interface. The level of production work depends on use and consequence.

Casale says a read-only dashboard for checking whether clients have stopped trading can be a relatively low-risk use of generated code if it does not drive trades or regulatory reporting. If those assumptions change, she wants developers to examine the code and the data it pulls.

Minsky asks what prevents unread generated code from doing something unexpected. He points to browser sandboxing and network access controls as useful boundaries, while noting that a full program or a shared Webs instance can have different permissions.

Casale does not think the safety tradeoffs are solved. More AI-generated code also means more review, and reviewers can tire, especially when they are the first developer to inspect a piece of code.

Minsky describes ideas under consideration: a separate document defining allowed behavior, checks that enforce it, or an independent model session reviewing a change against the specification. The small document defining the boundaries would still need human review.

Casale says her sales team has a related workflow: the person building an app writes what it is meant to do, then a skill reviews both the description and the app to suggest whether a developer should inspect it. It is not always right and may be conservative, but helps triage a growing volume of work.

Minsky says the firm has begun hiring for formal methods, which can prove certain properties of programs. He sees this as useful in some places but says the right combination of safeguards remains difficult to determine as model capabilities change.

Casale compares being a desk developer with being a forward-deployed engineer. At Jane Street, both she and the traders she supports work for the same firm, making it easier to discuss costs, priorities and tradeoffs openly.

At Palantir she cared deeply about customers’ problems, but the commercial relationship also shaped conversations: her employer wanted to sell its software, while the customer wanted to accomplish its own work. She says understanding a workflow in detail is common to both roles.

At Palantir, a forward-deployed engineer was expected to feed reusable findings back into a central product. At Jane Street, Casale spends more time on her own desk, with multiple teams supplying different components rather than a single core product.

Her team’s chat-to-quote tool illustrates the challenge. It served sales traders for a long time before the team realized that proprietary trading desks faced a similar problem and generalized the service for them too.

Casale says some engineers excel at user-facing work while others excel at turning recurring needs into a shared product. Her team has begun organizing around those strengths while allowing people to learn a wider range of work.

She thinks different desks sometimes build similar tools before realizing they could share one. Minsky offers Webs as a related but different case: one desk built it first, then generalized it; the value of improving it locally created an opportunity cost to broadening it sooner.

Minsky observes that Palantir hires people for distinct tracks such as forward deployment and product engineering, whereas Jane Street tends to hire strong generalists and decide where they fit. Casale says that changed how she thought about recruiting.

Casale contrasts repeated, similarly shaped Jane Street interviews with Palantir’s approach of examining different facets separately. The former seeks confidence in broad software development skills; the latter aimed to find something a person did exceptionally well as well as sufficient ability elsewhere.

Minsky says a Jane Street interview can still reveal multiple qualities: coding habits, communication, the ability to listen and a well-calibrated sense of one’s own uncertainty. He recalls that simply asking difficult puzzles in the firm’s early days did not always identify successful colleagues.

Casale describes a Palantir learning interview in which candidates were taught a made-up programming language and asked to solve problems in it. The goal, disclosed to candidates, was to observe how they learned and used help from the interviewer.

Other interviews focused on debugging an existing program from client bug reports, or breaking down a large open-ended design problem. Each could reveal communication and problem-solving, but was organized around one specific skill.

Casale says a focused facet can expose a candidate who is outstanding at one thing but merely adequate elsewhere. She worries a broad process could miss someone especially strong at talking to users and decomposing their problems.

Minsky sees two risks in that approach: an employer might choose facets that do not map to useful jobs, and a single interview gives a noisy measurement that can miss a capable person or overstate a skill. Casale agrees the process was noisy.

Hiring experiments also have a long feedback loop. Casale estimates that after changing a process, recruiting, onboarding and observing a hire may take roughly a year and a half before the decision can be assessed.

Minsky worries that a generalist process can underrate specialists. For example, performance engineers are asked to prioritize simple code during interviews even though thinking deeply about performance is central to their work; their habits could be mistaken for inability to follow direction.

Casale says experienced candidates bring habits from their careers, and interviewers must distinguish a habit that is useful at Jane Street, one that is easy to change and one that could impede success. Different backgrounds and regional candidate pools make standardization harder.

Minsky compares recruiting to looking for lost keys under a streetlight because that is where one can see. A process may reliably find good people without being the only way to find them; a growing organization must keep testing how to search more widely.

Casale says she initially worried that experienced and campus candidates were being assessed too similarly. She thinks the firm may miss people who could succeed in different ways, while still valuing a generalist culture that makes changing teams possible.

Minsky values the flexibility to reorganize teams and move people to new problems. Common technology choices can help, although even supporting both OCaml and Python involves work and can affect how easily engineers move between projects.

The firm also has useful specialists, including hardware and front-end engineers. Casale enjoys front-end work because a small improvement can make an immediate difference to a user; Minsky argues that designing flexible, performant interfaces is challenging work.

If she could change one thing, Casale would give every candidate—especially an experienced hire—at least one interview selected for a strength visible in their background. A performance specialist might get a performance problem; a former CTO might get a high-level leadership or design problem.

She would not replace the generalist process or hire solely on that specialty. Her goal is to give candidates a chance to demonstrate an ability that broad interviews might otherwise leave hidden, and to make that evidence available when deciding whether they would succeed.