A high-reliability automation starts by routing the hard cases away

SUMMARY

In 30 seconds

An early document-indexing agent reached about 80–83%, but the failures combined several different problems: model errors, organizational ambiguity and genuinely novel cases. Matthew Jewkes says his team built separate routes for them and, after extensive evaluation, reached 99.95% accuracy on a selected set of business-critical tasks. The routine path ran automatically while uncertain cases went to experts. He argues that grouping those exceptions can also help people spot unmet demand and decide where to invest.

1Give the routine workflow an escape hatchJewkes uses AI to recognize inputs that require different handling instead of forcing every case through one path.
2Concrete examples helped experts identify three different failure typesThe team reviewed about 1,000 evaluations and separated critical errors, ambiguity and unusual exceptions.
3Routing received as much attention as improving critical-path accuracyUncertain or novel cases could be held for people, while business-critical cases used a heavily tested path.
4The selected critical path reached 99.95% after extensive iterationJewkes reports improving it from about 95% after excluding ambiguous and unusual cases, then comparing a random sample with historical work.
5Grouped exceptions can reveal decisions or demand that scattered cases hideExperts can decide whether to specify a rule, stop low-value work or investigate a potential opportunity.
6The investment makes sense where the process and value are establishedHigh reliability requires engineering, evaluation, trust and organizational change; Jewkes favors substantial, recurring business problems.

Takeaways

Separate critical errors, undecided cases and unusual exceptions before optimizing a single accuracy number.
Give an automation a clear way to stop and route cases outside its routine path.
Evaluate with domain experts using concrete examples, then invest in inspecting failures.
Group exceptions so experts can identify patterns and choose which problems merit attention.
Reserve expensive high-reliability automation for established, valuable processes with clear customer pain.

TopicsAgent engineering Workflow automation Evals Enterprise AI

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

Protect expert attention with a routine path

Jewkes advocates automation with a way to stop and escalate cases outside its intended scope.

Matthew Jewkes asks leaders to consider their strongest domain experts: people repeatedly interrupted by urgent customer issues and decisions. As routine execution becomes cheaper, he argues, deciding what deserves attention becomes more valuable.

He advocates fully automating a well-defined routine path while reserving human attention for exceptions. In his view, autonomous agents can be useful in development but hard to maintain and capable of generating additional decisions for experts.

An automation should have an escape hatch. AI can check whether an input fits the routine path, classify unusual or business-critical situations and route them for different handling instead of forcing every case through one workflow.

Separate the failures in a biopharma workflow

The team used concrete evaluations to distinguish critical errors from ambiguous decisions and unusual exceptions.

Jewkes describes a biopharma document-indexing system at Clarivate. He says it processed tens of thousands of tasks per month and reached 99.95% accuracy on a selected set of business-critical tasks, after substantial development investment.

SLIDE 1 / 7Biopharma indexing caseJewkes reports the scale and accuracy of the selected business-critical tasks.

The initial agent-like approach reached about 80–83% after a few sprints. The team gathered roughly 1,000 evaluation examples and reviewed them with domain experts, giving the experts concrete cases rather than asking them to fill in a blank spreadsheet.

SLIDE 2 / 7Three kinds of difficult casesThe review distinguishes ambiguous decisions, critical errors and unusual exceptions.

The review revealed three kinds of problems: genuinely critical errors, ambiguous cases where experts or the organization had no settled answer, and unusual exceptions requiring additional human judgment. These should not all be treated as the same model failure.

Jewkes uses a simplified drug-target example to show ambiguity: experts may disagree about whether an indirect biological effect counts as a target. He contrasts that with emerging drug mechanisms that require humans to update the indexing scheme.

Route cases and improve the critical path

The team reports 99.95% accuracy on selected critical tasks after extensive testing and routing.

The team invested heavily in routing. Confusing or unusual inputs could be described and set aside; potentially critical cases entered a heavily evaluated path; unresolved outputs went to a person. Routine cases continued through the ordinary process.

SLIDE 3 / 7Route cases by business priorityThe workflow routes uncertainty, critical cases and undecided cases differently.

SLIDE 4 / 7Improving the critical pathThe reported figure applies after ambiguous and unusual cases were separated.

After excluding ambiguous and unusual cases, the team reached about 95% accuracy on the critical path with standard approaches. Jewkes says extensive evaluation and iteration raised that figure to 99.95%, and deployment followed a random-sample comparison with historical work.

He claims the automated process outperformed historical expert work at scale, while acknowledging that a PhD expert examining a single case closely could do better. He attributes the difference partly to workload, time pressure and human fatigue.

Turn exceptions into organized decisions

Grouping remaining work can guide expert attention and expose potential demand. Jewkes explains when the investment may be worthwhile.

Beyond saving time, the system groups the remaining work into critical reviews, ambiguous cases and unusual exceptions. Jewkes says this makes it easier for experts to decide where to spend their limited attention.

SLIDE 5 / 7Organizing expert attentionRemaining cases are grouped for review rather than mixed into routine work.

A backlog of ambiguous, marginally valuable cases can prompt an organization to decide whether to specify the rule or stop doing that work. Grouping unusual cases can reveal patterns that are hard to notice when they arrive one at a time.

He describes an expert using a structured set of exceptions to form an investment view before a major customer raised a related need. Jewkes sees such grouped exceptions as possible leading signals of market demand.

High-reliability automation is expensive to build: engineering time, evaluation review, organizational trust and ownership changes all cost money. He recommends choosing established processes with substantial value and clear customer pain.

SLIDE 6 / 7Where to investThe speaker favors established, valuable processes with visible customer pain.

Jewkes favors detail-oriented work with a reasonably specifiable process over work dependent on open-ended subjective judgment. For startups still discovering a business, he would retain human flexibility rather than rush into full automation.

SLIDE 7 / 7Techniques and evaluationThe slide lists practices for improving a high-reliability workflow.

His practical advice includes writing and reading evaluations, providing escape paths, versioning inputs and prompts, breaking tasks into smaller parts and capturing failures. He emphasizes that reviewing a large evaluation set is labor-intensive.

Jewkes closes with his prediction that competitive advantage will come less from selecting a particular orchestration framework and more from how organizations manage collective attention and judgment.