Why a plausible contract analysis missed $800,000

Natalia Connolly describes a large healthcare company acquiring a smaller specialist. Its procurement team faced hundreds of vendors, more than 40,000 contract documents and disconnected databases. The speakers use a synthetic replica to illustrate the engagement.

The buyer wants to know whether the combined company can negotiate better terms, estimate savings and act before renewal deadlines. The analysis must be traceable to contract pages and invoice lines so a negotiator can defend it.

The first Claude Code answer looked complete and plausible. Annual spend, the split between companies and the contract count were correct, but the total contract value was off by about $800,000.

The agent also gave December 1 as the notice deadline when the correct date was November 1. Waiting for its answer would have allowed the contract to renew automatically in the example.

Connolly explains that a long contract can silently fail during preprocessing and never reach the database. An amendment can also be extracted but left unlinked to its parent; that orphaned amendment accounted for the missing $800,000 in the example.

Another trap is adding every value in a contract and amendments even when each amendment restates the total. In her illustration, summing the numbers yields about $17 million, while the latest restatement sets the correct value at $7 million.

Tables that cross page boundaries cause another problem: the second page may lack column headings. The system needs the context of the whole table to interpret the numbers.

The pair spent about two weeks building a processing pipeline, then processed more than 40,000 contracts overnight. Connolly says this preparation made the resulting structured data easier to inspect and verify programmatically.

The five stages cover preprocessing and document recognition; document classification; extraction of terms, rates and spending; mapping services to spending categories; and an analyst agent working over the extracted data.

Kevin Madura notes that contracts can include charts and other visual material that naive text recognition misses. Their approach sent page images through DSPy to obtain structured information rather than preserving only chart labels.

When the team knew the engagement needed rate tiers, rebates or other fields, it defined that structure first. Claude Code could then use the normalized contract, spending and invoice data to build or call tools tailored to the deal.

Custom tools such as retrieving a rate-card page or vendor spend can encode business logic and known exceptions. The team favored code and database views for some rules rather than relying entirely on prompts.

The tools and configurations can carry knowledge through an engagement and govern data access. Madura describes allowing a defined set of operations and blocking other calls.

The incorrect renewal deadline illustrates why business context matters. The agent had used a date from a termination clause, while the question required a calculation from the renewal clause.

Contract-management databases may omit or mislink amendments. The team therefore reads underlying documents and reconstructs relationships among master agreements, statements of work and amendments, avoiding both omissions and double counting.

Access controls created another failure mode: a denied or malformed response from one company’s data source could look like a zero. Madura says explicit error feedback lets the agent stop, request human help or look for another authoritative source.

The team dispatched agents to investigate different savings opportunities in parallel. Their findings then needed reconciliation: two valid analyses can refer to the same vendor or invoice, and the same saving cannot be counted twice.

For that reconciliation, the team used a more prescriptive process specifying which data points to compare and what the next action should be. This structured step followed the agents’ more open-ended analysis.

An agent also needs to know what management has already started. A recommendation may look attractive in isolation but be irrelevant if the company began the same work months earlier.

Madura recommends retaining useful business context in memory, tools or similar mechanisms so later analysis does not repeatedly rediscover known limitations. The pair’s main lessons are to structure the data early and provide the right business context.