Why a plausible contract analysis missed $800,000

RÉSUMÉ

En 30 secondes

A procurement agent’s first answer looked complete, but a missing contract amendment left its total about $800,000 wrong and the wrong clause shifted a renewal deadline by a month. Using a synthetic version of a real engagement, AlixPartners’ Natalia Connolly and Kevin Madura explain how they structured more than 40,000 contracts, embedded business rules in tools and checked results. Their lessons include making access failures explicit and reconciling overlapping savings identified by parallel agents.

1A convincing answer missed $800,000 and a renewal deadlineThe synthetic example’s first analysis got several figures right but omitted an amendment and used the wrong notice date.
2An extracted amendment can still be invisible if it is unlinkedA preprocessing timeout can also leave a contract out of the database entirely.
3Restated totals cannot simply be added togetherIn the speakers’ example, adding successive totals produces about $17 million instead of the latest $7 million value.
4Two weeks of preparation preceded an overnight contract runThe team built a five-stage pipeline and then processed more than 40,000 documents.
5The right date came from a different clauseThe agent used a termination clause when the business question required the renewal clause.
6An access error can masquerade as zero spendingClear error feedback helps an agent stop, recover or ask a person for help.
7Parallel savings findings must be reconciledDifferent agents may identify the same underlying invoice or vendor, so a saving should be counted only once.

À retenir

Check that every document was processed and that amendments are linked to their parent contracts.
Interpret restatements and renewal clauses using the contract text and business context, not numbers alone.
Make data-access failures explicit so a missing value cannot be mistaken for zero.
Reconcile parallel agents’ findings before adding potential savings.
Keep evidence traceable to the contract page or invoice line, and preserve relevant business context.

ThèmesIngénierie d’agents IA en entreprise Data quality & curation Workflow automation

TEXTE INTÉGRAL

Lire le texte remanié, chapitre par chapitre

Afficher la transcription

Transcription seule

The procurement answer that looked correct

A synthetic merger example exposes a missing amendment and an incorrect renewal deadline in an otherwise plausible answer.

Natalia Connolly describes a large healthcare company acquiring a smaller specialist. Its procurement team faced hundreds of vendors, more than 40,000 contract documents and disconnected databases. The speakers use a synthetic replica to illustrate the engagement.

The buyer wants to know whether the combined company can negotiate better terms, estimate savings and act before renewal deadlines. The analysis must be traceable to contract pages and invoice lines so a negotiator can defend it.

The first Claude Code answer looked complete and plausible. Annual spend, the split between companies and the contract count were correct, but the total contract value was off by about $800,000.

DIAPOSITIVE 1 / 7The first procurement answerThe synthetic example shows an incorrect contract value and notice deadline among otherwise plausible figures.

The agent also gave December 1 as the notice deadline when the correct date was November 1. Waiting for its answer would have allowed the contract to renew automatically in the example.

Structure messy contracts before analysis

Unlinked amendments, restated totals and split tables require careful processing. The team built a five-stage pipeline.

Connolly explains that a long contract can silently fail during preprocessing and never reach the database. An amendment can also be extracted but left unlinked to its parent; that orphaned amendment accounted for the missing $800,000 in the example.

DIAPOSITIVE 2 / 7Missing and unlinked contractsA processing failure or an orphaned amendment can hide information from later analysis.

Another trap is adding every value in a contract and amendments even when each amendment restates the total. In her illustration, summing the numbers yields about $17 million, while the latest restatement sets the correct value at $7 million.

Tables that cross page boundaries cause another problem: the second page may lack column headings. The system needs the context of the whole table to interpret the numbers.

The pair spent about two weeks building a processing pipeline, then processed more than 40,000 contracts overnight. Connolly says this preparation made the resulting structured data easier to inspect and verify programmatically.

DIAPOSITIVE 3 / 7The five-stage pipelineThe team structures and maps documents before the analyst agent works on them.

The five stages cover preprocessing and document recognition; document classification; extraction of terms, rates and spending; mapping services to spending categories; and an analyst agent working over the extracted data.

Embed business rules in governed tools

Structured extraction and custom tools let agents work with normalized data and deal-specific logic.

Kevin Madura notes that contracts can include charts and other visual material that naive text recognition misses. Their approach sent page images through DSPy to obtain structured information rather than preserving only chart labels.

When the team knew the engagement needed rate tiers, rebates or other fields, it defined that structure first. Claude Code could then use the normalized contract, spending and invoice data to build or call tools tailored to the deal.

Custom tools such as retrieving a rate-card page or vendor spend can encode business logic and known exceptions. The team favored code and database views for some rules rather than relying entirely on prompts.

DIAPOSITIVE 4 / 7Governed custom toolsTools provide access to selected data and encode deal-specific rules.

The tools and configurations can carry knowledge through an engagement and govern data access. Madura describes allowing a defined set of operations and blocking other calls.

Handle silent failures and overlapping findings

The speakers explain clause selection, missing relationships, access errors, double counting and business context.

DIAPOSITIVE 5 / 7A one-month deadline errorThe example distinguishes a termination deadline from the relevant renewal notice.

The incorrect renewal deadline illustrates why business context matters. The agent had used a date from a termination clause, while the question required a calculation from the renewal clause.

Contract-management databases may omit or mislink amendments. The team therefore reads underlying documents and reconstructs relationships among master agreements, statements of work and amendments, avoiding both omissions and double counting.

DIAPOSITIVE 6 / 7Access failures can appear as zeroMissing access should be reported as an error rather than interpreted as no spending.

Access controls created another failure mode: a denied or malformed response from one company’s data source could look like a zero. Madura says explicit error feedback lets the agent stop, request human help or look for another authoritative source.

The team dispatched agents to investigate different savings opportunities in parallel. Their findings then needed reconciliation: two valid analyses can refer to the same vendor or invoice, and the same saving cannot be counted twice.

DIAPOSITIVE 7 / 7Parallel findings can overlapThe illustrative reconciliation matches underlying invoice lines to avoid counting the same saving twice.

For that reconciliation, the team used a more prescriptive process specifying which data points to compare and what the next action should be. This structured step followed the agents’ more open-ended analysis.

An agent also needs to know what management has already started. A recommendation may look attractive in isolation but be irrelevant if the company began the same work months earlier.

Madura recommends retaining useful business context in memory, tools or similar mechanisms so later analysis does not repeatedly rediscover known limitations. The pair’s main lessons are to structure the data early and provide the right business context.