Coding Agents Need More Than Git to Undo Their Mistakes

Tanmay and Dolly Sah show why retries can duplicate external actions, and test a way to record and verify the state an agent must restore.

TSTanmay SahSpeakerDSDolly SahSpeaker
Channel
AI Engineer
Published
Nov 8, 2026
Video
16:38
Read
5 min
Slides
5
Original
English
Open on YouTube
Scroll to read and the video shrinks to the bottom-right corner. Captions follow your reading language.
Reading languageAvailable in 2 languages
You're reading the original English.
SUMMARY

In 30 seconds

A coding agent can pass its tests and still be unable to undo an external action after a failure. In UndoBench, only 47% of trials that had succeeded normally recovered after an injected failure; the researchers found that retrying a timed-out request could create duplicates. Tanmay and Dolly Sah propose capturing the original state and testing recovery before accepting an agent’s self-modification. In a separate EvoUndo evaluation, extending the available undo operations restored 180 of 197 changes with recovery defects.

  1. 1Task success did not guarantee recoveryAcross 2,880 paired UndoBench trials, 84% of normal runs succeeded, while 47% of the normally successful cases recovered after an injected failure.
  2. 2A timeout can hide a successful external actionIf a ticket was created but the acknowledgment was lost, retrying can create a duplicate. Naive retries produced duplicate effects in 53.33% of the relevant trials.
  3. 3Capability gains can come with recovery defectsOf 478 improvements in 600 self-evolution tasks, 197 had recovery defects, approximately 41%.
  4. 4Undo needs both state information and suitable operationsThe researchers report improved recovery when they supplied precise change information or extended the available undo operations, in separate subgroups.
  5. 5Richer undo operations recovered 180 of 197 defectsAfter extending the recovery language, the team restored about 91% of defective mutations and increased admissible improvements from 281 to 461 out of 600 tasks.
  6. 6Verify recovery before admitting a changeEvoUndo records the original state, tests the change and recovery, and uses an independent verifier before allowing an improvement.

Takeaways

  1. A timeout can mean an operation succeeded but its acknowledgment was lost; checking state before retrying can prevent duplicates.
  2. In UndoBench, 47% of normally successful trials recovered after a failure was injected; the success and recovery rates use different denominators.
  3. In EvoUndo, 197 of 478 capability-improving changes had recovery defects.
  4. Capture the original state before a change and independently verify the restored state afterward.
  5. Choose among retry, restore, compensation, and escalation according to the effect and whether recovery can be verified.
FULL TEXT

Read the edited text by chapter

01

Why a passing test is not enough

Git cannot reverse every effect an agent produces. The researchers test whether agents can recover when execution fails.

Tanmay Sah and Dolly Sah present a missing undo capability for coding agents. An agent might patch a repository, pass its tests, and create a release tag, only for a downstream integration to fail. The question is whether it can return everything to the state before the work began.

Git can show and reverse changes in a repository, but an agent can also make API calls, change runtime configuration, or send notifications. Retrying a failed operation does not necessarily reverse anything that already happened. The researchers therefore ask whether agents can recover, not just finish a task.

In UndoBench, the researchers compared normal execution with runs in which a failure was injected, across 2,880 paired trials. Agents succeeded in 84% of normal runs, but successfully recovered in only 47% of the trials that had succeeded normally. Sah cautions that the percentages use different denominators.

Normal success and recovery rates
SLIDE 1 / 5Normal success and recovery ratesThe recovery denominator is the 2,406 normally successful trials, not all 2,880 trials.
02

A lost acknowledgment can cause duplicates

A timed-out ticket request may already have succeeded. Retrying safely depends on the phase in which a failure occurs.

Consider a request to create an external ticket. The server creates ticket 781, but its acknowledgment is lost. The agent sees a timeout; that means it did not receive the response, not that the operation failed.

A timeout can create duplicate tickets
SLIDE 2 / 5A timeout can create duplicate ticketsThe slide traces a lost acknowledgment and a retry that creates a second ticket.

If the agent retries without checking whether the first request committed, it creates a second ticket. In UndoBench, naive retries caused duplicate effects in 53.33% of these kinds of trials. A failed response and a failed operation are different events.

The phase of failure matters. Before a mutation, retrying is generally safe because nothing changed. During a mutation, the system may be partly modified and its state needs inspection. Afterward, an action may already have committed even if its acknowledgment was lost.

03

An improved agent can be harder to recover

Repository history misses runtime and external changes. EvoUndo records the original state and evaluates recovery defects in self-modifications.

Git revert cannot cover runtime state such as tool registries, middleware, and configuration, or external effects such as deployments and database writes. The recovery mechanism must match the kind of effect the agent produced.

This matters when agents change their own execution environment, or harness: prompts, tools, middleware, event listeners, resources, and routing. A change can improve an agent's capabilities while making its environment harder to recover.

Undo needs information about the original state. If an agent changes a timeout from 30 to 60, reversing it requires knowing the old value was 30. EvoUndo captures a pre-state witness before the change so it can restore and verify that state later.

The EvoUndo evaluation covered 600 self-evolution tasks. Of 478 changes that improved capability, 197 had recovery defects—about 41% of the capability-positive cases. Better performance and reliable recovery did not necessarily arrive together.

Capability gains and recovery defects
SLIDE 3 / 5Capability gains and recovery defectsOf 478 capability-positive mutations, 197 failed recovery verification.
04

Give recovery the state and operations it needs

Repair prompts alone did not recover the defects under the original language. Precise state information and richer undo operations improved results in separate tests.

The researchers tested four verifier-guided repair strategies on the 197 naturally occurring recovery failures. None succeeded under the original recovery language. Sah says some attempts lacked precise information about the prior state; others could not express the operation needed to undo the change.

The team tested the two limitations separately. Providing precise information about what changed and where improved recovery from zero to 38 of 48 cases. Adding richer undo operations improved it from zero to 142 of 143 cases in a different subgroup. Both state information and expressive recovery operations matter.

Recovery after extending the undo language
SLIDE 4 / 5Recovery after extending the undo languageThe researchers compare 0 of 197 recovered with the original language to 180 of 197 with the extension.

After extending the recovery language, the team recovered 180 of the 197 defective mutations, about 91%. The number of admissible improvements rose from 281 to 461 of 600 tasks, a 30-percentage-point increase. Sah then hands over to Dolly Sah to explain the process.

05

Test and verify recovery before admission

EvoUndo captures a state witness, exercises the change and recovery, and independently verifies restoration. Only changes meeting capability and recovery checks are admitted.

Dolly Sah describes EvoUndo's admission process. The agent proposes a change with an effect contract and recovery plan, and the system captures the original state. In a test environment, the change and recovery plan are executed, while an independent verifier checks whether the original state was restored.

EvoUndo admission process
SLIDE 5 / 5EvoUndo admission processA change is recorded, tested and independently checked before it is admitted or sent for repair.

A change is admitted only when it improves capability under the user's criteria and passes the recovery check. Otherwise, the agent attempts a repair or the verifier rejects the change. Recoverability becomes part of the decision to accept a change, before deployment.

In the timeout example, EvoUndo first reads and records the original value of 30, then the agent changes it to 60. If recovery is needed, the saved witness supplies the value to restore. An independent verifier compares the resulting value with the original; merely running an undo command is not enough.

06

Match the response to the effect

Dolly Sah distinguishes retrying, restoring, compensating, and escalating. The right choice depends on what changed and what can be safely verified.

The appropriate response depends on what happened. If nothing changed, retrying may be safe. If the original state is known, restore it; if an external action committed, a compensating action may be needed. If safe recovery cannot be verified or the verifier rejects it, the agent should stop and escalate.

Dolly Sah closes by contrasting the two projects. UndoBench shows that successful execution does not guarantee successful recovery. EvoUndo puts recording, mutation, recovery, and independent verification into a process that accepts self-modifications only when both capability and recoverability requirements are met.

The researchers present EvoUndo as a step toward making recovery a fundamental capability of autonomous agents. Their aim is for agents to recover when actions go wrong as well as carry out the actions in the first place.

More from this channel

5 slides58:03EN
AI Engineer· Nov 17, 2026How Codex Connects Context, Cloud Work and Agent DelegationDominik Kundel, Charlie Guo · OpenAIOpenAI’s Dominik Kundel and Charlie Guo show how connected context, quick feedback and remote agents fit into real work.Coding agentsDeveloper workflows & testingAgent engineering+2
5 slides19:38EN
AI Engineer· Nov 17, 2026Simon Willison: AI Agents Can Solve More, but Engineering Still Feels HardSimon Willison · DatasetteA beeping detector, a strategy game and a pelican benchmark show how coding agents changed his work in 2026.Coding agentsAgent engineeringAI security+2
4 slides19:55EN
AI Engineer· Nov 16, 2026Brandon Thomas: AI Agents Can Help Rebuild Creative Apps in the OpenBrandon Thomas · ArtCraftArtCraft’s team built an initial Photoshop-like editor in about a day, then enlisted a community to expand open creative tools.Open SourceGenerative mediaSoftware engineering+2
5 slides23:19EN
AI Engineer· Nov 15, 2026Diogo Almeida: Reliable AI Needs a Different Training GoalDiogo Almeida · TypeSafe AIAlmeida argues that models trained to assist people are poorly suited to unattended automation, and calls for software built around reliable intelligence.AI agentsModels & trainingSoftware engineering+2
6 slides17:31EN
AI Engineer· Nov 14, 2026Lauren Tan: Make AI-generated code trustworthy with enforceable checksLauren Tan · SpaceXSIThe Grok Bot engineer describes the architecture, verification and review changes behind her team’s high-volume agent workflow.Coding agentsDeveloper workflows & testingSoftware engineering+1
2 slides17:31EN
AI Engineer· Nov 12, 2026Claude Code needs less steering, but unattended work needs stronger checksLydia Hallie · AnthropicLydia Hallie describes how permissions, isolated environments and verification can take over tasks people once handled during every session.Coding agentsDeveloper workflows & testingAgent orchestration+1