In 30 seconds
A coding agent can pass its tests and still be unable to undo an external action after a failure. In UndoBench, only 47% of trials that had succeeded normally recovered after an injected failure; the researchers found that retrying a timed-out request could create duplicates. Tanmay and Dolly Sah propose capturing the original state and testing recovery before accepting an agent’s self-modification. In a separate EvoUndo evaluation, extending the available undo operations restored 180 of 197 changes with recovery defects.
- 1Task success did not guarantee recoveryAcross 2,880 paired UndoBench trials, 84% of normal runs succeeded, while 47% of the normally successful cases recovered after an injected failure.
- 2A timeout can hide a successful external actionIf a ticket was created but the acknowledgment was lost, retrying can create a duplicate. Naive retries produced duplicate effects in 53.33% of the relevant trials.
- 3Capability gains can come with recovery defectsOf 478 improvements in 600 self-evolution tasks, 197 had recovery defects, approximately 41%.
- 4Undo needs both state information and suitable operationsThe researchers report improved recovery when they supplied precise change information or extended the available undo operations, in separate subgroups.
- 5Richer undo operations recovered 180 of 197 defectsAfter extending the recovery language, the team restored about 91% of defective mutations and increased admissible improvements from 281 to 461 out of 600 tasks.
- 6Verify recovery before admitting a changeEvoUndo records the original state, tests the change and recovery, and uses an independent verifier before allowing an improvement.
Takeaways
- A timeout can mean an operation succeeded but its acknowledgment was lost; checking state before retrying can prevent duplicates.
- In UndoBench, 47% of normally successful trials recovered after a failure was injected; the success and recovery rates use different denominators.
- In EvoUndo, 197 of 478 capability-improving changes had recovery defects.
- Capture the original state before a change and independently verify the restored state afterward.
- Choose among retry, restore, compensation, and escalation according to the effect and whether recovery can be verified.
Read the edited text by chapter
Why a passing test is not enough
Git cannot reverse every effect an agent produces. The researchers test whether agents can recover when execution fails.
Tanmay Sah and Dolly Sah present a missing undo capability for coding agents. An agent might patch a repository, pass its tests, and create a release tag, only for a downstream integration to fail. The question is whether it can return everything to the state before the work began.
Git can show and reverse changes in a repository, but an agent can also make API calls, change runtime configuration, or send notifications. Retrying a failed operation does not necessarily reverse anything that already happened. The researchers therefore ask whether agents can recover, not just finish a task.
In UndoBench, the researchers compared normal execution with runs in which a failure was injected, across 2,880 paired trials. Agents succeeded in 84% of normal runs, but successfully recovered in only 47% of the trials that had succeeded normally. Sah cautions that the percentages use different denominators.

A lost acknowledgment can cause duplicates
A timed-out ticket request may already have succeeded. Retrying safely depends on the phase in which a failure occurs.
Consider a request to create an external ticket. The server creates ticket 781, but its acknowledgment is lost. The agent sees a timeout; that means it did not receive the response, not that the operation failed.

If the agent retries without checking whether the first request committed, it creates a second ticket. In UndoBench, naive retries caused duplicate effects in 53.33% of these kinds of trials. A failed response and a failed operation are different events.
The phase of failure matters. Before a mutation, retrying is generally safe because nothing changed. During a mutation, the system may be partly modified and its state needs inspection. Afterward, an action may already have committed even if its acknowledgment was lost.
An improved agent can be harder to recover
Repository history misses runtime and external changes. EvoUndo records the original state and evaluates recovery defects in self-modifications.
Git revert cannot cover runtime state such as tool registries, middleware, and configuration, or external effects such as deployments and database writes. The recovery mechanism must match the kind of effect the agent produced.
This matters when agents change their own execution environment, or harness: prompts, tools, middleware, event listeners, resources, and routing. A change can improve an agent's capabilities while making its environment harder to recover.
Undo needs information about the original state. If an agent changes a timeout from 30 to 60, reversing it requires knowing the old value was 30. EvoUndo captures a pre-state witness before the change so it can restore and verify that state later.
The EvoUndo evaluation covered 600 self-evolution tasks. Of 478 changes that improved capability, 197 had recovery defects—about 41% of the capability-positive cases. Better performance and reliable recovery did not necessarily arrive together.

Give recovery the state and operations it needs
Repair prompts alone did not recover the defects under the original language. Precise state information and richer undo operations improved results in separate tests.
The researchers tested four verifier-guided repair strategies on the 197 naturally occurring recovery failures. None succeeded under the original recovery language. Sah says some attempts lacked precise information about the prior state; others could not express the operation needed to undo the change.
The team tested the two limitations separately. Providing precise information about what changed and where improved recovery from zero to 38 of 48 cases. Adding richer undo operations improved it from zero to 142 of 143 cases in a different subgroup. Both state information and expressive recovery operations matter.

After extending the recovery language, the team recovered 180 of the 197 defective mutations, about 91%. The number of admissible improvements rose from 281 to 461 of 600 tasks, a 30-percentage-point increase. Sah then hands over to Dolly Sah to explain the process.
Test and verify recovery before admission
EvoUndo captures a state witness, exercises the change and recovery, and independently verifies restoration. Only changes meeting capability and recovery checks are admitted.
Dolly Sah describes EvoUndo's admission process. The agent proposes a change with an effect contract and recovery plan, and the system captures the original state. In a test environment, the change and recovery plan are executed, while an independent verifier checks whether the original state was restored.

A change is admitted only when it improves capability under the user's criteria and passes the recovery check. Otherwise, the agent attempts a repair or the verifier rejects the change. Recoverability becomes part of the decision to accept a change, before deployment.
In the timeout example, EvoUndo first reads and records the original value of 30, then the agent changes it to 60. If recovery is needed, the saved witness supplies the value to restore. An independent verifier compares the resulting value with the original; merely running an undo command is not enough.
Match the response to the effect
Dolly Sah distinguishes retrying, restoring, compensating, and escalating. The right choice depends on what changed and what can be safely verified.
The appropriate response depends on what happened. If nothing changed, retrying may be safe. If the original state is known, restore it; if an external action committed, a compensating action may be needed. If safe recovery cannot be verified or the verifier rejects it, the agent should stop and escalate.
Dolly Sah closes by contrasting the two projects. UndoBench shows that successful execution does not guarantee successful recovery. EvoUndo puts recording, mutation, recovery, and independent verification into a process that accepts self-modifications only when both capability and recoverability requirements are met.
The researchers present EvoUndo as a step toward making recovery a fundamental capability of autonomous agents. Their aim is for agents to recover when actions go wrong as well as carry out the actions in the first place.
More from this channel
58:03EN
19:38EN
19:55EN
23:19EN
17:31EN
17:31EN