Dmitry Buykin: How to Maintain Code When Agents Outrun Review

Coding agents made code cheap, but they also made disorder cheap. When my project grew beyond what I could understand, I deleted unused tests and boilerplate and rewrote it. I lost useful pieces, and the mess returned as agents kept writing. The alternative I found is a small weekly cleanup supported by feedback built into the system.

To stay below the point where changes exceed our ability to check them, we first need to know what correct means. Consider changing an amount on an already approved order: one agent authors instructions, another follows them in production. Our system spans 110 countries and roughly 10,000 scenarios. A change normally stays in one slice: one process in one country.

I estimate three quarters of the system is prose, so that prose needs engineering too. We tried mutation testing: deliberately break a sentence and see whether tests fail. Almost none did. Rewriting is easy for agents; knowing what must survive is harder. Tests can help with code, but people must judge the business instructions.

Agents tend to fix visible problems while breaking consistency, and people get tired too. We reserve expert judgment for the places that need it. When a business instruction says to check with a team, developers examine traces with coding agents to determine whether the step can be automated. In the order example, the team lead's decision remains human because of the data we can connect.

Business experts know the rules; developers see the underlying system connections. We have agents record those connections so the runtime can check them. The guide to experts puts the division simply: business tells us what; developers work out how.

We keep one document for two readers. Agents need exact instruction blocks, while experts review and sign the surrounding explanation. A second copy would drift. Experts also need a plain description of code and see each task as the agent saw it; their notes become the next change, tested on that same task.

Experts correct individual tasks while agents use the same instructions all day. We track accepted and rejected outcomes to find where people still need to look. One instruction might set a limit at 1,500 while code says 1,000: the same order can be accepted in one place and rejected in another without an error.

Even a prohibition can expose a tool to the model simply by naming it. Prose has no compiler, so we built checks that lint changed files into a ledger. Lint cannot stop copies drifting, however, so we also watch an agent's path through the instructions.

When the same path repeats, we turn it into a workflow tool and remove the prose that walked those steps. Our first tools made matters worse because the prose still named their internal steps, causing agents to run them again. Now prose retains decisions; a path becomes code only when it repeats and agrees across countries.

What varies remains readable prose inside a developer-owned frame. Code prepares facts and executes the outcome; experts own the decision table, and the model selects a row. If none fits, a person takes over. Connecting small model decisions through workflows is a next step, not something we have built yet.

The boundary can move in either direction. We moved hundreds of lines of expressions from code into prose because experts could not read them. A regular expression can still spot a critical signal, but a larger collection belongs in configuration. Judgment stays where answers vary and experts can inspect them.

Control starts with what the model sees. When we let it choose what to load, it loaded everything and invented a pricing check because prices were visible. Now the runtime selects only the facts needed for the task: three order facts in this example, while it retains the full record.

For a large data set, the model passes a short handle to data the runtime already holds. The runtime validates the handle before a write and checks consistency, keeping context small. A model with the right facts can still cause damage, so changes are controlled during development and again at runtime; code opens the write path.

Several safeguards stand before a write and another checks that it landed. If a write appears to fail and the agent retries, a journal ensures it runs only once. Control therefore involves more than assigning permissions to an agent.

At first I read every change, and the system picked up my habits. As volume grew beyond one person's capacity, I burned out and stopped reading every diff. Agents can make a technical lead the bottleneck in weeks rather than years, so I needed a process that did not run through one person's head.

My first gauge is whether I am too tired to explain what changed. A dashboard may be green while that warning is active, so I stop and fix the process. I also built instruments to watch changes and their effects. Before introducing one check, we replayed it against several thousand past sessions; it would have fired on less than one percent.

When one signal looks wrong, we wait to see if it returns. A cluster across slices calls for fixing the underlying concept. An agent left alone may add conditions until today's tasks pass; removing them exposes gaps. An expert's note becomes an evaluation, and an agent searches related country slices for the same failure.

Those related slices differ deliberately by market and local law, so the fix goes only where the failure is confirmed. Ten to twenty feedback loops can be open at once. Users need a screen to evaluate outcomes directly rather than relying on me to carry messages between them and my agents.

Each loop records its work in a ledger, and agent memory lives in files because sessions end. We distinguish what happened from what is true now; the next session starts with the current state. Issues must connect to their root cause so agents and checks act on evidence rather than wrong facts.

Checking is the limiting step. My agents spend almost nine tenths of their time on a change verifying it, and it takes a larger share of my week too. We rank evidence by what it proves: replaying real tasks is strongest. Before results arrive we write down what would disprove us, and a new check must deliberately fail once before we trust it.

AI refactors can quietly drop a branch. In the order example, losing the over-limit branch would let an order through without a team lead. We freeze real tasks and replay them against old and new versions. If no real task reaches a branch, that is no evidence about the branch.

A second agent checks work against a checklist without seeing the author's reasoning. Over six months it grew to do most of my checking. Agents can also quietly skip the end of a batch when their context fills: in one backlog, items looked mostly covered, but only about a quarter could actually be closed. The ledger exposes the gap.

The choice is not between keeping everything and deleting everything at once. Across a year of one process, some branches merge, some drift, and some are dead but appear alive. A single cleanup does not stop regrowth; the forces that produced the mess keep working.

We give each kind of growth a gauge, a threshold and a ratchet: a count that can only go down. Two gauges run today and six more are planned. Each week real use decides what stays; before removing something no real task reaches, we name it and record the evidence.

Tests can keep retired variants alive because agents preserve every option a test still mentions. Now a retired variant takes its test with it. Small weekly sweeps use the evidence I lacked during the original rewrite, and I have not had to make that wholesale deletion again.

After production, I saw my own principles embedded in the design. The system's skills and memory carry my habits and biases: it copies whoever corrects it. I use three questions for each of the six areas and suggest starting with one your repository cannot yet answer. Build the feedback loops while you can still read the changes.