Brian Douglas: Turn agent corrections into reusable review knowledge

I am Brian Douglas, and I want to look at what happens when engineers correct their coding agents. Those moments can contain knowledge that is otherwise left inside a private session.

I can use an agent to produce Rust even though I do not have much professional experience with it. But junior engineers traditionally learn through repeated practice, and delegating those repetitions can leave a gap in judgment.

My open-source Tapes project captures agent sessions. Other telemetry collectors and coding tools can do the same; the point is to retain the interaction data, not to require one particular product. I also use a runtime to make agent environments reproducible.

I have used parallel agents for well-defined work such as documentation, tests and cleanup. That helps with repetitive tasks, but it does not automatically give me the Rust experience to judge every generated choice.

Our team stores sessions centrally and adds labels. One label is pushback: when an engineer corrects the agent. I look for places where my teammate explains an API convention or points out that a service already exists.

Code may compile while still violating decisions the team has made. Because I cannot always spot those choices in generated Rust, I may accept them until a human reviewer points them out in the pull request.

I collect the labeled traces into a dataset and demonstrate a prototype called Twin Review on a pull request. It attempts to offer comments modeled on a teammate’s earlier corrections; the examples come from captured agent sessions.

I initially say the prototype is open source, then correct myself: Twin Review has not been released because its data is sensitive. The broader pattern can be implemented with other tools and an appropriate dataset.

Useful labels include corrections, successful sessions and regressions. A recurring example is an agent recreating a library that already exists; an experienced teammate may notice that immediately, whereas a less familiar developer may not.

My recommendation is to start by turning these traces into skills or instructions and then create evaluations. You can use the results to see whether the skill captures the team’s conventions before deciding whether model adaptation is worth the work.

I have also experimented with adapting smaller models and with preference optimization. Those approaches have different compute requirements; for most teams, I suggest trying the simpler skill first and using whatever session-capture tool they already have.

The practical starting point is to preserve the corrections and outcomes already produced during development. Those records can become examples for a review workflow, a skill or an evaluation rather than disappearing when the session ends.