Ben Holmes: Use Agent Traces to Improve Agent Skills

Ben Holmes, a developer-experience engineering lead at Warp, describes agents that review their own past work and suggest changes to their configuration. His focus is the improvement loop around a software factory, not just the agents that implement code.

A software factory automates stages of development: taking tasks from a board or messages, researching, writing a specification with a person, implementing, capturing evidence and reviewing a pull request. Monitoring feeds the next iteration.

Holmes calls the work of improving that system factory engineering. Teams might refine skills, code architecture or static analysis; this talk focuses on skills and configuration that shape agent behavior.

The proposed loop collects conversations, scores them against team-defined rubrics, groups failures and has another agent propose changes. After a human reviews those changes, the team can continue measuring whether results improve.

The first step is getting agent conversations into one place. Local tools keep histories on individual machines; a team may centralize them, or use a shared cloud runner whose isolated jobs produce traces in a common system.

A useful trace includes more than messages: capture tool and filesystem calls, external integrations and their outputs. Keep the resulting code diff or pull request too, so a reviewer can connect what the agent did with what it produced.

Holmes suggests scoring redundant tests when teams see agents adding tests that do not match their coverage strategy. A rubric should reflect the team’s actual preferences for unit, integration and end-to-end coverage.

Another criterion is verbosity: excessive comments or unclear summaries can slow code review. The goal is communication that helps a reviewer understand a change without unnecessary noise.

Efficiency requires looking at tool calls, not just the final answer. Holmes says agents can persist through outdated instructions, repeatedly probe the environment and eventually succeed without reporting the wasted work.

He shows an LLM-as-judge rubric in Markdown. It defines the desired behavior, score labels and what counts as a failure; he recommends starting with a concise set of criteria and improving it through use.

For each sampled conversation, a scoring agent evaluates the chosen criteria. In his example, excessive unit tests and time spent discovering the test environment fail their respective rubrics, while clear communication passes.

Scoring can be triggered after a pull request is merged, with sampling used to control token cost. Each report should explain why it passed or failed, giving the next agent evidence rather than just a number.

A scheduled improvement agent can inspect recent failures, find recurring causes and propose changes to skills or code. Holmes illustrates a daily run that receives failure reports and access to the relevant repository.

For an individual setup, he demonstrates Skill Doctor, a one-time analysis of local agent histories. It scores sessions in batches and produces a report of patterns and suggested configuration changes.

One example finds repeated struggles with occupied ports and recommends improving test-environment instructions. Another captures a recurring preference about changelog language, reducing repeated back-and-forth.

Holmes argues that a visible history of scores helps teams understand what a change did. A dashboard can show efficiency over time and mark when a skill change was merged, although a correlation alone does not prove causation.

In Warp’s setup, agents score conversations continuously and a scheduled process proposes pull requests for human review. One reported fix corrected configuration that had prevented a triage agent from spawning subagents.

Holmes closes by inviting teams to try the workflow or run Skill Doctor locally. The practical idea is to expose patterns hidden inside successful-looking runs and turn repeated friction into reviewable improvements.