Keiji Kanazawa rebuilt his agent system after reading its code

SUMMARY

In 30 seconds

After three weeks of agent-led refactoring, Microsoft product manager Keiji Kanazawa found that his supposed agents were ordinary workflows, tests hit paid production inference, and a daily report was empty. He started over with explicit specifications and outcome rubrics while continuing to delegate coding. His video-analysis agent now selects keyframes and explains equations, but even its agent-written rubric overfit one lecture. He tests across examples, revises criteria and has coding agents review each other.

1The “agent” had no agent trace to inspectA misplaced sentence in a digest led Kanazawa to discover that labels in an approved design did not match the implementation.
2Unit tests called production inference roughly 30 times in one dayThe unexpected token bill was about five times his normal weekly spend, and a daily reflection job also produced empty reports.
3He restarted with explicit specifications and outcome rubricsHe retained coding agents but gave the rebuilt system clearer goals and compared four agent frameworks.
4A video agent pairs selected frames and equations with explanationsThe prototype extracted about 440 keyframes from a lecture and supports questions about individual sections.
5The rubric itself overfit the lectureIt performed poorly on five other videos until he directed the agent to generalize its criteria.
6He owns the goal while agents review each other’s codeKanazawa still delegates most implementation and iterates on evaluation criteria rather than reading every line.

Takeaways

Check that implementation matches the meaning of its labels, not just an approved plan.
Separate tests from paid production inference and watch spending alerts.
Describe the outcome with evaluation criteria and revise them after inspecting real results.
Test rubrics on multiple examples to catch overfitting.
Use repeated adversarial reviews between coding agents while retaining ownership of the goal.

TopicsCoding agents Evals Agent engineering Developer workflows & testing Human-AI Collaboration

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

A personal research workflow becomes an agent project

Kanazawa describes his preference for delegating code and how a podcast summarizer expanded into a scheduled research system.

I let coding agents refactor my codebase for three weeks, then read the result. There is a debate between letting agents spend tokens to do work and slowing down to read their code; I am strongly on the first side.

I am a product manager at Microsoft with a background in development and AI research. My busy day job left little time for podcasts, so I built skills to download, transcribe and summarize them. The project grew to crawl X and web pages, accept emailed links and run on a schedule, becoming an LLM-powered wiki.

The original steps were deterministic, such as downloading and transcribing a video. I added more agent-like tasks, including deep research and daily reflection on my Claude Code use, and wrote a specification for a runtime with a registry of scheduled agents.

I had Codex and Claude take turns implementing and reviewing each other's work. Then a digest put a sentence in the wrong section. I asked Claude Code where the agent's traces were so I could diagnose the error and build an evaluation; it replied that the system was not really an agent.

A misplaced digest sentence exposes the refactor

The intended agent was not implemented as one; production-backed tests generated an unexpected bill and a scheduled report was empty.

When I read the implementation, I disliked a design I had apparently approved. I had read plans and accepted familiar terms such as agents and skills, but the implementation did not match those names. The supposed skills lacked skill files and scripts, while the agents did not run on agent SDKs.

SLIDE 1 / 8Labels in the refactored codeThe code treats actions as skills, illustrating the mismatch Kanazawa found.

A provider then alerted me early in the month that I had exceeded my normal budget: about $100, or five times my usual weekly spend. Codex had connected unit tests to production inference, and around 30 test cycles in a day consumed paid tokens.

SLIDE 2 / 8Production inference in unit testsThe slide describes about 30 test runs against production in one day.

I also discovered that the daily reflection job was producing empty reports. During the refactoring, the prompts had stopped running, even though the scheduled task itself still ran.

Restart with explicit goals and compare tools

He rebuilt with a registry, specifications and outcome rubrics, comparing four coding frameworks for his needs.

After three weeks of asking agents to refactor, I did not trust either tool to repair the existing system. I started a new implementation with fresh specifications and outcome rubrics. I still used coding agents: as a product manager, I was not planning to write the system myself.

SLIDE 3 / 8Starting the agent runtime againThe new design calls for an explicit agent, specification and outcome rubric.

The new runtime has a registry of agents and uses an agent SDK. I give the coding agent instructions for an agent that can produce output on a canvas. To choose tools, I asked four frameworks to build the same kind of experience.

I settled on alternating Claude Code and Codex for personal projects. At work I mostly use Copilot CLI. Pi built a good-looking experience quickly, but asked me engineering questions about database and API architecture that I did not want to decide as a product manager.

SLIDE 4 / 8Comparing four coding toolsKanazawa describes why he chose different tools for personal and work projects.

Extract video frames and evaluate the result

A video-analysis agent turns a technical lecture into an explorable canvas. Iteration on the rubric corrects layout and overfitting.

I rebuilt the transcription, summaries and email features. Earlier my YouTube workflow summarized transcript text alone; then I found a blackboard lecture by Reiner Pope about the mathematics of LLM inference, directly relevant to my day job, and wanted to understand the images too.

I built a video deep-dive agent that extracted about 440 keyframes from the lecture and used an evaluation rubric to select useful images and extract equations. Its canvas presents chapters, blackboard material and explanations. I can jump into the video or ask the agent to explain an equation.

SLIDE 5 / 8The video deep-dive agentThe example extracts lecture frames and creates an explorable canvas.

SLIDE 6 / 8Criteria for evaluating the canvasThe rubric asks about explanation, citations, coverage and further reading.

My old workflow gave an agent a specification and approved or rejected its plan. Now I also set a rubric describing the outcome I want. For the video canvas that includes clear explanations, simplicity and links to the correct moments.

I worked deeply on one lecture, then found that the approach could also handle presentations, coding demos and terminal recordings. Agents wrote the rubrics under my direction; I iterated on them and let the agents repeatedly improve the output against the criteria.

Initially the equations were grouped at the beginning followed by a wall of text. I wanted each equation placed alongside its explanation, so I told the agents to change the rubric. I inspect the result, change the criteria and let the agent improve again.

SLIDE 7 / 8Improve output and test the rubricThe loop evaluates generated output and exposes an overfitted judge.

I discovered that an agent-written rubric had been tuned to succeed on that one lecture. It did poorly on five other videos. After I asked the agent to stop optimizing for that single example, the rubric generalized better and scores improved across the set.

Review loops and human ownership

Agents optimize against criteria and review one another. Kanazawa retains control of goals while continuing to delegate most code.

There are nested loops: agents help create the rubric, optimize the output against it, and I revise the rubric based on what I see. I also have Claude and Codex review one another's code and suggest fixes over several adversarial review rounds.

SLIDE 8 / 8Coding agents review each otherA repeated review process has the agents find and fix problems.

I remain on the side of delegating implementation. I own the goal and tell agents what outcome and working approach I want, but still do not read most of the code. For me, spending tokens this way lets a product manager build experiences I had wanted for years.