30 秒でわかる
Coding agents need the same disciplined measurement as other engineering systems, argues Warp founder Zach Lloyd. He proposes defining agent workflows in code, recording their runs and evaluating results so teams can find failures and test improvements. At Warp, another agent can analyze failed runs and propose a change for human review. Lloyd also recommends replaying real company tasks against different models and configurations: public benchmark rankings alone cannot tell a team what works best for its own work.
- 1Individual prompting limits automation and makes cost harder to manageLloyd says individually configured local agents depend on human prompting and make organization-wide cost, productivity, security and governance harder to assess.
- 2Define the workflow in code and improve it in a loopHis software factory specifies agents, triggers, tools and workflows in code, then measures results as work moves from an issue through implementation and verification.
- 3Record agent runs so failures can be investigatedConversations, artifacts and screenshots provide the raw material for improving the system. Sandboxed compute and interchangeable models add repeatability and options to test.
- 4An evaluator can flag redundant tests across runsLloyd demonstrates an evaluation that scores agent output for unnecessary tests. The resulting evidence can help a team ask why the failure occurs and how to change the workflow.
- 5Agents can propose fixes to their own workflowA separate agent examines patterns in failed runs and proposes a pull request against the factory definition, making the change available for review.
- 6Benchmark models and configurations on your own tasksLloyd recommends replaying real tasks against alternative models, agent harnesses and configurations, comparing quality with cost rather than relying on public rankings.
ポイント
- Define agent workflows in versioned code so changes can be tested and rolled back.
- Record agent conversations and artifacts; they become the evidence for diagnosing failures.
- Use evaluations to find recurring problems, then propose a reviewable change to the workflow.
- Compare models and agent configurations on real tasks from your own organization, including their cost.
- Track throughput, cycle time and human interactions per pull request alongside quality.
チャプターごとに整えた全文を読む
Why individually prompted agents are hard to scale
Lloyd says human prompting limits automation while fragmented local setups make cost and governance difficult. He introduces his proposed alternative, a software factory.
I have spent a long time as an engineer, including leading engineering on Google Docs as it scaled to hundreds of millions of users. I now run Warp, which builds tools for companies to create their own software factories and is also known for its agentic terminal. Our aim is to help developers ship better software more quickly.
Most developers I meet are still running coding agents locally, sometimes several at once, through a terminal or development environment. I think that is starting to change. If every agent must be prompted interactively, the amount of automation is constrained by how quickly people can direct it.
Engineering leaders also tell me that when each person chooses a local setup, model and skills, it is hard to know whether agents are being used efficiently in cost and productivity terms. Independent setups can create security and governance problems too.
Define the factory and measure its work
A factory describes agents and workflows in code and improves them in a continuous loop. Lloyd argues that measurable quality, cost and throughput are necessary to make that loop useful.
I do not define a software factory as a loose collection of automations, AI teammates, shared tools or personal agent swarms. Those are useful building blocks. For me, the factory is a code-defined, closed-loop system that continuously measures and improves cloud coding agents.
To scale such a system, define it declaratively: which agents run, what triggers them, what tools they can access, where they run and how work moves through the stages you choose. Think of a set of configuration files in the style of infrastructure defined as code.
The loop might begin with an idea from a person or an automated signal, then take an issue through triage, implementation, verification and monitoring. As agents do more work, the evidence from those runs can help make the loop more efficient.
Everything needs to be measurable. If you choose how to deploy agents by feel, you cannot be confident that you are getting the quality you want at the best cost or maximizing throughput. The engineering problem is how to build a system around these agents so that they can be used effectively.
The tools and infrastructure for improving agents
Evaluations, benchmarks and experiments become part of the engineering toolkit. Lloyd outlines layers for configuration, run data, sandboxed compute, models, orchestration and access.
That calls for a new set of tools. When I learned engineering at Google, debuggers, profilers and browser developer tools were part of the craft. For agent systems, the skills include setting up evaluations, building improvement loops, writing benchmarks and testing alternative configurations.
Product engineering still matters: an engineer remains responsible for whether the shipped product solves a real customer problem. But I think engineers also need to learn how to improve the system that builds the product.

At the base of the infrastructure stack is the factory definition in code. This lets you branch, test alternative configurations and roll back. It also allows coding agents to suggest changes to the way the factory operates.
Above that sits data and context. When agents run across an organization, record their conversations and artifacts, including videos and screenshots. Those records are the raw material for improving the factory over time.
A compute layer needs sandboxing: isolated execution can improve security and repeatability, potentially across multiple platforms. Above it, treat the choice of model as something to test. The system should support multiple models and coding-agent harnesses, because they have different properties and a team may build its own.
An orchestration layer moves work through the factory. The system should also be accessible through APIs and the Model Context Protocol so that other agents can use it and so it can connect to tools where work already happens, such as issue trackers, chat and code repositories.
See and measure every run
A demonstration shows a code-defined configuration and a shared view of agent work. Metrics and online evaluations help identify patterns such as redundant tests.
In Warp’s demonstration, the factory configuration describes policies, repositories, triggers and a blueprint for moving work through stages. Other components in the factory also have code definitions. The idea is the same even if a team builds its own system.

A shared live view shows what agents across the team are doing. I can open an individual run to see where it started, its conversation and the artifacts it produced. The important part is not just watching in real time; it is retaining the information to improve the system.
Once run data is stored centrally, a team can examine cycle time, throughput, cost and the number of human interactions per pull request. Without that information, it is difficult to know how automated the workflow is or where resources are going.

Online evaluations add another layer. As agent sessions run and produce recorded output, another agent can inspect and score them on a chosen dimension. One evaluation we use looks for redundant tests, a failure mode where coding agents write tests that are not needed.

The next question is why the unwanted tests are being written and whether the factory can be changed to reduce them. Record the runs, see where they are in the workflow, collect their output and use evaluations to find opportunities for improvement.
Use failed runs to improve and benchmark the system
Agents can suggest workflow changes after analyzing evaluation failures. Teams can replay their own tasks to compare models, harnesses and other configurations.
We call the next step self-improvement. Rather than having a person inspect every score, an agent looks at failed runs, searches for recurring patterns and suggests a possible solution. It presents that solution as a pull request against the factory definition.

Another essential tool is a benchmark built from your own real tasks. Public discussion often declares a new model, skill or agent harness the best, but to answer that for your own work you need to replay representative tasks and compare the results.

Run the same tasks under different models, harnesses or factory configurations and compare the trade-offs in quality and cost. You can also test whether removing a particular skill makes your factory perform better or worse.
Lloyd says Warp has used this process to improve throughput and lower cost: establish a baseline, score runs, benchmark and test changes repeatedly. His closing advice is to engineer the system around agents, while continuing to apply the engineering skills used to solve product problems.
同じチャンネルの記事
56:08EN
14:28EN
19:08EN
22:25EN
17:50EN
14:07EN