Zach Lloyd: Treat Coding Agents as a System You Can Measure and Improve

Warp’s founder argues that teams can improve coding agents by recording their work, testing changes on real tasks and measuring cost and quality.

ZLZach Lloyd登壇者 · Warp
チャンネル
AI Engineer
公開日
2026年11月20日
動画
18:30
読了時間
5 分
スライド
6
原文
英語
YouTube で開く
スクロールすると動画は右下に小さくなって再生を続けます。字幕は閲覧言語に合わせて切り替わります。
閲覧言語2 言語で読めます
この記事はまだ日本語に翻訳されていないため、英語の原文を表示しています。
要約

30 秒でわかる

Coding agents need the same disciplined measurement as other engineering systems, argues Warp founder Zach Lloyd. He proposes defining agent workflows in code, recording their runs and evaluating results so teams can find failures and test improvements. At Warp, another agent can analyze failed runs and propose a change for human review. Lloyd also recommends replaying real company tasks against different models and configurations: public benchmark rankings alone cannot tell a team what works best for its own work.

  1. 1Individual prompting limits automation and makes cost harder to manageLloyd says individually configured local agents depend on human prompting and make organization-wide cost, productivity, security and governance harder to assess.
  2. 2Define the workflow in code and improve it in a loopHis software factory specifies agents, triggers, tools and workflows in code, then measures results as work moves from an issue through implementation and verification.
  3. 3Record agent runs so failures can be investigatedConversations, artifacts and screenshots provide the raw material for improving the system. Sandboxed compute and interchangeable models add repeatability and options to test.
  4. 4An evaluator can flag redundant tests across runsLloyd demonstrates an evaluation that scores agent output for unnecessary tests. The resulting evidence can help a team ask why the failure occurs and how to change the workflow.
  5. 5Agents can propose fixes to their own workflowA separate agent examines patterns in failed runs and proposes a pull request against the factory definition, making the change available for review.
  6. 6Benchmark models and configurations on your own tasksLloyd recommends replaying real tasks against alternative models, agent harnesses and configurations, comparing quality with cost rather than relying on public rankings.

ポイント

  1. Define agent workflows in versioned code so changes can be tested and rolled back.
  2. Record agent conversations and artifacts; they become the evidence for diagnosing failures.
  3. Use evaluations to find recurring problems, then propose a reviewable change to the workflow.
  4. Compare models and agent configurations on real tasks from your own organization, including their cost.
  5. Track throughput, cycle time and human interactions per pull request alongside quality.
全文

チャプターごとに整えた全文を読む

文字起こしのみ
01

Why individually prompted agents are hard to scale

Lloyd says human prompting limits automation while fragmented local setups make cost and governance difficult. He introduces his proposed alternative, a software factory.

I have spent a long time as an engineer, including leading engineering on Google Docs as it scaled to hundreds of millions of users. I now run Warp, which builds tools for companies to create their own software factories and is also known for its agentic terminal. Our aim is to help developers ship better software more quickly.

Most developers I meet are still running coding agents locally, sometimes several at once, through a terminal or development environment. I think that is starting to change. If every agent must be prompted interactively, the amount of automation is constrained by how quickly people can direct it.

Engineering leaders also tell me that when each person chooses a local setup, model and skills, it is hard to know whether agents are being used efficiently in cost and productivity terms. Independent setups can create security and governance problems too.

02

Define the factory and measure its work

A factory describes agents and workflows in code and improves them in a continuous loop. Lloyd argues that measurable quality, cost and throughput are necessary to make that loop useful.

I do not define a software factory as a loose collection of automations, AI teammates, shared tools or personal agent swarms. Those are useful building blocks. For me, the factory is a code-defined, closed-loop system that continuously measures and improves cloud coding agents.

To scale such a system, define it declaratively: which agents run, what triggers them, what tools they can access, where they run and how work moves through the stages you choose. Think of a set of configuration files in the style of infrastructure defined as code.

The loop might begin with an idea from a person or an automated signal, then take an issue through triage, implementation, verification and monitoring. As agents do more work, the evidence from those runs can help make the loop more efficient.

Everything needs to be measurable. If you choose how to deploy agents by feel, you cannot be confident that you are getting the quality you want at the best cost or maximizing throughput. The engineering problem is how to build a system around these agents so that they can be used effectively.

03

The tools and infrastructure for improving agents

Evaluations, benchmarks and experiments become part of the engineering toolkit. Lloyd outlines layers for configuration, run data, sandboxed compute, models, orchestration and access.

That calls for a new set of tools. When I learned engineering at Google, debuggers, profilers and browser developer tools were part of the craft. For agent systems, the skills include setting up evaluations, building improvement loops, writing benchmarks and testing alternative configurations.

Product engineering still matters: an engineer remains responsible for whether the shipped product solves a real customer problem. But I think engineers also need to learn how to improve the system that builds the product.

A layered stack for agent factories
スライド 1 / 6A layered stack for agent factoriesLloyd diagrams configuration, run data, compute, model inference, improvement, orchestration and access.

At the base of the infrastructure stack is the factory definition in code. This lets you branch, test alternative configurations and roll back. It also allows coding agents to suggest changes to the way the factory operates.

Above that sits data and context. When agents run across an organization, record their conversations and artifacts, including videos and screenshots. Those records are the raw material for improving the factory over time.

A compute layer needs sandboxing: isolated execution can improve security and repeatability, potentially across multiple platforms. Above it, treat the choice of model as something to test. The system should support multiple models and coding-agent harnesses, because they have different properties and a team may build its own.

An orchestration layer moves work through the factory. The system should also be accessible through APIs and the Model Context Protocol so that other agents can use it and so it can connect to tools where work already happens, such as issue trackers, chat and code repositories.

04

See and measure every run

A demonstration shows a code-defined configuration and a shared view of agent work. Metrics and online evaluations help identify patterns such as redundant tests.

In Warp’s demonstration, the factory configuration describes policies, repositories, triggers and a blueprint for moving work through stages. Other components in the factory also have code definitions. The idea is the same even if a team builds its own system.

The factory definition in a repository
スライド 2 / 6The factory definition in a repositoryThe demonstration organizes agents, automations, benchmarks, scorers and workflow stages as code.

A shared live view shows what agents across the team are doing. I can open an individual run to see where it started, its conversation and the artifacts it produced. The important part is not just watching in real time; it is retaining the information to improve the system.

Once run data is stored centrally, a team can examine cycle time, throughput, cost and the number of human interactions per pull request. Without that information, it is difficult to know how automated the workflow is or where resources are going.

Measuring agent productivity
スライド 3 / 6Measuring agent productivityThe dashboard shows human interactions per pull request, cycle time, pull requests and agent runs.

Online evaluations add another layer. As agent sessions run and produce recorded output, another agent can inspect and score them on a chosen dimension. One evaluation we use looks for redundant tests, a failure mode where coding agents write tests that are not needed.

An evaluation for redundant tests
スライド 4 / 6An evaluation for redundant testsAn example scorer tracks whether an agent’s delivered tests contain unnecessary or trivial cases.

The next question is why the unwanted tests are being written and whether the factory can be changed to reduce them. Record the runs, see where they are in the workflow, collect their output and use evaluations to find opportunities for improvement.

05

Use failed runs to improve and benchmark the system

Agents can suggest workflow changes after analyzing evaluation failures. Teams can replay their own tasks to compare models, harnesses and other configurations.

We call the next step self-improvement. Rather than having a person inspect every score, an agent looks at failed runs, searches for recurring patterns and suggests a possible solution. It presents that solution as a pull request against the factory definition.

Proposed changes from self-improvement runs
スライド 5 / 6Proposed changes from self-improvement runsThe demonstration lists recurring problems and suggested fixes that can be opened for review.

Another essential tool is a benchmark built from your own real tasks. Public discussion often declares a new model, skill or agent harness the best, but to answer that for your own work you need to replay representative tasks and compare the results.

Comparing model cost and correctness
スライド 6 / 6Comparing model cost and correctnessA benchmark plots measured correctness against average cost on the team’s tasks.

Run the same tasks under different models, harnesses or factory configurations and compare the trade-offs in quality and cost. You can also test whether removing a particular skill makes your factory perform better or worse.

Lloyd says Warp has used this process to improve throughput and lower cost: establish a baseline, score runs, benchmark and test changes repeatedly. His closing advice is to engineer the system around agents, while continuing to apply the engineering skills used to solve product problems.

同じチャンネルの記事

スライド 7 枚56:08EN
AI Engineer· 2026年11月21日Ahmad Osman: Local AI Needs Better Systems, Not Just Better ModelsAhmad Osman · OsmanticThe Osmantic founder explains how hardware, tools and task-specific tests determine whether local AI is useful.AI cost & performanceモデルと学習AI agents+2
スライド 3 枚14:28EN
AI Engineer· 2026年11月20日Zach Bruggeman: Ramp Made Its Coding Agent a Shared Team ToolZach Bruggeman · RampRamp made agent sessions visible to colleagues, spreading working practices and giving reviewers evidence for changes.コーディングエージェント開発ワークフローとテストCompany Culture+2
スライド 5 枚19:08EN
AI Engineer· 2026年11月19日Thariq Shihipar: Keep a human-readable layer in AI-built softwareThariq Shihipar · AnthropicA fighting-game experiment gives its creator readable rules to edit while an agent handles the underlying engine.コーディングエージェントHuman-AI Collaboration開発ワークフローとテスト+1
スライド 6 枚22:25EN
AI Engineer· 2026年11月19日Fabian Hedin: How Lovable keeps coding agents working for hoursFabian Hedin · LovableLovable separates agent history from execution, checkpoints working computers, and feeds verification results back into each task.コーディングエージェントAgent orchestrationSandboxes & isolation+2
スライド 5 枚17:50EN
AI Engineer· 2026年11月18日Melvin Johnson: Train coding agents to solve tasks, not game testsMelvin Johnson · Google DeepMindGoogle DeepMind’s post-training lead explains harder tasks, reviewer-style grading and rewards for honest reports.コーディングエージェントモデルと学習評価(Evals)+2
スライド 3 枚14:07EN
AI Engineer· 2026年11月18日Walden Yan: Build the Feedback Loops That Let Coding Agents WorkWalden Yan · CognitionCognition reports 40 times more merged changes after making its code testable by agents and assigning longer-running goals.ソフトウェアファクトリーコーディングエージェントDeveloper productivity+2