Zach Lloyd on the Infrastructure Behind a Software Factory

Shawn Wang introduces Zach Lloyd of Warp to discuss software factories: cloud infrastructure for teams using coding agents. Lloyd says he wants to make the concept concrete and measurable rather than treat it as a vague label for using more agents.

Lloyd defines a software factory as a cloud-based agentic system with specified roles and workflows for building a product. The system has some autonomy, but teams should be able to measure, tune and benchmark it over time.

Companies often arrive with disconnected automations. Lloyd argues that the factory itself should be defined in code, including its repositories, skills, MCP integrations and workflow. Warp’s platform associates a factory with a product surface, such as a mobile or server application.

A workflow might include triage, specification and implementation. Warp represents the configuration in a YAML file. Lloyd distinguishes product engineers, who submit work and ship features, from people engineering and improving the factory itself.

Work can enter from Slack, a task tracker, GitHub, an API or MCP. Whatever the entry point, the task starts a defined workflow. Lloyd’s goal is to bring those different activities into one observable system.

The system collects agent conversations, artifacts and traces. In a demonstration, Lloyd shows a mobile task initiated in Slack, including the conversation, computer-use screenshots and the steps taken by agents. A record across many tasks provides data for improvement.

Teams can examine delivery time, human interactions per merged pull request, throughput and cost. Lloyd says Warp uses cost per PR and slices spending by person, automation or development stage to find where models and workflows can be tuned.

Another layer evaluates agent behavior. Warp supplies general scorers, and teams can define their own for specific failure modes, such as changes that break a build. The rules are customizable and represented in code.

One example looks for redundant tests that an agent wrote while exploring a solution and then left behind. An LLM judge reviews sampled historical runs against a prompt. The sampling rate matters because running the judges also costs money.

Wang asks who uses these views. Lloyd says they are often for an engineering manager, CTO, developer experience group or platform team, rather than every product engineer in daily work.

His general principle applies beyond Warp, he says: store the relevant work and make it observable and measurable. That allows teams to compare models and harnesses based on their own work instead of relying only on impressions or social-media recommendations.

A separate improvement loop can inspect failed scores and suggest changes. In one example, the system identified triage cases where a subagent might have helped. Since the factory configuration is code, an agent can propose a pull request to change it.

Lloyd says a proposed next step would replay the historical cases to test whether the suggested change actually helps. He describes that replay capability as future work, not a feature already available in the demonstration.

For benchmarking, teams can curate real tasks, such as front-end changes, and replay them across a matrix of models and harnesses. Lloyd says the platform supports comparisons with several external harnesses as well as Warp’s own.

The example plots correctness against average cost. Lloyd compares one configuration near 95% correctness and $6 per task with another near 93% and $2. Those numbers apply to the demonstrated task set, and the screen notes that some scores are incomplete.

A team could use its own benchmark results to route tasks to different models or harnesses. Lloyd calls this factory engineering: treating the agent setup as an engineering system rather than guessing which configuration is best.

Wang asks about the infrastructure stack behind the factory. Lloyd starts with a declarative definition in code, analogous to infrastructure as code. Version control lets teams branch, test and revert workflow changes, while agents can propose improvements.

The next layer collects conversations, artifacts and interactions across tools. A task might start in Slack, continue in a tracker, produce a GitHub review and eventually cause a production incident. A complete record can help a person or agent investigate what went wrong.

The system also needs integrations and access controls. Larger organizations may need different permissions for marketing, sales and engineering. Lloyd stresses that deployment becomes more complicated as organizational requirements grow.

He recommends treating compute as a pluggable layer: teams may choose different sandbox providers or platforms. Linux sandboxes can cover many tasks, while an iOS workflow may need a Mac virtual machine and simulator.

Inference and harnesses can also be replaceable. Lloyd says customers may bring existing model-provider commitments, purchase inference through Warp or use compatible external harnesses. The goal is flexibility as models and tools change.

For the improvement layer, Lloyd reports that Warp reduced its own cost per pull request from roughly $80 to $15–20 by adjusting model choice and avoiding redundant subagent work. He says throughput rose 40% over a month while the measured revert rate did not rise.

An orchestrator reads the factory definition and moves work through stages, invoking models, skills and automations. Lloyd views it as one component of the platform rather than the whole platform.

The final layer is access: people should be able to start work from the tools they already use. Lloyd says a large share of Warp’s own work starts in Slack, but a task tracker, terminal, API or MCP can also provide an entry point.

Wang asks whether a team really needs all these layers. Lloyd says a few people may find a full factory excessive and should use simpler tools. He suggests roughly five to ten engineers as a possible lower bound, while noting that suitability depends on the team.

Larger companies tend to have more bespoke infrastructure and require implementation work. Some growth-stage teams using common collaboration and task tools may find setup easier. A shared record can connect feedback, incidents and the right colleagues to the work.

Asked for the strongest criticism, Lloyd points to implementation fit and economics. Very large companies with substantial internal developer platforms may build their own systems, while small teams may find subscription tools more economical.

The speakers discuss how subscription pricing can differ sharply from API-based usage. Lloyd says he does not know whether every practice companies use with multiple subscriptions is allowed; the economics and applicable terms depend on the actual provider and plan.

Lloyd says Warp is positioning its factory product around optimization and productivity rather than reselling tokens. He sees model-provider pricing as a difficult competitive challenge for independent coding-agent businesses.

For his upcoming talk, Lloyd plans to distinguish product engineering from factory engineering. The latter develops the skills and tools to improve the system that helps build software, alongside the work of building the product itself.