Zach Lloyd: make the software factory measurable and editable

Zach Lloyd, founder and CEO of Warp and a former Google Docs engineering lead, describes software factories as infrastructure for development with agents. Warp sells a factory platform, which he uses for his demonstration.

Lloyd says many companies are still at interactive prompting: engineers direct local agents to build features and fix bugs. They want more automated development that delivers business value with manageable cost and governance, but a person can become the bottleneck.

He argues that a compromised local agent may have the same access as its user, while individually configured tools and skills are difficult to audit. His proposed approach is to sandbox agents and run them in the cloud.

In his model, people submit work and agents participate in triage, specifications, implementation and verification. Organizations define how people and agents interact, then measure and improve the workflow over time.

Lloyd recommends treating the factory as an infrastructure stack, open to different models and agent frameworks and adaptable to existing systems. Define workflows, skills, tools and execution environments in version-controlled code, so versions can be tested and agents can propose changes.

He says larger customers want control over their input and output data, infrastructure and inference, with the ability to choose models and frameworks. He calls this AI sovereignty and argues that companies should own the core ingredients of the system.

Public benchmarks are not enough to select a model for a particular organization, Lloyd argues. Record real tasks, score agent runs, use the results to find failure patterns and build benchmarks that compare configurations on the company's own work.

He distinguishes product engineering, where people use the factory to build features and fix bugs, from factory engineering: improving skills, context and model choices so the system itself performs better. Platform teams or product engineers can do that work.

In a Warp Factories demonstration, Lloyd shows a manager's view of productivity, automation, cycle time and spending. The system records agent runs, conversations and artifacts, which become material for examining and improving the process.

Engineers can submit work from tools they already use, such as Slack, Linear or GitHub. Lloyd says the factory collects the work centrally while preserving its context. An orchestrator routes work through steps such as triage, specifications, implementation and review.

The factory's repository contains agent definitions, automations, benchmarks, execution settings, skills and webhooks. Lloyd says versioning this definition makes the system itself available for testing and automation.

Warp uses scoring agents to inspect completed runs, including a check for redundant tests. Lloyd describes this as an online evaluation: identified failures become candidates for improving the process. The scoring shown in the demo relies on a language model as judge.

A self-improvement loop aggregates recurring failures, looks for patterns and proposes changes to the factory definition. In one example, the system reports 37 failures associated with agents not reading relevant skills and proposes a pull request to address the pattern.

Lloyd recommends building a benchmark from representative past tasks, then replaying them with different model configurations. Rather than relying only on public leaderboards, a team can compare quality and cost for its own code and workflows.

In his example, GLM 5.3 Flash scores about 93% at roughly $2, while GPT 5.6 Sol scores around 95% at roughly three times the cost. These are scores on the displayed internal task set, not a general comparison; Lloyd later explains they use a model as judge.

Lloyd summarizes the stack as factory definitions in code, data and context, sandboxed compute, flexible inference, measurement and improvement, orchestration and access through existing tools.

Asked about verification, Lloyd recommends specifications that changes can be checked against and computer-use verification that produces evidence from the user's perspective. He suggests these can reduce the burden of reviewing a pile of generated pull requests.

On budget planning, he suggests measuring cost per pull request and separating the share spent on scoring and benchmarking, then estimating project costs. His main recommendation is to build a process that can be measured and improved.

For benchmark curation, Lloyd recommends starting with a hand-picked representative mix of small, medium and hard tasks. A factory with stored sessions can also suggest past runs to include. He says the demonstrated correctness chart uses an LLM judge; human grading would also make sense but was not yet added.

Asked whether Warp Factories is open source, Lloyd says it is not, though it accepts existing tools and is not restricted to enterprises. He thinks the platform is most useful for teams and may be excessive for an individual.