Walden Yan: Build the Feedback Loops That Let Coding Agents Work

Cognition reports 40 times more merged changes after making its code testable by agents and assigning longer-running goals.

WYWalden Yan연사 · Cognition
채널
AI Engineer
게시일
2026년 11월 18일
영상
14:07
읽기
5분
슬라이드
3
원문
영어
YouTube에서 열기
스크롤하면 영상이 오른쪽 아래로 작아져 계속 재생됩니다. 자막은 읽기 언어를 따릅니다.
읽기 언어2개 언어로 읽을 수 있음
이 글은 아직 한국어 번역이 없어 영어 원문으로 표시합니다.
요약

30초 요약

Cognition says merged pull requests rose about 40-fold in a year while its engineering team grew 2.7-fold. Co-founder Walden Yan attributes the change to more than adopting a coding agent: teams now own customer problems end to end, while isolated virtual machines let agents test changes and send evidence back through Slack. He describes agents improving GPU inference through repeated measurements and building an Android app against an iOS reference over 44 days. The figures are Cognition’s account of its own workflow; merged changes measure throughput, not quality by themselves.

  1. 1Merged changes rose faster than engineering headcountYan reports a roughly 40-fold rise in merged pull requests alongside 2.7-fold headcount growth over a year. He believes quality held up, while the chart measures the number of merged changes.
  2. 2Small pods own the customer problem, not just implementationTwo or three engineers handle bug reports, product decisions and quality checks, delegating described work to agents.
  3. 3Agents test against local services instead of live dependenciesCognition moved services into a virtual machine with mocked external dependencies and rebuilds the environment hourly.
  4. 4Agents can improve the process that runs themA daily channel surfaces proposed changes to build speed, tests and other sources of wasted time.
  5. 5Give agents measurable jobs with reporting and escalationFor latency, Yan proposes instrumentation, alerts and recurring updates rather than a separate prompt for every fix.
  6. 6GPU optimization combined repeated trials with human approvalAn agent tested throughput changes on allocated GPUs and presented its best candidates to an engineer.
  7. 7An Android app used the iOS app as a continuing referenceYan describes a 44-day effort with hooks, feedback and checks for parity; he says the result was planned for release.

핵심 정리

  1. Give agents isolated environments and repeatable tests before expecting them to complete work end to end.
  2. Organize engineers around customer problems, testing and product decisions as implementation becomes more automated.
  3. Turn an ongoing goal into measurements, repeated experiments and escalation points for human judgment.
  4. Cognition’s merged pull-request growth is a throughput measure; assess the quality and impact of changes separately.
전문

챕터별로 다듬은 전문 읽기

원본 자막만 보기
01

A software factory changes the engineer’s work

Cognition reports a sharp increase in merged changes. Yan describes a default agent-driven workflow and smaller teams with wider responsibility.

Cognition's merged pull requests grew roughly 40-fold over a year while engineering headcount grew 2.7-fold. I believe the changes are at least as high quality as before. That is our experience using merged changes as a measure of throughput; the important question is what we changed in our codebase and teams to achieve it.

Merged PR volume and engineering headcount
슬라이드 1 / 3Merged PR volume and engineering headcountCognition’s chart compares changes over roughly one year; the two lines measure merged PRs and engineering headcount.

The basic idea of a software factory is that development runs itself by default. At Cognition, Devin can take a change from a bug report through a merged pull request and subsequent monitoring, escalating when human judgment is needed. That default changes what engineers need to do.

If an engineer only picks a ticket and asks an agent to implement it, I don't think they are set up for success. Once work is clearly described, we can send it to Devin. Engineers need to take responsibility for defining and directing more of the work.

We split larger teams into small pods of two or three engineers and gave them wider ownership as well as groups of agents. Engineers listen to customer bug reports, test the product, do product management and maintain their own list of work for agents to implement.

02

Make testing possible without live services

Agents receive customer reports through Slack and return evidence of fixes. Local services, mocks and hourly VM builds let them test independently.

We work from Slack throughout the process. A customer can send a bug report with a video; we forward it to an internal channel and tag Devin. It proposes a fix and returns a recording as evidence, which we can share with the customer without opening an editor or GitHub most of the time.

That workflow required changes to the infrastructure underneath it. Previously, testing meant connecting to staging services and a database, then having engineers click through the flows. We moved services into a local Docker Compose setup and mocked third-party dependencies so an agent can run the application inside one virtual machine.

An agent-testable codebase
슬라이드 2 / 3An agent-testable codebaseThe slide contrasts tests against live services with a single VM, hourly snapshots and maintained testing skills.

We rebuild these virtual machines hourly against the latest code. If a snapshot build fails, Devin is the first responder and fixes it. An agent started from Slack therefore gets a fresh machine with a recent version of the code and can test without touching live services.

Agents also need instructions for testing the codebase. Our cloud agents write and update their own testing skills as the code changes. As a result, some engineers can review and test work without opening a laptop, and several use iPads in their development workflow.

I think controlled cloud virtual machines can also reduce the risk from secrets left on individual developer machines. Testing need not be simple: when we built voice and Zoom participation, Devin created multiple accounts inside a VM, joined a call and exchanged voice messages to check the feature.

03

Let agents improve the factory

Agents analyze development history and propose improvements to builds, tests and scripts.

A cloud agent setup records code changes, pull requests and decisions. We run agents over that history to find wasted time or tokens, opportunities for scripts and missing tests. They can then suggest improvements to the process that produces the software.

Every day Devin posts ideas to a channel. One suggestion might save a minute in continuous integration by changing the Node setup; another might catch more bugs with a lint script before AI code review. The agents can help optimize the factory, not just work inside it.

04

Move from individual tasks to measurable jobs

Yan suggests giving agents an outcome and the measurement, alerts and reporting needed to pursue it over time.

The next shift is from giving an agent a task to giving it a job. If I care about the 99th-percentile latency of an application—the slow end of users' response times—I want the agent to establish the measurement, create alerts, break down the work and report progress, rather than wait for me to specify every step.

Make a goal measurable wherever possible. That might mean measuring click latency, layout shifts or onboarding retention. If a team can express a useful outcome as a number, I encourage it to set up an agent to work toward improving that measure continuously.

05

Testing GPU inference improvements

An agent tried throughput strategies on allocated hardware and escalated its strongest candidates for human approval.

Cognition received Nvidia Vera Rubin GPUs before inference kernels had been built for them. We allocated a set of GPUs to an agent and gave it a goal: maximize output tokens per minute. It tried approaches over about a week and sent its best candidates to a human engineer for final approval.

Inference throughput experiments on Vera Rubin GPUs
슬라이드 3 / 3Inference throughput experiments on Vera Rubin GPUsThe chart lists successive changes and compares the observed output-token throughput with a GB300 reference.

The agent found eight or nine changes. In the workload shown, the cumulative result moved throughput from below our GB300 comparison to significantly above it. The point is not a single clever prompt: the system had hardware, a measurable target, repeated tests and a human approval step.

06

Maintaining an Android app against an iOS reference

A long-running agent used code changes, customer feedback and QA to pursue parity. Yan says a working app emerged after 44 days.

A second example began because our small team lacked capacity to staff an Android application fully. An engineer, Samuel, asked whether the existing iOS app could act as a blueprint and gave Devin responsibility for maintaining Android parity.

It was not one short session. He set up continuing hooks to watch the iOS codebase, listen for user feedback in Slack, improve the Android app and test it against the existing product. The agent could compare details and escalate the most important decisions.

After 44 days we had a working application that we planned to release. Our timeline shows days with few or no human messages and as many as 40 or 50 pull requests merged. Engineers could inspect recordings in Slack rather than repeatedly opening the agent session.

The lesson I take from this is that agent results depend on the loops around them. With testing, feedback and goals in place, a team can ask agents to sustain work over time rather than issue a fresh instruction for every change.

같은 채널의 다른 글

슬라이드 5장58:03EN
AI Engineer· 2026년 11월 17일How Codex Connects Context, Cloud Work and Agent DelegationDominik Kundel, Charlie Guo · OpenAIOpenAI’s Dominik Kundel and Charlie Guo show how connected context, quick feedback and remote agents fit into real work.코딩 에이전트개발 워크플로와 테스트에이전트 엔지니어링+2
슬라이드 5장19:38EN
AI Engineer· 2026년 11월 17일Simon Willison: AI Agents Can Solve More, but Engineering Still Feels HardSimon Willison · DatasetteA beeping detector, a strategy game and a pelican benchmark show how coding agents changed his work in 2026.코딩 에이전트에이전트 엔지니어링AI security+2
슬라이드 4장19:55EN
AI Engineer· 2026년 11월 16일Brandon Thomas: AI Agents Can Help Rebuild Creative Apps in the OpenBrandon Thomas · ArtCraftArtCraft’s team built an initial Photoshop-like editor in about a day, then enlisted a community to expand open creative tools.Open SourceGenerative mediaSoftware engineering+2
슬라이드 5장23:19EN
AI Engineer· 2026년 11월 15일Diogo Almeida: Reliable AI Needs a Different Training GoalDiogo Almeida · TypeSafe AIAlmeida argues that models trained to assist people are poorly suited to unattended automation, and calls for software built around reliable intelligence.AI agents모델과 학습Software engineering+2
슬라이드 6장17:31EN
AI Engineer· 2026년 11월 14일Lauren Tan: Make AI-generated code trustworthy with enforceable checksLauren Tan · SpaceXSIThe Grok Bot engineer describes the architecture, verification and review changes behind her team’s high-volume agent workflow.코딩 에이전트개발 워크플로와 테스트Software engineering+1
슬라이드 2장17:31EN
AI Engineer· 2026년 11월 12일Claude Code needs less steering, but unattended work needs stronger checksLydia Hallie · AnthropicLydia Hallie describes how permissions, isolated environments and verification can take over tasks people once handled during every session.코딩 에이전트개발 워크플로와 테스트Agent orchestration+1