How Duolingo uses agents to add tests without skipping human review

Duolingo uses coding agents to write unit tests for its iOS app, then routes their pull requests through automated checks and human review. Staff engineer Kush Agrawal builds developer tooling for more than 100 engineers contributing to the company's client apps.

The company releases mobile apps weekly and runs more than 600 live A/B tests, Agrawal says. Its iOS and Android apps are more than a decade old; the iOS repository has grown from roughly 200,000 executable lines in 2019 to nearly two million.

More people are contributing to iOS, including backend and Android engineers, product managers and designers. Agrawal credits AI coding tools with helping make that possible. About half the repository follows a model–view–viewmodel, or MVVM, architecture; the project focused there because that code was easier to test than the legacy half.

In March, overall unit-test coverage was 11%. Repositories were at 11%, view models at 9% and data sources at 5%; together the MVVM slice was at 9%. As humans and agents produce code faster, Agrawal says, verifying a change is becoming a bigger bottleneck than writing it.

Tests give agents feedback when changes break continuous integration, or CI, and document expected behavior. Eight months later, reported overall coverage had reached 24%; repositories reached 72%, view models 58% and data sources 41%, bringing the combined MVVM slice to 59%.

The team's loop is to identify files that need tests, have an agent write them, verify quality and CI, get an engineer to review and merge, and incorporate feedback. In production, several Temporal workflows coordinate through shared state in S3 rather than one simple script.

Temporal orchestrates the workflows, Claude Code writes code, S3 keeps shared state, GitHub handles pull requests, labels and actions, and Datadog provides observability. Agrawal describes Temporal as a durable execution engine: if a worker crashes, the workflow can resume, with scheduling, retries, timeouts and idempotency supporting parallel work.

The team split the project into four milestones: a local feasibility test, scheduled backfilling of tests for existing code, a pull-request lifecycle manager, and a label-triggered path for tests on new code. Each milestone tested a different part of turning agent output into reviewed changes.

The local prototype ran an Xcode coverage report and a file selector, then delegated test writing and pull-request creation to subagents. In its first batch, eight of 17 pull requests merged; in the second, 19 of 40 did. The roughly 50% merge rate mattered less than learning where agents failed before building production infrastructure.

Agents sometimes mixed XCTest and Swift Testing in the same file, so the team supplied explicit rules. A large singleton was also difficult to mock when passed as a concrete class. The team migrated callers to a protocol-based interface and added a lint rule, making those files easier for both people and agents to test.

Concurrency issues and timing-based tests were another source of CI flakiness. The team improved the examples and rules in its testing skill. With those lessons, it moved to a scheduled workflow to add tests for code already merged.

After each merge to the iOS main branch, a GitHub Actions job generates an Xcode coverage report and uploads it to S3. The backfill workflow reads that report and runs a selector that ranks Swift files according to how confidently an agent can test them.

File type is the strongest selection signal: view models, repositories and data sources receive a 2.5-times boost, while legacy code ranks lower and large files are penalized. The team skips UI code for now, considering snapshot tests a better fit. The aim is to prioritize files likely to yield useful tests.

A child workflow runs for each selected file in parallel. Its agent clones the repository, reads the repository's testing skill, writes tests and any necessary mocks or fakes, then opens a pull request labeled as a generated test. The company's cloud agent platform wraps Claude Code and Codex, connects internal tools, and records activity and cost.

Generated pull requests clearly identify themselves as AI-generated and use a standard description. The assigned engineer can approve for auto-merge, push changes, reassign the review or close the request with feedback. Human approval remains part of the process.

Once the backfill produced tens to hundreds of open pull requests, the team added a lifecycle workflow. Every couple of hours it inspects open generated-test requests and takes one action per request, working toward green CI, quality checks and a human review.

The workflow closes requests with merge conflicts or those open for about three weeks. It waits if CI is still running or a reviewer is already assigned, assigns a reviewer when CI is green, and tries to repair failures. A rejected file enters a 30-day cooldown to avoid repeatedly sending engineers the same request.

For a pre-commit failure, a deterministic GitHub action applies the fix without invoking an agent. For an actual test failure, another agent reads CI logs and pushes a repair. The system stops after five attempts and closes the request instead of retrying indefinitely.

When CI passes, the workflow uses git blame on the source file to find a leading contributor, assigns that person and enables auto-merge. Approval merges the pull request; rejection triggers the cooldown. For newly written code, an engineer can add a write-test label to their own pull request, invoking the same infrastructure on its changed files and receiving a linked draft request.

The first agents ran on Linux machines without Xcode or Swift, so lint problems often surfaced only after a 20- or 30-minute CI run. Agrawal says 13.6% of requests failed on lint for a period and two rules accounted for 62% of those failures. The team improved prompts, auto-fixed eligible lint issues and removed low-signal rules that could not be fixed automatically.

Moving agents to Mac runners let them build and lint locally before opening a pull request. Quality remained a separate issue: agents sometimes changed production code to make tests pass, altered or disabled existing tests, or inserted long waits that could make CI flaky.

The first quality guardrail is deterministic: close a request touching anything other than a test, mock, fake or stub. Duolingo also uses Claude and Codex to review pull requests, and an agent resolves P0 review comments before a human sees the change.

A dedicated language-model judge reviews the test pull-request diff for production edits, changed or deleted tests, disabled tests and long waits. Agrawal says it rejected 73 requests in its review set. Each rejection explains why; the team feeds that information back into prompts and skills to reduce repeat mistakes.

Since March, the pipeline has merged more than 700 test pull requests and added about 225,000 lines of test code, Agrawal says. Every merged request received human review and approval. The MVVM slice rose from 9% to 59% coverage and the app overall from 11% to 24%.

The pipeline produced 34% of new test files, 44% of new test cases and 10% of all new Swift lines since March, according to Agrawal. Its share had recently fallen as engineers wrote more tests themselves, which he sees as the desired outcome.

Agrawal says the identify–write–verify–resolve–iterate loop is not specific to unit tests. Duolingo is starting to reuse it for screen modernization to MVVM and SwiftUI, Android coroutine migrations and snapshot-test generation.

As a future experiment, he asks whether a fast general-purpose classifier could replace the handwritten lifecycle decision tree: take the pull-request state, choose the next action and execute it in milliseconds. He presents this as an idea, not a system already in production.