Uber traded an always-green trunk for faster AI-scale merges

Preetam Dwivedi says Uber has six major monorepos with more than 5,000 contributors, over 36,000 commits and 27,000 pull requests per week. The talk focuses on the Go repository, its largest and a major source of scaling challenges.

Two changes can each pass tests against the same original trunk but fail when combined. If A merges first and B was never tested with A, merging B can break the main branch. At the scale of thousands of developers, such failures impose a continuing recovery burden.

Uber's SubmitQueue tests plausible future states before merging, so that changes can land while keeping the trunk green. It analyzes affected build targets to run independent work in parallel, uses historical results to prioritize likely useful speculative builds, and lets smaller ready changes bypass slower large ones.

Dwivedi says AI-generated work changed the arrival rate: peak merges rose from roughly 90 per hour in January 2026 to 130 in March, a 44% increase. At the same time, the 95th-percentile landing time grew from about 30 minutes to three or four hours, even though the trunk stayed green.

Manjari Akella describes three responses: admission gating to shape traffic, a deferred queue to reduce the burden on developers, and conflict relaxation to recover parallelism. Each comes with a trade-off.

Admission gating classifies requests by cost, for example by the build targets they affect. Small changes can enter during busy periods, while expensive changes that touch many targets are initially rejected so they do not block the active queue.

Akella says this raised business-hour throughput from roughly 130 to 160 merges per hour, an increase of over 20%. The cost was that developers had to find a quiet time and resubmit; maintainers of widely used libraries were especially affected.

Deferral replaces rejection with a persistent waiting queue. Developers submit once, and the system moves expensive requests into the active queue when capacity permits, often outside peak hours. That removes the need to reschedule manually, but it does not eliminate the wait.

The third change addresses how overlapping build targets serialize requests. If A overlaps B, B overlaps C, and so on, a chain of connected changes may be treated as one component, limiting parallel work even when some overlaps are weak.

Conflict relaxation scores the strength of overlap rather than treating every overlap as equally important. SubmitQueue keeps stronger relationships and drops weaker links until it reaches a configured amount of parallelism. The score combines target overlap and newly introduced dependency edges.

In Akella's example, six requests are divided into three independent queues by dropping the weakest links. Instead of continually rebuilding the entire conflict graph, the system assigns the new request's relationships when it enters the queue and preserves prior decisions and speculative builds.

Akella reports that conflict relaxation raised peak-hour throughput from around 160 to about 250 merges per hour, sometimes more. The price is that some combinations land without being tested together, so the trunk can break. Post-merge validation becomes necessary.

Dwivedi returns to the A and B example: each change passes separately but their combined state can fail after landing. Uber uses Stovepipe to validate commits after they merge, detect a broken build and trigger a recovery process.

Stovepipe tracks two references. Main advances as changes land; verified main stays at the latest state known to pass. If commits B through D fail validation, verified main remains at A. When a later state passes, the verified pointer can advance to it.

When Stovepipe detects a failure, it alerts the on-call engineer, who prepares a fix or revert and sends it through SubmitQueue. Stovepipe keeps validating the changing trunk and moves the verified pointer once a state passes. Recovery is currently manual.

Dwivedi reports that by October 2026 peak throughput reached roughly 300 merges per hour, while the 95th-percentile landing time was about 30 minutes. The speakers compare this with the approximately 90-per-hour January baseline. SubmitQueue and Stovepipe are developed in the open.

The team is exploring batching related changes to reduce duplicate builds, reporting health at the service or project level instead of only for the whole repository, and automating fixes or reverts to reduce recovery time.

The resulting strategy is hybrid: test important relationships before merge, relax weaker ones to gain throughput, and validate the landed state afterwards. Consumers can use the verified-good state while the system continues checking main.