Harjot Gill’s AI software factory shifts review to release risk

CodeRabbit’s founder describes thousands of agent-made changes, expensive testing, and a driving license for engineers who ship reliably.

HGHarjot GillInvité · CodeRabbit
Chaîne
The Pragmatic Engineer
Publié
18 nov. 2026
Vidéo
1:27:31
Lecture
12 min
Original
anglais
Ouvrir sur YouTube
En faisant défiler, la vidéo se réduit en bas à droite et continue. Les sous-titres suivent votre langue de lecture.
Langue de lectureDisponible en 2 langues
Cet article n’est pas encore disponible en français ; il s’affiche dans l’original en anglais.
RÉSUMÉ

En 30 secondes

CodeRabbit founder Harjot Gill says coding agents let him merge over 5,000 changes in a month, mostly maintenance and infrastructure work. To make that pace viable, his team gives agents tests, simulations and production feedback, while engineers decide which releases are worth the risk. Developers who repeatedly introduce regressions can lose permission to ship without a human reviewer. Gill reports spending $250,000–$275,000 a month on his own AI use; host Gergely Orosz questions whether that level of spending, or Gill’s interrupted sleep, makes sense for other companies.

  1. 1Most of Gill’s 5,000 changes maintained the factoryThe CodeRabbit founder says many changes fixed testing, caching and deployment bottlenecks rather than adding customer-facing features.
  2. 2Agents need a way to test their own workGill describes formal models, simulated failures, real development environments and staged production releases as successive feedback loops.
  3. 3Developers can lose their license to ship aloneCodeRabbit traces regressions to changes and can require human review again after repeated problems, followed by a proving period.
  4. 4More generated code drives up testing costsGill says projected continuous integration costs reached a vendor quote of roughly $13 million a year, prompting the company to build its own system.
  5. 5Gill spends $250,000–$275,000 a month on his own AI useHe says the company spends several million dollars a month on tokens, before the cost of testing and sandboxes.
  6. 6Orosz questions the sleep and spending assumptionsThe host sees a possible business case in the crowded developer-tools market, but does not recommend waking at night to unblock agents or assume every startup needs the same spending.

À retenir

  1. Give agents ways to verify changes locally, in simulations and after a staged rollout.
  2. Direct limited human review toward the value and risk of a release, with someone accountable for a rollback.
  3. CodeRabbit can withdraw an engineer’s permission to ship without review after repeated regressions.
  4. Budget for test infrastructure and compute as well as model tokens when increasing the volume of changes.
  5. Gill sees rapid spending as a competitive bet for venture-backed startups; Orosz questions how broadly it applies.
TEXTE INTÉGRAL

Lire le texte remanié, chapitre par chapitre

Transcription seule
01

A founder ships thousands of changes with agents

Gill describes his unusually high volume of AI-assisted changes. Much of the work removes bottlenecks in the systems that test and deploy code.

Harjot Gill says he merged more than 5,000 pull requests in a month by keeping coding agents busy. Most were not new product features: they fixed flaky tests, caching, and the continuous integration system that checks code, or removed other bottlenecks slowing his team down.

Gill traces the acceleration to recent improvements in coding models. His team uses several tools, while he works mainly in Codex and says he has stopped using the code editor that was central to his work for decades. He expects the number of changes agents can make to rise further.

The pace has also cut into his sleep: Gill feels pressure to keep agents working because he sees the value they can create when he knows what to build. He describes issuing instructions on a laptop during his commute while his car uses automated driving features.

Faster engineering created a marketing bottleneck. CodeRabbit built a content studio in a code repository, with slide decks represented in HTML and CSS. Agents also propose changes to advertising based on performance data; merging a pull request deploys the changes to Google Ads.

02

From observability to an AI review bottleneck

Gill traces the idea for CodeRabbit to his earlier startups and his experience with GitHub Copilot. He sees coding as a means to test ideas and deliver value.

Gill began in telecom, working on highly available distributed systems, and came to the University of Pennsylvania with the intention of starting a company. His first startup, Netsil, listened to network traffic to understand performance as applications split into microservices. New Relic acquired it, and he stayed to help integrate its technology.

His second company, FluxNinja, worked on preventing cascading failures by controlling load. There, GitHub Copilot gave him an early sense that AI would write much more code. The team was already moving fast enough that review was a bottleneck, even after building a tool to manage stacked pull requests.

Gill says Copilot seemed able to anticipate what he needed and turn comments into substantial code. He and his team founded CodeRabbit around the expected review bottleneck, though he says they underestimated the eventual market: individuals as well as companies wanted AI-assisted review.

Orosz asks whether abandoning a familiar editor was hard for someone invested in the craft of coding. Gill says code was a means to an end: he enjoyed writing it, but enjoys taking an idea to a working product and finding out whether it creates value more.

03

Give agents ways to check their own work

Gill describes formal models, simulations and staged rollouts that let agents improve their output. He says small teams shipping quickly need infrastructure once associated with much larger companies.

As an example, Gill says an agent spent six weeks building a Rust-based system for storing Git data on object storage, inspired by a Cursor post. He says it is now used in production, although he has not read its code. He describes its progress as repeated improvement against a way to check the work.

That feedback can start before code is written. CodeRabbit modeled its Slack bot's distributed behavior with TLA+, a tool for specifying and checking system states, and invested in simulations. Gill says the team has also started using Lean, a proof tool, for areas such as billing.

The next checks happen in emulated services, then real development environments, and finally production. CodeRabbit releases several times a day, Gill says, initially exposing features to a small group and letting an agent watch performance and logs before expanding the rollout.

Orosz notes that large companies built sophisticated rollout systems at great expense. Gill thinks small teams shipping at agent speed now need similar testing, deployment and risk controls. The opportunity, in his view, is to make those tools usable by teams far smaller than the companies that first built them.

04

Running a software factory

Gill separates interactive product crafting from agents pursuing longer-running goals. Local compute, visibility and handoffs limit how many loops his team can run.

For Gill, a software factory is different from a one-on-one chat with an assistant. He compares interactive UI work to shaping clay on a potter's wheel: he makes repeated decisions about the result. He says only a couple of his many sessions work that way; others pursue goals with less supervision.

Agents continuously look for flaky tests, improve deployment processes, and watch the health of components such as billing and knowledge bases. They can add measurements where they cannot see enough, or try a fix. Each test or release offers another chance to learn whether a change worked.

The interface for managing these loops is still unsettled. Gill would like agents to run in cloud sandboxes and be handed between colleagues across time zones, so someone else can answer a question while he sleeps. At the moment, local machines limit how many sessions engineers can run.

Cloud execution also brings setup, security and ongoing costs, Orosz observes. Gill argues that a shared environment would make spending easier to audit: the company can see what work an agent is doing instead of knowing only how many tokens a developer used locally.

05

Human review moves toward release risk

Gill says engineers still decide whether changes should ship and who is accountable for them. CodeRabbit restricts the freedom to ship without review when developers repeatedly cause regressions.

Asked whether code review is dead, Gill draws a distinction: he considers human line-by-line review dead, but says human review remains necessary. AI makes it easier for support staff, designers and background agents to propose changes before a conventional planning process would have prioritized them.

The pull request becomes a decision point: should this change ship at all, or should it be closed? If it is worthwhile, someone must judge its risk. A routine change may move quickly, while a database migration or structural change can require an engineer's attention.

At CodeRabbit, Gill says engineers review releases using risk indicators and decide whether a change can go out or needs to be rolled back. The person making that decision must understand the potential impact and take responsibility if it fails; engineers and site reliability staff stay involved after release.

Human judgment is scarce, Gill says, so it should be spent on the likely blast radius and on experience an automated system may lack. An engineer who knows a large database table, for example, may recognize that a migration could fail in production.

CodeRabbit also gives some developers a 'driving license' to ship without another human reviewer. Eligibility depends on experience and past performance. The team traces fixes back to the changes that introduced regressions and looks at how long a problem was exposed to users.

After enough strikes, a developer goes back to human review for a proving period before regaining that freedom. Gill distinguishes between people who question and check an agent's work and those who give vague goals, fail to follow up, or let it make unnecessary architectural changes.

Orosz compares engineering responsibility to hiring a plumber who will return to fix a mistake. Gill agrees that people must be accountable for how they steer agents, while cautioning that complex software can fail in ways that are hard to anticipate.

If agents ship more changes, operations must keep up. Gill says the goal cannot be to prevent every outage; it is to shorten the time a bad change remains in production. Agents need to inspect logs and traces as well as alerts, which cover only the failure scenarios people anticipated.

He acknowledges that CodeRabbit has not fully scaled this monitoring. Some faulty changes have stayed in production longer because no agent was watching that part of the system. Where monitoring loops exist, he says, the team can also diagnose and resolve issues faster.

06

Managing code quality and compute costs

Maintenance agents clean up a codebase that changes quickly, while tests and infrastructure consume substantial compute. Gill describes the tradeoffs in choosing models and measuring quality.

Orosz raises another quality problem: generated code can be repetitive and leave technical debt. Gill says letting agents ship introduces some disorder, so he also runs maintenance agents to clean up flaky tests, poor abstractions and duplicated work. The team refactors as it learns what structure the code needs.

The cost of running checks has risen sharply alongside the volume of changes. Gill says a continuous integration vendor quoted roughly $13 million a year at the company's projected usage. The team built its own system and uses agents to remove redundant tests while retaining more useful integration and end-to-end checks.

Gill wants checks to complete in about three minutes and says they currently take around four. Parallelizing tests can reduce waiting time but increases compute spending. He also reports trouble obtaining enough CPU capacity in a cloud region, forcing the company to move workloads.

To improve efficiency, Gill says he set an agent a goal of migrating services from TypeScript's previous runtime to Bun over a weekend. He sees infrastructure work with clear constraints and repeatable tests as easier for agents to tackle than product interfaces, where human taste has more influence.

Agents can repeatedly test an optimized container configuration or cache against measurable results. Gill says that with good tests and validation he is seeing fewer outages at CodeRabbit as its codebase becomes easier for agents to work in. He calls this an observation he still needs to measure, rather than a general benchmark.

CodeRabbit must also evaluate rapidly changing AI models for its own product. It uses a combination of models and chooses which ones perform each subtask, balancing quality, cost and speed. Gill says testing that mix consumes enough of the team's capacity that adding another vendor is difficult.

Cheaper models can explore code and prepare context; expensive reasoning models are reserved for deeper work. Gill says open models help put downward pressure on pricing, although his team has limited time to evaluate alternatives. Early fine-tuning work lost its advantage quickly when better base models arrived.

Instead, CodeRabbit invests in the surrounding system that supplies the right context and coordinates the review. Gill says code analysis and graphs help it select relevant information without spending unnecessary tokens. Reviews on public open-source projects give the company feedback to improve that system.

07

What formal checks and AI spending buy

Gill gives a concrete example of finding concurrency bugs in a Slack bot and estimates his personal and company AI spending. He and Orosz debate the economics of venture-backed startups.

Orosz asks where formal methods have actually helped. Gill returns to the Slack bot: its queues, messages and sandbox states produced many possible interactions. Modeling those states with TLA+ and testing different sequences in simulations helped the team find race conditions it had overlooked.

Gill estimates his own AI usage at $250,000–$275,000 a month, with the company spending several million dollars a month on tokens as well as substantial sums on checks and sandboxes. He says moving the check infrastructure in-house may have cut that particular bill by roughly half, although the move was recent.

Gill uses cheaper models where appropriate but is willing to pay for faster, more capable ones when speed matters. He argues that token bills are only part of a software factory's cost: more generated changes also consume test infrastructure, CPU capacity and energy.

Orosz challenges Gill's public suggestion that even a large seed round may be insufficient. Gill narrows his argument to fast-growing, venture-backed companies in competitive markets. A small team can now direct far more spending into agents and supporting infrastructure than it once could into its own coding labor.

Orosz notes that Gill's spending is exceptional and asks how it creates business value. Gill says CodeRabbit can replace some vendor tools with internal systems and learn where customers will encounter bottlenecks next. Building those capabilities for itself can also inform products it might sell.

He expects competitors to copy shallow product ideas in days or weeks, shortening the time a startup has to establish itself. Orosz restates the argument as a bet specific to companies seeking rapid venture-scale growth: pay to learn and reach customers faster, with fewer engineers initially.

Gill compares the possible future of software to mass-produced toys and watches: factories can supply a large market quickly, while handcrafted products may remain a premium niche. Orosz questions whether today's expensive factory will become a cheaper, readily available service, and says he wants to disagree with parts of the analogy.

Competition also comes from unexpected places, Orosz says: established technology companies are turning internal developer tools into products. Gill argues companies must rethink staffing and budgets as agents do more implementation, while Orosz distinguishes new venture-backed startups from established businesses with different economics.

08

Hiring, collaboration and the limits of the pace

Gill looks for initiative and systems thinking and organizes cross-functional rooms to shape products. He and Orosz discuss interrupted sleep, burnout and how employees experience startup work.

Hiring has not slowed uniformly at CodeRabbit. Gill says the company is expanding field sales because customers still need conversations and help setting up systems. In engineering, he is reconsidering which capabilities matter as the tools change.

Gill's hiring post asked for either 1,000 thoughtfully shipped pull requests in 30 days or 500 hours of strategy games. He compares directing agents to managing workers in Civilization: each release is a turn, and a person must select useful goals and coordinate several activities at once.

He says he is looking both at experienced engineers willing to unlearn habits and at recent graduates who take initiative and learn quickly. His argument is that knowing which problems matter gives someone a useful list of work to delegate; he does not treat raw coding speed as the only criterion.

The team also uses in-person 'war rooms': product managers, designers and engineers agree on a plan and shape the product together while one person steers the implementation. Orosz recalls similar rooms at Uber before AI; Gill says the faster feedback now makes this kind of collaboration more frequent.

Orosz asks about the cost of that pace. Gill says agents get stuck and ask questions, and he sometimes wakes at 3 a.m. to unblock them. He describes the habit as unsustainable and says the team has not found the right way for people to share responsibility for running agents around the clock.

Orosz says he prioritizes sleep for both immediate performance and possible long-term health, and asks whether the agents could wait until morning. Gill describes his current pattern as split sleep and attributes it to agents running on his local machine. He hopes shared cloud sessions and shifts might eventually remove that dependency.

On burnout, Gill describes feeling depleted at his previous startup when sustained effort did not produce the desired outcome. At CodeRabbit, he says growth and customer response make hard work feel different. He presents this as his experience and says he does not require employees to follow his schedule.

Orosz asks whether employees benefit from seeing how their work connects to the business, even if they have a smaller financial stake than the founder. Gill says autonomy and visible progress can motivate people, and that he tries to stay involved with engineers and incidents rather than simply manage from a distance.

Both discuss the appeal and risk of startup work. Orosz points to the possibility of financial reward and durable professional relationships even when a company fails. Gill says people can gain valuable experience from taking a risk, including learning what did not work.

09

Skills to build and a skeptical closing

Gill values broad experience and initiative. Orosz closes by questioning whether the sleep schedule and spending levels make sense beyond Gill’s particular situation.

Asked which skills will matter, Gill puts initiative first: he likes candidates who build side projects and can find problems worth solving. He also values breadth and systems thinking, because an agent can help someone with a range of experience work across more areas.

Gill recommends The Black Swan and describes the current period as one in which capabilities and consequences are difficult to predict. He says he would not have expected five years earlier to see AI doing so much of the work he now observes.

After the interview, Orosz says he remains skeptical of waking at night to prompt agents and of applying Gill's spending assumptions to every startup. He sees a stronger case for fast spending in the unusually competitive AI developer-tools market, and singles out the driving-license approach as a way to give engineers a stake in reliability.

Sur la même chaîne

1:03:39EN
The Pragmatic Engineer· 16 nov. 2026Podcast · 2 voixSatya Nadella: When Code Is Cheap, Verification Becomes the WorkInvité : Satya Nadella · Anime : Gergely OroszMicrosoft’s CEO says teams need stronger verification and control over their AI systems as coding agents change how software is built.Software engineeringAI agentsWorkflows de développement et tests+2
1:25:20EN
The Pragmatic Engineer· 10 nov. 2026Podcast · 2 voixSean Allen: Make software correctness part of the designInvité : Sean AllenThe Antithesis engineer argues for earlier guarantees, more thoughtful test workloads and planning for production failures.Workflows de développement et testsSoftware engineeringSécurité+1
1:19:29EN
The Pragmatic Engineer· 5 nov. 2026Podcast · 2 voixScaling Slack and GitHub: Maude Lemaire on Load, Shards and Reliable PlatformsInvité : Maude Lemaire · Anime : Gergely OroszThe principal engineer explains a Slack data migration, realistic load testing, GitHub’s shard constraints and her approach to AI-assisted performance work.Infrastructure et déploiementSoftware engineeringObservabilité et fiabilité+1
1:41:17EN
The Pragmatic Engineer· 28 oct. 2026Podcast · 2 voixTitus Winters on Software Engineering, Teams and CS EducationInvité : Titus Winters · Anime : Gergely OroszA conversation on Google-scale refactoring, testing, AI and how engineers learn.Company BuildingLeadership
1:56:28EN
The Pragmatic Engineer· 21 oct. 2026Podcast · 2 voixWill Larson: The Revised Rules of Engineering ManagementInvité : Will Larson · Anime : Gergely OroszThe 2010s taught engineering managers never to make a technical decision again. Will Larson, CTO of Imprint and author of Staff Engineer, argues that era is over: managers who cannot get back into the work are in a career trap.ManagementLeadershipCareer Growth+2
1:34:17EN
The Pragmatic Engineer· 14 oct. 2026Podcast · 2 voixInside Uber’s Developer Infrastructure for AI AgentsInvité : Uday Kiran Medisetty · Anime : Gergely OroszUday Kiran Medisetty explains gateways, context, background coding and review.Workflows de développement et testsIngénierie d’agentsInfrastructure et déploiement