Nouha Dziri on Making Coding-Agent Teams More Cost-Efficient

Nouha Dziri, a senior researcher at Cohere, presents work on making teams of coding agents more effective without allowing their costs to grow unchecked. Instead of relying only on longer reasoning or many independent attempts, a lead can split a task among workers with separate contexts and combine their reports.

Long coding tasks can fill a single agent's context window and take a long time. Independent samples repeat the whole task, whereas workers can explore separate pieces and return compact findings. That advantage depends on the work being decomposable and the pieces sufficiently independent.

A team also raises coordination and spending questions. Adding agents can quickly expand the token budget; the team must decide who leads, which phase gets help, how workers communicate, how many to use and which models should play each role.

In an initial open-ended setup, the agents did not automatically form a clean team. They often tried to build separate versions, communicated through the lead and sometimes duplicated work or interfered with one another. Dziri's team therefore studied the harness and instructions explicitly.

The researchers frame team design as compute allocation: how much additional task reward does the next dollar buy compared with a solo agent? They compare worker stage, model, team size and lead model, seeking higher reward without excessive extra inference cost.

Their minimal hierarchy gives a lead a delegation tool and workers that operate through a shell in a shared workspace. The lead decides how many workers to spawn and what they do, receives their reports, consolidates results and can supervise, resume or stop workers.

The team compares a blocking lead, which waits for all delegated work, with an asynchronous lead that continues working while reports arrive. In the initial setup, the latter sometimes implemented its own guesses, ignored late findings or submitted before workers finished.

ProgramBench supplies a black-box executable and documentation in an otherwise empty workspace. An agent must probe the program, discover its interface and edge cases, and implement a compatible version. Dziri calls the tasks hard and cites an FFmpeg task on which the leading model passed only a small share of tests.

With minimal instructions, a team led by a blocking agent beat a solo agent by nearly ten points, but cost roughly four to seven times as much. The blocking variant outperformed the initial asynchronous setup, although it also cost more and took longer.

The researchers then compared minimal instructions with a much more detailed planning, implementation and verification prompt. Stronger guidance improved both solo and team performance. It also taught the asynchronous lead when to wait, allowing it to catch up with the blocking variant in this experiment.

Even with better prompting, a full team could cost six to seven times as much as a solo run. Dziri asks where extra inference spending actually helps, and compares assigning workers only to planning, only to implementation or only to verification.

In the planning-only setup, workers probe the executable and report plans; the lead implements and verifies. In implementation-only, the lead plans and divides coding work. In verification-only, the lead plans and implements before asking workers to check the result.

On ProgramBench, planning workers achieved almost the same reward as workers used throughout the process at about half the cost. Dziri attributes this to the benchmark's bottleneck: discovering a black-box program's behavior and edge cases before implementation.

Her efficiency comparison finds more reward per added dollar for planning than for implementation or verification in this setting. Increasing workers in those latter stages raised cost without a comparable improvement over the solo baseline. These observations are specific to the tested tasks and harness.

Parallel planners can probe different parts of the executable in fresh contexts and return compact reports, giving the lead a broader foundation. The team observed an early burst of useful delegation; delegating throughout the entire process added hundreds of calls and substantial cost without improving the measured reward in that comparison.

Implementation-only workers had a weaker foundation because one lead had to discover the behavior alone. Each worker then saw only a slice of the task and could implement incompatible pieces. Providing more context did not automatically solve the problem: workers sometimes exceeded their boundaries and duplicated work.

Most messages went back to the lead rather than directly between workers, leaving the lead to coordinate and stitch the results together. Dziri says this integration burden increased latency and cost while reducing reward when the pieces did not fit.

The result changed on Commit0, where documentation, a skeleton and unit tests make the intended plan more explicit. There, Dziri says, implementation was the bottleneck and allocating more workers to implementation produced the largest improvement. She refers viewers to the paper's appendix rather than presenting the full results in the talk.

Next the researchers vary model strength. With a strong Sol lead and cheaper Luna planning workers, they report nearly the same performance as a team using the stronger model throughout, with a small performance tradeoff and substantially lower cost. The comparison illustrates that not every role needs the most expensive model.

The reverse arrangement did not compensate for a weak lead. Dziri summarizes the tested pattern as the lead setting the performance ceiling and the workers influencing the price. She cites one configuration with a Terra lead and Luna workers scoring below a solo Sol agent.

The team then fixes a Sol lead and Luna planners while varying planner count from two to sixteen. The sixteen-planner configuration is reported as 38% cheaper than the all-Sol planning team and roughly 70% cheaper than the team using workers across every stage. Those savings refer to these particular baselines, not every possible solo or frontier configuration.

Within their budget, increasing planning workers appeared helpful on this benchmark, although Dziri expects an eventual ceiling. Scaling implementation or verification workers did not show the same benefit in their tested ProgramBench setting.

The researchers also gave agents a tool for messaging peers. About 84% of messages went from workers to the lead, while worker-to-worker communication was rare. This hierarchical communication pattern appeared across the model families they tested.

More communication did not translate into higher reward in these experiments. Dziri reports that some messages from the cheaper model were malformed, stronger models wrote more text, and adding messaging changed little for some teams while reducing scores for Luna teams.

She cautions against a general conclusion that peer messaging is unnecessary. The harness split work into mostly independent tasks, and workers lacked clear visibility into which peers existed. A shared message board or different task structure might change the result; the team leaves that for future work.

The researchers also tested other model families with the same tools. Some leads delegated and benefited, while others delegated rarely or failed to improve despite delegating. Dziri argues that both choosing an appropriate task to delegate and making use of the returned work are necessary.

Her main takeaway is to allocate workers to the stage that actually limits a task, rather than simply increasing headcount. A strong lead paired with less expensive workers can approach the stronger team's results at lower cost in the tested setup.

Open questions include how much larger teams should organize themselves, whether they can choose a structure under a budget and deadline, and how agents can communicate without routing everything through one lead. She also wants leads that adapt without extensive task-specific prompt tuning.

Dziri closes by emphasizing safety and security as autonomous agent teams become more capable. She calls for research into securing infrastructure and aligning agents, then credits the research team and points viewers to the paper and blog for further detail.