Optimizing Agent Accuracy, Cost and Latency

AI21’s Or Dagan discusses optimizing agent accuracy, cost and latency. He presents his company’s Maestro approach and benchmark results, which should be read as vendor claims under specific test conditions.

He uses BrowseComp Plus, a research benchmark, as a running example. A benchmark score does not by itself establish performance on every production task.

The first optimization category is selecting models, prompts and tools; a second is allocating additional computation at inference time.

Testing several model and prompt configurations can reveal which combinations work for a task. More tools are not automatically better if the agent cannot use them effectively.

He plots accuracy against cost and introduces a Pareto frontier: configurations where no alternative in the measured set improves one dimension without worsening another.

Vertical scaling gives a model more work or computation for one attempt. Its value depends on the task and whether the extra cost produces a meaningful improvement.

Horizontal scaling runs multiple attempts or agents and then selects among results. It requires a useful way to rank, verify or combine the candidates.

Parallel execution may reduce additional wall-clock latency compared with sequential attempts, but it can increase total compute cost and depend on available capacity.

Dagan says a portfolio of different models can help when their errors do not fully overlap. The benefit depends on diversity, selection quality and actual workloads.

With many configurations, a developer must decide which to run first, when to escalate and how to trade cost against response time. The search space grows quickly.

He argues that manually tuning every decision is expensive and may miss useful combinations. Measurement and a representative set of tasks are necessary for any optimizer.

AI21’s proposed approach learns estimates of success, cost and latency for actions in context, then uses them to orchestrate model and tool choices at runtime.

Dagan describes sampling the configuration space rather than exhaustively running every combination. Prediction errors and distribution shifts can still affect choices.

He presents additional benchmark results, including Deep Research Bench, and claims improvements to the accuracy–cost frontier. These are vendor-reported results, not independent guarantees.

Instead of fixing a number of search cycles in advance, he proposes dynamically deciding when further work is worth its cost. The stopping rule depends on the estimates and budget.

He closes by arguing that agent optimization can be automated for relevant parts of a configuration space. Real deployment still needs monitoring, evaluation and validation on its own traffic.