Jason Mancuso: Scaling agent reinforcement learning across mixed GPUs

Jason Mancuso of Modal argues that treating models only as interchangeable static backends can limit differentiation. Teams deploying agents must balance cost, reliability and the quality of the user experience.

He says some customers fine-tune open models to own inference and meet reliability objectives. Others work with proprietary data or specialized tasks that general models do not handle well enough out of the box.

Mancuso also sees interest in integrating more of the agent stack to compete on task quality per dollar, especially in coding. He expects interest in fine-tuning to spread as more sectors develop mature agent products.

He contrasts supervised fine-tuning on recorded traces with reinforcement learning in an environment. Agent sessions branch across many turns, so training only on recorded paths may leave a model unprepared for paths it encounters later.

RL can generate and evaluate new trajectories during training, giving the model a chance to explore more of the task space. Mancuso presents that exploration as one reason teams hope RL will generalize beyond recorded traces.

He argues that a production harness and some form of evaluation or change measurement are a starting point for an RL environment. Representative tasks can be generated from production traces, including cases where the agent failed.

The training loop has two broad parts. Sampler GPUs run the model in the environment to generate rollouts; a trainer consumes those results, updates the model and sends new weights back to the samplers.

A sequential loop keeps rollouts aligned with the latest model but can leave hardware idle. Running rollout generation and training asynchronously improves utilization, especially for long agent tasks.

The tradeoff is staleness: a rollout may come from an older model version while the trainer has already advanced. The resulting difference between sampling and training policies can make the update less stable.

Additional mismatch arises from systems details: inference and training stacks, numerical precision, routing and hardware may not behave identically. Mancuso says the method needs to correct important differences without overwhelming the training signal.

The recipe starts with GRPO and adds masked importance sampling. The method adjusts token-level weighting for differences between the sampling and training policies while masking extreme ratios that could dominate an update.

A second component, score centering, addresses drift that remains or is reintroduced by masking. Modal’s accompanying technical explanation describes subtracting an expected score term so that the retained training signal is less distorted.

The third component is top-p sampling with mask replay. Sampling restricts candidates to a set containing the chosen cumulative probability mass, and replaying that mask makes the trainer work with the same support.

Mancuso shows an ablation using Qwen3.6-35B-A3B on agentic coding tasks. He reports that combining the corrections made the tested training run more stable than the alternatives shown.

He then explains why distributed RL matters to Modal. The company provides training, inference and sandbox environments, but aggregates GPU capacity across many providers and regions rather than a single uniform pool.

That aggregation offers flexibility but fragments available capacity. Mancuso says small clusters can be scheduled relatively quickly, while finding a contiguous block for a larger cluster can take much longer.

For agent RL, he estimates roughly two inference GPUs for every training GPU in the relevant workload. If all of them must sit in one cluster, much of the scarce contiguous capacity is occupied by rollout inference rather than training.

The proposed approach moves inference outside the main cluster. Mancuso says successive model-weight updates are sparse enough to compress substantially, reducing the amount of data sent from the trainer to rollout engines.

In the reported Qwen run, the training cluster operated in one location while inference GPUs ran across multiple cloud providers. Mancuso reports that sparse delta compression made this arrangement practical in their test.

The team also mixes GPU types and precision settings. A trainer on one accelerator and samplers on other accelerators can introduce numerical and kernel differences that would traditionally threaten stability.

Mancuso shows a mixed-hardware run and says the mismatch was several times larger than in a homogeneous setup, yet reward and gradient behavior remained comparably stable under the reported recipe.

He reports that rollouts cost 15% less in this run because some inference used less expensive hardware. The number describes that particular workload and fleet; it does not establish a saving for every RL deployment.

The team also used different precisions, including BF16 and FP8 on H200 pools. Mancuso says the corrections allowed learning from those quantized rollouts in the tested setup, even with additional mismatch.

He points to Modal’s technical write-up and says the implementation builds on Miles and SGLang. His broader recommendation is for teams to build competence beyond the agent harness, including evaluation, inference and training.

Mancuso closes by arguing that greater ownership of the stack can support differentiation in competitive agent markets. The reported techniques are evidence from a particular coding-agent setup, not a guarantee that all RL problems are solved.