Nathan Lambert on TMax: An Open Training Recipe for Terminal Agents

SUMMARY

In 30 seconds

Nathan Lambert introduces TMax, a project releasing terminal-agent data and open models built from Qwen. He explains why balanced environments, stable reinforcement learning and reproducible checkpoints matter when improvements on strong models are hard to measure.

1Open terminal agentsTMax combines model weights, environments and a training recipe.
2Diverse environmentsThe data pipeline targets balanced terminal skills.
3Stable RLNumerical precision and algorithm choices address instability.
4Shared checkpointsIntermediate artifacts allow others to inspect training dynamics.

Takeaways

TMax releases roughly 15,000 terminal-task RL environments and models from 2B to 27B parameters.
The models build on already post-trained Qwen Instruct models.
Training stability required attention to numerical precision and RL configuration.
Intermediate checkpoints and open data support reproducible comparisons.

TopicsAI agents Reinforcement Learning Post-training Open-weight models Data quality & curation

FULL TEXT

Read the edited text by chapter

Show transcript

Transcript only

TMax and the need for recipes

Lambert introduces terminal-agent models and explains why reproducible gains are hard on strong baselines.

TMax is one of my last papers at AI2: an open release of models from 2B to 27B parameters and a data and reinforcement-learning recipe for terminal agents. Terminal-Bench measures how well agents use a computer terminal to solve tasks.

I think the data work is unusually valuable. The released models start from Qwen Instruct and improve on its existing post-training. Agentic terminal tasks are harder than the math-centered RL problems that dominated much of the previous year, and gains on strong models can be small.

SLIDE 1 / 3TMax model resultsThe paper compares terminal benchmark results across model sizes.

This is why I think the community needs recipe work, not only isolated algorithmic demonstrations. As models improve, careful, reproducible infrastructure and incremental improvements become essential to understanding which changes actually help.

Data and training stability

The project releases diverse environments and studies the infrastructure and algorithm choices needed for stable RL.

The project releases roughly 15,000 reinforcement-learning environments, along with supervised fine-tuning data for terminal skills. The team designed a diverse data pipeline and balanced coverage across the kinds of tasks represented in the evaluation. Hamish and Oscar led the work; I played a small part.

SLIDE 2 / 3Training precision and benchmark comparisonsThe figure shows spikes in trainer–generator log-probability differences.

Making the recipe work required substantial infrastructure. Lambert points to instability in asynchronous RL: a gap between training and generation log probabilities can produce sharp spikes. For Qwen 3.5, using higher numerical precision in the language-model head helped stabilize training.

The team also investigated training stability through the RL algorithm and sampling. DPPO changes the clipping used in PPO-like methods, while larger GRPO groups use more completions per prompt to improve the gradient signal. Both are part of the reported recipe.

SLIDE 3 / 3RL stability comparisonsTraining curves compare algorithm and group-size settings.

Open artifacts and future research

Checkpoints and shared data let researchers compare training runs and investigate small benchmark gains.

The release includes models, data, code and intermediate checkpoints so others can examine training dynamics on a shared task. Lambert relates this to the clean starting points and checkpoints released with OLMo 3 RL Zero.

Benchmark margins between strong open models are often just a few points, making meaningful progress difficult to identify. Lambert hopes the shared recipe will help researchers test lasting changes in agentic RL and build more data-focused work. He credits Hamish and Oscar for leading the project.