Nathan Lambert on Tool Use and Training Language-Model Agents

Tool use fills limitations of model weights. A model trained on historical text cannot reliably know a current fact, and weights alone generate tokens rather than move files or act in the world. A system can interpret some generated tokens as actions and return external information to the model.

Tools began as a way to address missing knowledge, but have become a trained skill central to modern agents. Supervised fine-tuning, preference learning, reinforcement learning and distillation all apply. I think of a useful language-model system as weights, tools and a harness that manages them together.

Early examples included calculators, weather APIs and database search. Search and code execution are especially general: one retrieves information outside the weights, while the other lets the system carry out structured processes. Deep research, browser tools and coding agents built on these ideas.

Tool use is the broad strategy: a model emits a structured request, an external system executes it and the result returns to the token stream. Function calling is a more specific, often JSON-based schema for parsing and validating requests. Code execution is one kind of tool, where generated code runs externally and returns output.

A computation such as printing many digits of pi illustrates the difference: a model may have memorized some digits, but a program can compute them. A deterministic tool can complement probabilistic token generation within a single response.

The foundations go back to neural systems handing work to programs and to retrieval-augmented generation. WebGPT in 2021 used human feedback to train browsing and synthesis. ReAct interleaved reasoning with actions, a pattern that resembles later tool-integrated reasoning.

In 2023, Toolformer used synthetic examples with API calls, while projects such as Gorilla and ToolLLM explored many APIs. Function calling and code execution entered products and APIs. MCP later offered a standardized interface, and by 2025 reasoning models combined search and code execution with more capable reasoning.

ReAct's important idea is that reasoning can maintain a plan while actions obtain information from an external environment. Toolformer illustrates a data-generation approach: insert useful API calls into text, use results and train on the resulting examples. Tool evaluation likewise evolved from format compliance toward solving complex tasks reliably.

The model still only generates tokens. When it emits a tool call, external orchestration detects the request, executes it and inserts the result into the context. The model resumes generation and may repeat the loop several times before answering. Tool definitions specify names, arguments and expected outputs.

A basic harness can be a loop linking model output to tools and passing results back. Models in integrated products can be optimized for a familiar environment; open models often need to generalize to arbitrary customer tools. The connection between model and tool is software, not an independent action performed by the weights.

MCP standardizes interfaces for capabilities such as resources, prompts and tools across hosts, clients and servers. Lambert considers standardization useful but not mandatory for tool use. For training, consistent formats and the ability to generalize to the intended domains matter most.

A modern harness includes the system prompt, tool definitions, orchestration loop, context management, permissions and sandboxing. It can also manage subagents or parallel work. The same weights can behave differently in different harnesses, so an agent benchmark score reflects their interaction.

During training, tool outputs produced by the external system are generally masked from the model's next-token loss. Another design choice is whether to retain reasoning from earlier user turns: discarding it can improve efficiency, while retaining it may preserve useful context. Training and inference must handle these choices consistently.

Tool formats differ among APIs and may use JSON, XML-like tags or function syntax. A model's training and serving formats need to agree. Constrained decoding can restrict next-token choices to valid formats, though Lambert says newer models have become more reliable. Context compaction is another important implementation detail for long tasks.

Training tool use brings environments back into reinforcement learning. An agent can interact repeatedly with tools before a verifier grades the outcome. The familiar post-training toolbox still applies: supervised trajectories, preference learning and RL can all be used at different stages.

OpenThoughts-Agent illustrates the data work behind supervised tool trajectories: source and mix tasks, filter prompts, select teachers, generate completions and filter results. Each decision matters even if the pipeline diagram looks simple. Lambert notes strong results and wants to see more detail about RL in future work.

Environments are a bottleneck for agentic RL. A useful task must be correct and learnable while still challenging; it is easy to generate tasks that are trivial or impossible. T-MAX generated roughly 15,000 containerized terminal environments, varying difficulty, personas and verifiers, and discarded data that did not meet the desired distribution.

Running many environments brings infrastructure failure modes. Thousands of CPU sandboxes may need container images; bandwidth limits can leave GPUs idle. Missing files, expired API credentials or other small errors can invalidate rollouts. Scalable sandbox infrastructure becomes part of the training problem.

Lambert describes T-MAX as an open effort that released models, code, data and lessons from training terminal agents. Unlike earlier math RL runs where scores could rise steadily, long and diverse tool-use runs were prone to collapse after hundreds of steps. Much of the work went into stability.

One source of instability is numerical mismatch between the inference system producing rollouts and the training system taking gradient steps. Both infrastructure and algorithms must account for the difference. Lambert describes a representative T-MAX run using two training nodes and six inference nodes for two to three days, plus sandbox costs.

That is a qualitatively different expense from earlier math-focused RL, with substantial effort sometimes buying only one or two points on a hard benchmark. Frontier reports increasingly discuss environments and sandboxes in detail, even when they reveal less about the precise reinforcement-learning algorithm.

Kimi K3's report, as Lambert describes it, shows that more RL compute improved scores and increased the number of actions taken. Kimi and NVIDIA also discuss training with multiple harnesses: a fixed harness can lead to overfitting, and the same model can score very differently when the surrounding software changes.

Specialist domains add another complication. A single batch can contain highly varied tool tasks, and performance may saturate. Multi-teacher on-policy distillation lets teams train specialists in different domains and combine their capabilities into a final model, reflecting the growing complexity of agent RL.

Reward hacking returns as tool trajectories become longer and more complicated. Agents may manipulate tests or other parts of an environment to obtain a reward without doing what was intended. Rubrics, intermediate feedback and verifiers introduce new opportunities for over-optimization.

Long rollouts create difficult credit-assignment and cost problems. If an attempt takes hours, generating multiple samples for every prompt is expensive and can leave resources idle. Teams must balance algorithm choice, verifier design and a harness that itself changes over time.

The broad lesson is that tools extend the fundamental capabilities of weights, while the software and environments around a model are indispensable. Agentic RL is still early: progress requires both research and the unglamorous work of making infrastructure stable and scalable.