Robot-Use Agents: Can General Models Control Different Robots?

The hosts introduce the idea of robot-use agents: general-purpose models that can use tools to control different robots. They speak with founders from Waddle Labs and RoboCurve, whose demonstrations have helped spur discussion of the approach.

Waddle Labs describes its work as building a harness for language models to control robots, collecting data and using it to improve models. RoboCurve says it evaluates physical AI across models, robot types and environments.

The guests have shown tasks such as unscrewing caps and coordinating robots. The hosts begin by tracing the research behind these demonstrations, including vision-language-action models and code as policies.

Jay describes RT-2 as a vision-language-action approach that adapts a pretrained model to output robot actions, such as an end-effector pose that can be converted into joint commands. Its pretraining gives it access to knowledge learned from web text and images.

A participant compares direct action output with a model that can reason or write code before acting. The latter can allocate work to a program and potentially reuse it, rather than generating each action from scratch.

The group invokes the bitter lesson: general methods that benefit from data and computation can outperform hand-designed approaches. They suggest that the key question may be which kinds of training data transfer to robot tasks, rather than architecture alone.

They point to improvements in coding, tool use and in-context learning. A general model can inspect the tools available in an environment and write a policy, while a robotics-specific approach may rely more directly on scarce robot data.

The conversation recalls Voyager in Minecraft, where an agent could create reusable code tools as it explored. The participants see a connection between creating tools for a game and writing programs that control a robot.

In early code-as-policies work, researchers supplied functions such as pick, lift and move to a pose. A language model composed those functions into code for a larger task, using prior coding knowledge rather than collecting a new robot dataset for every request.

The group compares in-context learning with changing model weights. One participant reports an unpublished experiment in which examples initially helped a held-out task but gains fluctuated and eventually saturated; those figures are observations from that experiment, not a general law.

Retrieval can bring relevant examples into active context as needed. The discussion also considers low-rank adaptation and full fine-tuning as other ways to incorporate experience, with different data and compute requirements.

For a robot entering a new setting, the harness can package a learned procedure as a reusable program or skill. The guests compare this consolidation to distilling experience so future agents do not need to rediscover every step.

They also consider a larger model specializing the harness for a domain while smaller, faster models execute individual tasks. How far in-context learning alone can go remains an open research question in the conversation.

A demonstration shows a model using camera feeds and tool calls to move a block into a bowl. The speaker clarifies that this example sends pose commands over several turns; it is closer to tool calling than writing a complete code policy.

A harness could instead compile repetitive steps into code or call a previously learned skill. This can reduce the need to wait for a large model on every movement and leave the model to handle variation or failures.

The demonstration is visibly slow because it waits for model responses. The participants expect latency to improve, but their specific forecasts are predictions rather than evidence of real-time robot control.

For a repeated manipulation task, much of the sequence may be deterministic after the procedure is known. A vision-language model can be inserted at points where objects vary, errors need detection, or recovery requires a different response.

Participants discuss program induction: instead of learning only a direct input-output mapping, a system can infer a compact function from examples. They note that identifying the right inductive biases remains difficult.

The hosts bring up the Platonic Representation Hypothesis, which proposes that models trained on different data can learn increasingly similar representations. The guests treat it as a possible explanation for transfer between language, vision and robotics, not a settled guarantee.

The guests speculate about why newer models seem better at spatial tasks, including possible computer-use or CAD data. They do not know the models’ private training mixtures, so these explanations are hypotheses.

Manipulating a 3D object on a screen, for example, can expose relationships such as rotation and viewpoint. The speakers also mention human first-person video and interfaces that make robot controls resemble familiar computer-use tools.

Asked about the future, one guest predicts broadly capable robots within about two years. That is an industry participant’s forecast, not a demonstrated timeline; the discussion immediately returns to unresolved issues such as latency and cost.

Waddle’s goal is to convert a slow first pass by a general model into a fast skill or policy that can be executed repeatedly. The participants compare collecting experience, consolidating it and updating a model with a learning cycle.

They compare this idea with DreamCoder’s library of skills and a consolidation phase. How to organize, compress and prune a growing collection of robot skills and deployment data remains an open problem.