Jim Fan on World Models and the Future of Robotics

NVIDIA robotics researcher Jim Fan opens with a memory of the first DGX-1 delivered to OpenAI in 2016. He uses the rapid development of language models as the starting point for a proposed parallel path in robotics.

Fan sketches language-model development as pre-training, instruction tuning, reinforcement learning and automated research. This is his high-level framing of a complicated history, used to motivate a research agenda rather than a guaranteed sequence.

His “great parallel” asks whether models can predict physical world states, be fine-tuned on robot actions and then improve through reinforcement learning. He separates the problem into a model strategy and a data strategy.

Fan argues that vision-language-action models often devote much of their capacity to language and semantic knowledge, while physical motion requires a stronger emphasis on visual dynamics and action. This is his assessment, not a settled comparison of all VLA systems.

Video models can produce plausible scenes involving gravity, reflections and motion, suggesting they learn useful regularities from visual data. Fan also shows a maze example where the generated video violates geometry, illustrating that plausible frames are not a guarantee of physical correctness.

He proposes action fine-tuning to connect a video model’s predictions to robot behavior. The aim is to make the space of possible future videos useful for selecting actions in a particular physical setting.

[DreamZero](https://arxiv.org/abs/2602.15922), which the captions misrecognize as “Dreamer,” jointly models future video and continuous robot actions. Fan calls this family world-action models and reports demonstrations of tasks not seen in the training set.

He says that in the examples, prediction quality and action success are closely related: hallucinated future scenes can accompany failed actions. He explicitly acknowledges that the system does not perform every prompted task reliably.

The data strategy starts with teleoperation, where a person controls a robot to gather demonstrations. Fan argues that hardware availability, operator effort and downtime limit the amount of data that one robot can collect.

The Universal Manipulation Interface moves some data collection off the robot: a person handles an instrumented gripper directly. Fan describes related wearable devices intended to capture useful human demonstrations with less robot time.

His team’s Dex-UMI uses a wearable mechanism mapped to a five-finger robot hand. Fan shows a policy trained on wearable data without teleoperation data for the demonstrated example; that result is limited to the tested setup.

Even wearables require dedicated collection effort. Fan wants data collection to become less intrusive, drawing an analogy to vehicle data gathered during ordinary driving. His proposed scale and feasibility remain questions for future work.

[EgoScale](https://research.nvidia.com/labs/gear/egoscale/) uses large quantities of action-labeled first-person human video for dexterous manipulation. Fan reports pre-training on roughly 21,000 hours of human data, then using a much smaller amount of high-precision and robot data for action fine-tuning.

The examples include sorting cards, manipulating a syringe and adapting a shirt-folding task from one demonstration. They are research demonstrations, not evidence that a general-purpose robot nurse or household assistant is ready for deployment.

The EgoScale work reports a log-linear relationship between training-data scale and validation loss within its experiments. Fan presents this as a scaling law for dexterity; extrapolating it beyond the evaluated data and tasks is uncertain.

Fan predicts that teleoperation will become a small part of the data mix, with wearable sensors and first-person video playing larger roles. That is his forecast, not a result established by the demonstrations.

He also wants to scale the environments used for reinforcement learning. His team reports high success rates on certain real-robot tasks, but replicating physical setups at very large scale is expensive.

One approach scans a real scene, reconstructs interactive objects in a physics simulator and generates variations called digital cousins. A policy can then train in simulation before being evaluated back in the real world.

[DreamDojo](https://arxiv.org/abs/2602.06949) takes a different approach: a learned world model predicts future video and sensor states conditioned on actions. Fan proposes using such neural simulators alongside real robots and graphics-based simulation.

He summarizes the proposed training loop as compute producing environments and data. This is a research vision: simulated outcomes must still be validated against physical behavior, particularly for contact-rich and safety-critical tasks.

Fan lists a “physical Turing test,” configurable robot fleets and automated factories as future milestones. He speculates that the first might be two or three years away; the talk does not establish a benchmark or demonstrate that timeline.

His final ambition is robots that help design and build their successors. He expresses high confidence in reaching his broad end state by 2040, but the number is his subjective forecast rather than a statistically validated probability.