Jim Fan's Foundation Agent Vision: Skills, Robot Bodies, and Simulation

Matthew Berman introduces a TED talk by NVIDIA researcher Jim Fan on agents that can act in physical as well as virtual environments. Fan recalls watching AlphaGo defeat Lee Sedol in 2016. He contrasts the significance of that result with AlphaGo's specialization: it played Go, but could not automatically perform unrelated tasks such as cooking or laundry.

Fan's ambition is more versatile agents that can acquire many skills, control different bodies, and work across virtual and physical environments. He uses these three axes to organize the research landscape. This is a research vision, not a claim that such a general system already exists.

The first example is Voyager, an agent developed to explore Minecraft. Berman shows the research paper's comparison of items discovered over time and highlights the benefit of retaining skills. The graph covers a finite experiment: it does not establish unlimited learning or prove Voyager was the first agent ever to use tools.

Voyager has an automatic curriculum, an iterative prompting mechanism, and a library of executable skills. It writes code for a task, observes execution errors and feedback from the game, revises the program, and stores successful skills for later reuse. The paper reports comparisons with selected baselines within Minecraft, without updating the language model's parameters.

Fan explains that Minecraft's open-ended world lets the agent explore, gather resources, fight monsters, and craft items over extended runs. A community JavaScript API converts relevant game information into text and executes generated code. The system uses GPT-4 through queries; its abilities depend on that interface, prompts, environment, and experiment setup.

Feedback includes program errors, the agent's condition, and nearby world state. Voyager uses those signals to revise its attempts, then saves useful programs to a library. Berman sees reusable skills and memory as promising ingredients for future agents, although they do not by themselves establish general intelligence.

Fan gives an example in which the agent is hungry, considers nearby entities, and reuses an existing sword-crafting skill before attempting to hunt a pig. A high-level objective encourages acquiring unique items; the curriculum proposes progressively different challenges. This illustrates the system's task decomposition rather than independent proof of human-like understanding.

The next axis is embodiment. Fan describes MetaMorph, a Stanford research project using a transformer to control many simulated robot forms. Robot morphology is represented as input to the controller, which produces motor actions. The paper reports generalization and transfer within its modular robot design space, not a controller already proven on all physical robots.

Berman compares this with earlier work using language models and robot command libraries. Fan suggests that a richer vocabulary of bodies could someday extend to hands, humanoids, dogs, and drones. That broader coverage is an aspiration beyond the specific experiments shown.

Fan then moves to NVIDIA's Isaac simulation work. He describes accelerated physics simulation and shows examples of training motion and generating varied visual environments. Berman sees a potential bridge between many skills and many embodiments, but the simulation platform itself does not automatically combine them into a universal agent.

Fan gives an example of accelerated training and explains why realistic rendering and variation can help train vision and control systems. Berman suggests that sufficiently realistic simulations could prepare a robot for the real world. In practice, appearance is only one part of the problem: differences in dynamics, sensors, hardware, and unexpected conditions can still make simulation-to-reality transfer difficult.

Fan proposes that an agent exposed to many simulations might generalize to a new physical environment. Berman relates this to the philosophical idea that our own reality could be simulated. That is speculation; the ability to create better simulations is not evidence that we live in one.

The proposed foundation agent would take a description of its embodiment and task and output actions, trained across many environments. Fan compares this concept to scaling language models across text tasks and imagines many kinds of autonomous systems using a shared foundation. The talk presents a future research goal, not a demonstrated single system with those capabilities.

Berman closes with Eureka, which uses a language model to generate reward code for reinforcement learning rather than directly hand-programming every movement. In the research team's suite of 29 simulated tasks across 10 robot morphologies, generated rewards outperformed the compared expert-written rewards on 83% of tasks, with a reported 52% average normalized improvement. The team also demonstrated simulated pen spinning; these figures are specific to that evaluation.

Berman imagines future applications in animation and media and invites viewers to consider whether the combination of these approaches could lead to AGI. His conclusions about universal intelligence and replacing creative work are predictions, not results established by the experiments discussed.