Humanoid robots fell from $1.5 million to $30,000. The GPT-3 moment is still a verb

Jim Fan tells hosts Sonya Huang and Stephanie Zhan that his career in agents started almost by accident. In the summer of 2016 he had a Stanford PhD offer and nothing else on the calendar when friends said a new startup in town was worth a look. That startup was OpenAI. He became its first intern, already hearing talk of AGI, with Andrej Karpathy and Ilya Sutskever as mentors.

The project was World of Bits, later folded into OpenAI Universe: an agent that reads screen pixels and drives keyboard and mouse. Fan still likes the bet. Reply to email, play a game, browse the web — everything a person does on a computer fits that one interface. In 2016 the method was reinforcement learning; there were no transformers. It worked on the tasks they designed and did not generalize. You could not give the agent arbitrary language and have it do arbitrary things.

He took the same instinct to Stanford and a PhD with Fei-Fei Li, watching her vision lab move from static image recognition to agents that perceive and act in simulation or the physical world. After graduating in 2021 he joined Nvidia and never left. He now co-leads GEAR — Generalist Embodied Agent Research — which he summarizes in three words: generate actions. In a virtual world that is game AI and simulation. In the physical world it is robotics. Project GR00T, unveiled beside Jensen Huang at GTC, is the moonshot: a foundation model, an AI brain, for humanoid robots.

Why Nvidia? Compute, first. Foundation models need scale, and while language has scaling laws, the scaling law for embodied AI is still being written. Second, simulation. Before it was an AI company, Nvidia was a graphics company, with years of physics, rendering, and real-time GPU acceleration. That stack, Fan argues, is exactly what a robotics data pipeline wants.

Asked why so much of the industry still chases real-world data, Fan refuses the either-or. GEAR splits the problem into three buckets: internet-scale text and video, synthetic data from Nvidia simulators, and real robot logs collected by teleoperation. A strategy that works, he says, will mix all three into one pipeline rather than betting on a single source.

Internet data is the most diverse and carries common-sense priors. People film themselves, film each other, and upload instructional video, so a model can watch how humans handle objects and how objects behave. What it cannot do is download the motor commands. Those action labels do not live on the web, which is why simulation exists: every action is recorded, every consequence is visible, and the data scales with GPUs. Fan’s figure is stark. A real robot has 24 hours in a day. A GPU-accelerated simulator can run about 10,000 times faster than real time.

Simulation’s weakness never fully goes away. Physics will differ, pixels will differ, and the scenes will be less diverse than life. Real robot data has no sim-to-real gap, and it is expensive in both money and atoms: you hire people, and the world still only turns at one speed. Fan’s conviction is complementary strengths. Combine the three, and you cancel the holes. That, not a prettier renderer or a bigger real-world fleet alone, is how he thinks a robot foundation model actually gets built.

Fan will speculate on a research breakthrough in two to three years — the GPT-3 moment for robotics — and then he slows down. Putting robots in daily life also means affordable hardware, mass production, safety, privacy, and regulation, which will take longer. What would even count as the research moment? He borrows *Thinking, Fast and Slow*. System 1 is unconscious, fast motor control: grasping a cup of water without planning each fingertip. System 2 is slow reasoning and planning.

The missing leap, he says, is on the System 1 side. His favorite test is a single verb. Opening a door is not opening a window, a bottle, or a phone, yet people hear “open” and their hands already know. No robotics model he has seen can generalize that low-level motor control across objects. He wants a model that understands the abstract verb and then unpacks it into the right motion. That is the GPT-3 moment he hopes arrives in two to three years.

System 2 is in better shape. Frontier models already reason, plan, and write code. Gluing the two together is its own research problem: one monolithic model, clean and a single API, or a cascade that lets System 2 and System 1 talk. Frequencies fight the monolith. System 2 might decide once per second; grasping a cup wants something like a thousand tiny muscle decisions a second. A cascade may be wiser, but nobody knows whether the two sides should speak in text or in latent variables.

Fan’s own bet is still the data pipeline, not a brand-new architecture. Transformers take tokens in and tokens out; token quality sets model quality. Internet, simulation, and real-robot actions, once scaled and tokenized, can be compressed the way Wikipedia next-token prediction was. He does not think transformers have been pushed to the limit in embodied AI. The scaling law, he says, is just getting started, and emergence may show up as data and model size grow.

GEAR chose humanoids for a stubborn environmental reason. Restaurants, factories, hospitals, tools, and equipment were designed around human bodies and human hands. Hardware that is good enough, in principle, can do any task a reasonable person can do. Fan does not think that hardware is here today, but he expects the ecosystem to mature in two to three years. Once the body is affordable, the scarce piece is the AI brain: language in, human-level tasks out, in homes, restaurants, hospitals, and factories.

There is also a data reason. The internet is human-centered. Videos of five-fingered hands on objects are everywhere; videos of robot arms and grippers are not. Train on a form close to the data, then specialize later to arms and narrower bodies. For Project GR00T the focus is humanoid now, but the simulation and real-robot tools are meant to be general enough to move to other platforms. Full generality first, special cases second.

Robotics still has a constituency for narrow, environment-specific stacks. Fan answers with NLP history. Before ChatGPT, translation, coding, math, and creative writing each had its own model and pipeline — specialists. Then one model unified the lot: a generalist. Prompt it, distill it, fine-tune it, and you get specialized generalists. Historically those are far stronger than the old specialists, and cheaper to keep, because there is one API. In 2024, he says, most robot products are still specialists: specific hardware, specific tasks, specific data, specific pipelines.

GR00T is the attempt to force the generalist moment, humanoid first, then other embodiments. Specialists will show results faster. Nvidia is taking the slower, harder research path because it thinks the future belongs to generalists. The commercial wrapper is a full stack: Jetson Thor edge chips, announced with GR00T, plus the foundation model, plus the simulation utilities. Fan quotes Jensen Huang: everything that moves will eventually be autonomous. Not today, maybe in ten years or more — and if intelligent robots ever become as common as iPhones, the platform has to be built now.

The result Fan likes to cite is Eureka. A five-fingered hand learns pen spinning — superhuman relative to Fan, who gave it up in childhood and would fail a live demo. The method is to prompt a language model to write reward functions in Nvidia’s Isaac Sim API. A reward function is the spec of good and bad behavior. Roboticists usually hand-write it, which is tedious and expert-heavy. Eureka automates that design so the same idea can scale to other tasks, or even invent new ones, inside the simulator.

Why does robotics feel hot again after a trough? Fan names three shifts. Hardware has surged since late last year: Tesla’s Optimus, Boston Dynamics, and a wave of startups, with better dexterous hands and whole-body reliability. Prices have crashed. In 2001 NASA’s humanoid Robonaut cost north of $1.5 million each. Companies now put about $30,000 on a full-fledged humanoid, roughly a car. Manufacturing also tends toward raw-material cost as products mature, and a humanoid’s materials are typically only about 4% of a car’s, so he thinks prices can keep falling.

The third shift is models. System 2 — reasoning and planning — is already being handled by frontier systems, the GPTs, Claudes, and Llamas, which generalize and write code; Eureka itself rides that coding ability. Multimodal models are lifting perception. Fan wants to sit on that generality and add action tokens that drive humanoids. Stephanie Zhan adds the infrastructure around data: Isaac Sim, sim-to-real research, and cheaper teleoperation gear. The field is no longer waiting on a single missing trick. Several bottlenecks are moving at once.

Half of Fan’s mandate is virtual. His personal mission is embodied AI; gaming and robotics are two halves of the same agents. He plays Minecraft badly and wants an AI to avenge that. MineDojo was a platform for general agents in an open voxel world with no score and no fixed plot, trained on play videos, wiki pages, and Reddit talk. Voyager came after GPT-4, then the best coding model: treat code as action. Convert the 3D world to text, let the agent write programs against APIs, and give it a self-reflection loop when the code fails.

A working program becomes a skill, saved to a library so the next similar situation does not require trial and error. Fan describes that library as a codebase the language model authored with no human in the loop. An automated curriculum sits on top: the agent knows what it knows and what it does not, and proposes a next task that is neither too hard nor too easy. The only high-level order was to find as many novel items as possible. It chose the order of objects and tools itself. The name Voyager is a joke about how far it traveled. He calls the self-awareness an emergent property of a strong reasoning engine.

Gaming and robotics share a skeleton: perception in, actions out, exploration that collects its own data. Robotics is harder because of the sim-to-real gap; games train and test in the same world. Fan’s proposed “foundation agent” generalizes on three axes — skills, embodiments, and worlds, virtual or real. In games he wants NPCs that are actually alive, with memory and the power to change the story, so a title has infinite replay value, and worlds generated on the fly from text-to-3D, text-to-video, and language agents. Today’s open models, he says, still lack diverse, engaging behavior, and inference has to be cheap in the cloud or small enough to run on-device.

He does not treat virtual work as a side quest. Domain randomization trains a robot in 10,000 parallel simulations with slightly different gravity, friction, and weight. If an agent masters all 10,000 realities, the physical world is the 10,001st. DrEureka extends Eureka by letting the language model write the randomization parameters; the resulting policy transferred zero-shot, no extra fine-tuning. The demo is a robot dog walking on a yoga ball. A viewer asked a real dog to try. The real dog failed. Fan’s joke name for the benchmark is ADI: artificial dog intelligence.

On architecture Fan is not ready to declare a promised land. Data is still the bottleneck; action traces have to be collected in sim or on robots. Once that pipeline is mature, he would push tokens through transformers the way next-word prediction chewed Wikipedia. He has not seen the architecture exhausted. He is also watching alternatives, especially Mamba and test-time training, which do not attend over every past token and look more efficient at inference. They have not been scaled to frontier size. Until they have, he will not call a winner.

Outside embodied AI he is most excited by video generation as a world simulator: physics and rendering learned from data, the Sora-class models, which could become another data-driven gym for agents. On a ten-year horizon he wants coding agents at the level of human software engineers, and humanoid robots at human reliability and agility — with Project GR00T as a success, and a more personal wish he states twice. He wants robots to do his laundry. When? As soon as possible. He cannot wait.

Asked whom he admires, he names three people. Fei-Fei Li taught him research taste: identifying which problems are worth solving is harder than how to solve them, and the turn to embodied AI looks right in hindsight. Karpathy is “the great educator” who writes code like poetry. Jensen Huang surprises him as a CEO who cares about research and actually knows model details. For founders, Fan’s advice is unromantic. Papers are more practical than they used to be; keep up, download Nvidia’s open simulators, and get your hands dirty. If you believe there will be as many intelligent robots as iPhones, start today.