Robotics Hits Its GPT-3.5 Moment: Inside Gemini Robotics, From Embodied Reasoning to Origami

Asked what changed in the year since their last visit, Keerthana Gopalakrishnan points to a widely shared realization that imitation learning simply works, along with progress in reinforcement-learning walking for humanoids and a flood of cheap hardware that lets the community build and share. Ted Xiao adds that the field has moved past the simple academic tabletop pick-and-place of a decade ago toward more advanced embodiments, realistic deployment and commercialization.

Xiao says the demos that would have stunned everyone a year ago now look mundane. Any serious release today is on a humanoid, on hands, in the wild or on a bimanual arm, and these problems are significantly harder. Gopalakrishnan notes that rising funding and commercial interest also change how open the state of the art remains.

Host Nathan Labenz recalls that two years earlier they placed robotics at GPT-2. Gopalakrishnan does not think there has been a ChatGPT moment yet, and she doubts it will look like one, because anyone can try a chatbot on a phone but few people can try a robot brain. That creates a chicken-and-egg problem: robots must exist for the models to get data, and the models must be capable before robots spread.

Xiao puts robotics somewhere between GPT-3 and GPT-3.5, for two reasons. Language models of that era began to work out of the box rather than needing narrow fine-tuning, and today's pre-trained robotics model is likewise already a generalist, even though fine-tuning still gives much more reliable performance. The second reason is that he sees early signs of the scaling laws that turned language modeling into an engineering science.

Gopalakrishnan cautions that robotics is not one thing. Reasoning and perception can borrow heavily from general vision-language research and scale in familiar ways, but actions are progressing more slowly because nobody yet knows how to represent them well or derive their scaling laws. She sees the field forking, with one part close to commercialization and another still being worked out.

Xiao describes the Gemini Robotics release as a two-for-one bundle. Doing the full stack of frontier modeling gives the team the responsibility to improve the base intelligence that the robot foundation model sits on, rather than taking off-the-shelf models tuned for language benchmarks or visual question answering. He recalls viral examples where leading models could not tell left from right or big from small, and argues that if the base cannot, you cannot bolt action knowledge onto a broken foundation.

The ERQA benchmark, added near the end of the project, probes spatial reasoning, state estimation and trajectory reasoning. Examples include how many degrees to turn an oven dial to match the others, or whether a drawer is open, closed, full or empty. The researchers curated every image and question by hand so nothing could leak from training data or follow a template, and Xiao says Gemini 2.0 Flash and Pro did very well.

Gopalakrishnan sees this as resolving an old divide between people who want modular perception, planning and control pipelines and those who want end-to-end learning. Improving intermediate capabilities and improving the end-to-end model can reinforce each other, she says, and that was a real realization for her.

Labenz tries to map the release onto the nested control layers he knows from drone makers, with slow abstract planning on the outside and fast low-level control inside. The guests correct a common misunderstanding. Gemini Robotics-ER is a version of Gemini trained for spatial reasoning that runs in the cloud and is not meant to output low-level actions, while the Gemini Robotics action model is itself distributed, with a backbone in the cloud and a decoder on the robot.

Xiao says the ER model is already useful on its own for practitioners who want pointing, subpart localization or grasp-pose prediction to plug into a classical pipeline. The action model was trained by distilling knowledge from ER. Gopalakrishnan adds that the pieces can be composed in different ways, including keypoint-conditioned setups or a pure end-to-end model, and that the interfaces between them are modifiable.

Xiao argues that the modular, pipeline-style thinking applies during training, with pre-training, post-training and distillation. At inference time the system feels end-to-end, and any internal structure is learned rather than engineered by human experts. Gopalakrishnan points to a demo in which a researcher showed the robot unseen objects, folded paper and gave drawn instructions. It convinced her that robotics is an AGI-hard problem, needing action, language, multimodal and symbol understanding together.

Labenz presses on latency: a chatbot can think for 15 to 30 seconds, but a robot whose grip is slipping cannot. Xiao answers with an analogy to locomotion, where humanoids backflip and walk on rocky hills using small reinforcement-learning policies that act instinctively, much as the human spine handles a stumble without consulting the brain. Manipulation needs both continuous high-level planning and a fast reactive layer.

Gopalakrishnan also stresses that software operational safety and intervention mechanisms sit alongside the learned system, and that the right latency and design will only be learned by deploying robots into harder settings.

In the current design, the cloud model supplies the brain, while the on-device action decoder handles things like regripping when an object slips. Gopalakrishnan gives the numbers: the high-level system replans every 250 milliseconds and the local decoder controls at 50 Hz. Xiao adds that humanoids need huge numbers of floats at high frequency, which a large language model cannot produce token by token, so a fast local component is a natural fit.

Xiao calls the dexterity results mind-blowing and says they counter the long-held view that learning-based systems could not get low-level control from a vision-language model. He describes the release as one of the least cherry-picked he has worked on, with a very high volume of evaluation trials. The office has stacks of folded origami boxes taller than he is, and other tasks include opening a Ziploc bag to take out bread and scooping nuts with a metal spoon.

Gopalakrishnan was most struck by the tongs. Previous approaches used one gripper to hold the tool and another to operate it, but here the execution was far more dexterous than expected. She sees room to improve by combining dexterity with generalization: dexterous evaluations and generalization evaluations are still largely measured separately.

On failures, the guests say that on the tabletop Aloha platform there are few ways to fail catastrophically. The system fails like a toddler learning to grasp, retrying and sometimes looping, but not flinging its arms around. Xiao contrasts this with earlier RL-explore policies in Google's everyday-robot project, where a new trash object could end in a loud crash, and says he has not seen the adversarial instabilities once feared.

Labenz reads an example from the Asimov safety benchmark, where a robot chef correctly judges that pouring boiling water into a trash bin is undesirable, but notes accuracy in the 80% range. Gopalakrishnan explains that she and Xiao prompt models in good faith, while the benchmark authors probe in bad faith to see how badly they can fail. She says the safety of vision-language-action models is not yet where language-model safety has evolved.

Today, people watch the robots and use literal e-stops. The guests describe layers of semantic safety, operational safety and hard bounds. Gopalakrishnan says she does not expect to do away with classical safety limits, since the internet can fail or a cloud model can become unreachable. Labenz summarizes it as defense in depth, hoping for no correlated failures among the layers.

On where robots go first, Gopalakrishnan says it is up to the deploying groups: some think homes first, for the data, while others think homes demand a very high safety bar and low price and will come last. Xiao frames the open business question as whether a deployed data flywheel or in-house collection will drive faster progress. He says the flywheel is not fully ready yet, and a 6-year-old at home is among the last places he would trust these models.

Labenz notes that most robot data is human teleoperation, much like the instruction-tuning phase of early language models, and that a billion robot tokens is tiny next to the tens of trillions behind a foundation model. Gopalakrishnan agrees that data is a blocker, partly because hardware must be in the loop, and says internet data, human-uploaded video and simulation can help. She adds that the source matters: a billion tokens of generic conveyor-belt pick-and-place will not solve AGI.

Xiao says robot pre-training data needs two non-negotiable properties, high quality and diversity. Teleoperation gives quality but struggles with diversity, while simulation and generative video give diversity at the cost of doubts about quality and grounded physics. His stance is cautious optimism: real-world data is still gold, and synthetic sources are not yet ready to cover the space, though this may be the cycle where that changes.

He also clarifies scale, noting that existing public datasets are already in the tens of billions of tokens and that he would be happy with a scalable way to get a trillion. His longer-term argument is that there will never be another hundred trillion free human-generated tokens online, so future token growth probably has to come from robots interacting with the real world.

Labenz compares this to colleagues' medical and science work, where specialized data sets were upstreamed into the base model and fine-tuning became unnecessary, and asks whether robotics will follow. Gopalakrishnan says the trajectory points that way, and that much embodied-reasoning capability already lives in Gemini 2.0 Flash, while actions will take longer. Xiao says the Gemini Robotics-ER model is available to trusted testers with an open waitlist, and that pointing, bounding boxes and segmentation masks are increasingly available out of the box in the general models.

Xiao says his main updated belief is that general manipulation of anything at human level is the bottleneck for robotics, and solving it requires the world knowledge in a foundation model. He does not claim foundation models are needed for narrow tasks like folding clothes or doing dishes, but argues physical AGI needs one, integrated through the full stack. Gopalakrishnan frames the question as whether you think robotics is an AGI problem.

She notes that companies that started from action datasets are now adding embodied reasoning and will likely add audio and interaction, which means re-engineering a Gemini-like model at high capital cost. Even so, she does not see a pure big-tech win. She argues that language-model progress left room for products like Cursor, and she hopes more players build foundation models.

Labenz compares the fine-tuning results to his own 2022 GPT-3 bootstrap loops and asks how far a few thousand examples can go, for example for fine-grained assembly. Gopalakrishnan says fast adaptation works with few demonstrations on repeatable hardware, but the number needed rises with task difficulty and with how general you want the behavior to be. She expects models to eventually be fine-tuned to a person's own robot, task and home.

Xiao clarifies that in this release both the tens-to-hundreds fast adaptation and the thousands-of-examples fine-tuning are in-weight learning, not in-context prompting. Simple pick-and-place may need only 10 to 100 examples, while precise work with small parts might need thousands or tens of thousands. He hopes these upper bounds fall until any task can be learned from a few examples, or placed in context.

Gopalakrishnan says a notion broken in recent months was that humanoids, with more degrees of freedom, would not respond to the same recipe as Aloha. For narrow single-task situations, imitation learning just works, though a structured study of scaling is still needed. Labenz calls even thousand-example fine-tuning transformative, since much valuable work happens in controlled settings.

Gopalakrishnan thinks of hardware as the boundary and AI as what pushes on it. Moving from a single arm to Aloha made dexterity a research field, and moving to humanoids with hands opens multi-finger dexterity and whole-body tasks such as squatting for a low shelf. She also warns that humanoids are complex and often broken, and that VLA problems are now problems of scale, so getting enough humanoid data is an engineering challenge she expects to be solved.

Xiao agrees humanoids are the most challenging robotics problem to date and a great motivator for research, but he is not convinced they are the form factor that first touches society at scale. He remains only cautiously optimistic about how single-task sample efficiency carries over to multi-task whole-body humanoids, since the jump from Aloha to a humanoid dwarfs the jump from one arm to Aloha.

Gopalakrishnan adds that moving the head makes the problem non-Markovian, so memory and longer-horizon thinking become necessary. For wishes, she asks for good hands and better teleoperation for humanoids, where occlusion, VR delays and whole-body motion capture are open problems. Both praise 1X's work on motors designed for low-impact contact.