From GPT-2 to GPT-3 Moments in Robotics: Six DeepMind Papers on Bridging Robots and the Internet

Host Nathan Labenz opens by recalling that a year earlier, after RT-1 and PaLM-E, Gopalakrishnan described robotics as sitting between GPT-2 and GPT-3, held back by a lack of internet-scale data. Since then the team has published a rapid series of papers, and he asks whether everything has gone according to plan. The guests say it feels like an inflection point, perhaps now closer to the GPT-3 era.

Xiao's one-sentence explanation is that internet-scale foundation models took things that worked 20-30% of the time to 60-70%. A robot is a complicated engineered system with many components, and if each one is unreliable, getting the whole system working takes many iterations. With a much higher hit rate in every component, research iteration, engineering and data collection engines all speed up.

Gopalakrishnan adds that research is hard to plan and usually means breadth-first search over many approaches, doubling down on what shows signal. Over a longer horizon, she names the directions the team cares about: more data, better data efficiency of policies, self-improvement, and better use of foundation-model knowledge and human input.

Gopalakrishnan frames RT-1 as showing that enough in-domain data gives strong in-domain performance, while RT-2 is about generalization. By building on vision-language models, the robot can connect internet concepts to the motions in its robot dataset, with notable results on symbol understanding. Co-training, with image-language pairs and robot data in every batch, helps avoid forgetting the concepts learned from the web.

Xiao explains the older approach, where a pre-trained perception system feeds a representation into a downstream action policy trained on narrow robot data. The RT-1 dataset has about 130,000 teleoperated demonstrations covering only about 17 objects in an office-kitchen setting. In RT-2, the boundary between perception and control blurs, because both are treated as one visual question-answering problem.

Labenz notes this gives a single end-to-end model with one set of weights. Gopalakrishnan's favorite demonstration is a command to move to the sum of one plus two, which the robot interprets. She stresses the limit, though: the movements themselves come only from the robot data, so RT-2 transfers motions to internet concepts but does not invent new motions, since physics and grasping knowledge does not exist on the web.

To make this work, robot actions had to be expressed as language tokens rather than the custom action tokens RT-1 used. Xiao describes the broader goal as bridging two piles of data that sat far apart, robot-specific data and internet concepts. Gopalakrishnan notes that imitation learning is what works best so far, but demonstrations scale only linearly with humans and robots, so transfer from the internet, simulation or self-improvement is needed.

RT-X pooled datasets from many academic labs into a single model trained across robot embodiments. Xiao stresses that it was a decentralized team effort driven by the shared recognition that robotics data is not as plentiful as web data. They did not know at the start whether one generalist would be strong on every embodiment, and it did not win everywhere.

What shocked Xiao was that the generalist often outperformed specialists that had been tuned to individual labs' setups. Earlier multitask efforts usually gave up raw performance in exchange for robustness. Gopalakrishnan says the result shifted opinion: robots are perhaps only as different as English and Chinese, with similar concepts expressed differently. It also gave researchers a common benchmark, and showed how much everyone had been collecting overlapping pick-and-place data.

The embodiments were limited at first to single-arm manipulators with a two-finger gripper and one camera view, yet the datasets ranged from small toy-handling arms to big industrial arms, Google's mobile manipulator and dexterous cable-routing robots. Xiao says it is an open question whether the future has one morphology or many, noting that households already contain specialized robots like refrigerators and dishwashers.

Gopalakrishnan makes the case for humanoids: the world is built for humans, many tasks are bimanual, wheels get stuck on curbs, and learning from human videos works best when the body is close to a human's. She also acknowledges the limits of cost, payload and battery, and that extremely reliable tasks may favor specialized robots. Xiao adds a technical detail about RT-X: it infers which robot it is controlling purely from the image, so it cannot yet work zero-shot on a completely unseen robot.

On fine-tuning, the guests say some checkpoints are open source but large model sizes make serving and tuning hard for outside partners. Xiao calls RT-X a starting line rather than a finish line. Its most impactful result is that new research, such as Berkeley's Octo with diffusion heads, now assumes a cross-embodiment, multi-robot setting by default.

Labenz describes RT-Trajectory as working backwards: to prompt a robot with a line drawing, they took existing successful robot episodes and annotated the images with a line fitted to the trajectory the robot actually followed. A model trained on those annotations can then be prompted with a human sketch, or even a line drawn by a vision-language model. He suggests developers elsewhere can borrow the pattern of synthesizing the prompt side from data they already have.

Xiao says the value is granularity. If RT-1 or RT-2 fails, you can only reword the instruction, which cannot say that the arm is two centimeters too low or should approach from the left. With a trajectory sketch, changing the prompt under identical conditions produced qualitatively different behavior, and it let the team take the robot into new buildings where it just worked.

Gopalakrishnan sees a bridge to classical planning. RT-1 and RT-2 represent only short horizons, while the sketch represents a longer-horizon future and a vocabulary of contact interactions such as where to pick or open. She imagines expanding that vocabulary to motions like spinning or opening a faucet, and having multimodal models generate the sketches for another model to follow.

Labenz adds that the line carries a color gradient encoding relative position in three dimensions, and that working backwards in a broader setting takes more careful engineering than in a narrow one. He sees the result as a step toward a robot apprentice that you can show a task once or twice anywhere.

Gopalakrishnan says AutoRT started from the assumption that, after RT-1's strong in-domain performance, the next question was how robot learning looks at scale. The project used about 50 unique robots and around 20 at once, in real environments rather than staged setups. Since hundreds of teleoperators are impossible, the team used human supervision sparingly, for teleoperation on tasks policies could not do or after mistakes, and handed other coordination to language and vision-language models.

The constitution came from needing rules when few humans are watching. It has foundational, inviolable rules, embodiment-specific rules such as the robot having one arm, and environment-specific rules. Her analogy is that human behavior is shaped by laws with consequences, so a robotic code of behavior may be needed in the future.

Labenz asks about how rules get interpreted. Gopalakrishnan gives an example in which an instruction to avoid harmful things led the robot to refuse to pick up a cigarette. She says current problems come less from robots being too clever about rules and more from their being too incapable at zero-shot tasks in the real world.

Pressed on whether ChatGPT-style system prompts, which are easy to jailbreak, can work, she is emphatic that prompting alone guarantees no safety. In the experiments, humans had line of sight over several robots and traditional controls limited velocity, force and contact. She notes robot failures break things or hit people, safety is mission-critical, and robotics safety research must move from mathematically bounded specifications to language and semantic ones.

Labenz reflects that physical limits, such as force caps, speed and materials, offer intuitive controls outside the model, and wonders whether lessons from robot control could be back-ported to purely digital AI systems. He ties this to the idea of external engineering safeguards alongside alignment.

Labenz summarizes the paper as teaching robots to respond to verbal feedback, such as moving slower or grabbing lower, and doing better on the next attempt. The method trains on more chain-of-thought-style reasoning, and a rollout step envisions several possible futures and picks the likely best, which improved performance over choosing a path immediately.

Xiao highlights the tight feedback loop: users interact with a model that writes reward code controlling the robot, see the execution, and rate both individual responses and the whole conversation. He calls it the team's tightest cycle yet. By day, a frozen model is taught by users; overnight, those multi-turn conversations with dense ratings are distilled back into the model, so it is ready again the next day.

Gopalakrishnan compares this to learning to surf or ski, where you struggle during the day and are better after sleeping. In her mapping, the day is in-context learning and the night is retraining the weights. She also cautions that rating conversation quality by shortest path is too simple, because people care about correct code, responsiveness to feedback and stability.

Labenz describes PIVOT as overlaying numbered candidate points on an image and asking a vision-language model which one the robot should move toward, then iterating with new candidates clustered around the chosen one. Xiao corrects that the numbers also come with procedurally generated arrows representing motion. The model acts as a zero-shot frozen critic, choosing among visual annotations.

Xiao admits PIVOT is an odd one out among the papers, a result shared because it was surprising rather than a scaled-up trained system. Many assumed that dense contact and spatial reasoning would need hardcore robot data, yet these models seem to hold some notions of motion, causality and contact. He is clear that performance is far below RT-2 or fine-tuned policies, and that internet text alone will not solve contact-rich robotics.

Gopalakrishnan points to the trend toward zero-shot use of large models, with effort shifting to leveraging their knowledge efficiently so that collected robot data is complementary to what they lack. She also likes that image-based specification can express tasks hard to describe in words, such as folding a T-shirt.

Asked whether it will take 50 more papers or mostly scaling and integration, Gopalakrishnan says data remains the hard problem. Doubling data efficiency is as good as doubling data spend. RT-2 is a widely used baseline with gaps in reasoning, so she thinks a couple more breakthroughs are needed, including learning methods built on top of strong zero-shot ones and better interfaces between robotics and foundation models.

Xiao agrees that general-purpose, embodied-AGI-level robotics is a few breakthroughs away. He believes commercially useful applications are already more feasible than a few months or years ago, since many ingredients are ready. On Google's plans, Gopalakrishnan declines to speak for the company and says reliable, safe generalist robots are still some time away.

Her next research interests are the weak link of actions in vision-language-action models, including whether actions could become more like code, and dexterous bimanual tasks that need better representations. Xiao wants to go beyond language, asking whether physics and causality sit in a corner of their own or are spread through the data humanity has produced.

Xiao also wants to learn how much robot data is needed: a cherry on top, or a thousand or million times more. He calls these questions about the scaling laws of robotic capabilities, which the field does not yet understand well, and hopes to know more within a year or two.