130,000 Teleop Demos: Google's GPT Playbook for Robots

In 2022, Google's robotics team did something that looks almost stubbornly analog. People sat with Oculus VR controllers and teleoperated mobile manipulators through mock kitchens, over and over, for about a year and a half. The haul was 130,000 task demonstrations covering 700 tasks. Nathan Labenz, host of The Cognitive Revolution, divides that by workdays in the project and gets roughly 367 episodes a day.

Those episodes trained a model called RT-1, Robotics Transformer 1, which the team reports at a 97% success rate on those 700 tasks. Labenz's guest is Keerthana Gopalakrishnan, a robotics and machine learning engineer at Google whose Twitter bio reads "mother of robots." Over the prior year her name was on several of the field's most cited papers: SayCan, which showed a language model can act as a robot's planner; RT-1, which showed the large-scale pre-training playbook works on robots; and, that March, PaLM-E, which turns images, language, and actions into tokens for one embodied model.

Labenz's framing is that Silicon Valley still recites the line that bits are easier than atoms, and that AI's impact will therefore stay digital. Gopalakrishnan's papers, plus rapid gains in vision from PaLM-E and GPT-4, are his counterexample. Visual and language understanding already look rich enough to support general-purpose robots. What remains, in his view, is more general control — which she thinks will come from teaching robots to learn by watching humans — and then a race to manufacture many form factors for factories, offices, and homes.

In a lecture for ML Collective that Labenz had watched, Gopalakrishnan placed robotics between GPT-2 and GPT-3. Language models have already shown emergence; robots are starting to show it too, in reasoning and in low-level control. Google's recipe, as she states it, is transformers for control, natural language as the interface, and foundation models for reasoning. She thinks that recipe generalizes and still has a long way to scale with data and compute.

Labenz asks whether, within the decade, a domestic robot could do the dishes after dinner or pick up the kids' toys. Her answer is yes. The published tasks are office scenes: picking things up, cleaning, opening cabinets and fridges. She warns that many viral robot videos are staged. Traditional robotics was PID control and search; then came end-to-end conv nets; now transformers. One model that can cook, clean, plan, and talk — the way a person does — is the point of a robot that can enter many houses.

The picture she wants is simple: people speak to robots in language, the robots think and plan in a visual-language space, then they act in the real world. RT-1 already works across several Google kitchens. Those kitchens are still a tiny, similar slice of the world's kitchens. Bring the same policy to a private house and it will likely break, because the images are out of distribution. Scaling fast enough to absorb real-world variety is the live problem.

Vision and language got a free lunch: weakly supervised data already sitting on the web. Robotics does not. Reasoning and planning can share human text. Action data barely exists, and collecting it is an engineering operation few labs can fund. The robots themselves are mobile manipulators — they drive, they have an arm with a gripper that can take a wiping tool, they dock to charge. Humans teleoperate them; each episode records the instruction, the cameras, and the robot's state. Training is supervised learning on that pile.

The training grounds are "robot classrooms": several mock kitchens. Policies that look decent there graduate to Google's real kitchens. A median episode is perhaps 40 or 50 steps — picking might be ~30, opening a door is longer — and each step is about 11 tokens. Multiply by 130,000 episodes and you still have a toy compared with language or image corpora. Teleop with humans can only grow linearly with the number of operators. Robots have to break that curve.

Two faster paths, in her account, are autonomous collection and transfer from human video. You can stand up 50,000 robots if you have the money and let foundation models gather data zero-shot or with light fine-tuning, in the spirit of work like Code as Policies. Transfer is the omelet-versus-Kobe test: watch your mother cook from a third-person view and you can mostly reproduce it; watch Kobe Bryant and you will not play in the NBA, even with a good model of your own body. Easy skills transfer from watching; hard control still needs practice. She says the team has papers coming on human-to-robot transfer and larger VLMs for control, and she will not detail them before they are public.

Generalization has a similar curve. RT-1 first trained many skills on only 17 objects; the real world has millions. Collect a little data on about 100 objects, then use a vision-language model for zero-shot detection, and open-vocabulary manipulation gets much easier. Going from 17 objects to any object is hard; going from a hundred-plus to any object is a different problem. Tasks follow the same shape: five to 500 is algorithmic and hard; 500 to 5,000 gets easier; 5,000 to 50,000 is mostly scaling, distribution, and inference. She thinks the algorithmic problems are more or less down. They still do not let the robots handle hot objects or operate machines — not because the skills are unlearnable, but because they have not collected that data, and because a splash of hot coffee next to a child is not a research they want.

RT-1's aggressive move is to turn actions into tokens. For the Everyday Robots platform that is 11 variables: one to terminate the episode or choose arm versus base, seven for the arm (three for end-effector position, three for rotation, one for gripper close), and three for the base (X, Y, and heading). When RT-1 was jointly trained on KUKA arms and Everyday Robots, an input told the model which data it was looking at and the action spaces were mapped together. A robot dog might predict 24 variables; a humanoid just adds one predicted variable per extra degree of freedom.

Once actions are tokens, a transformer does not care that they came from a gripper rather than a sentence. She compares it to ViT: people said you could not chop an image into a token sequence the way language is a sequence of characters, and then CLIP, Stable Diffusion, and DALL-E followed. Data interoperability — images, language, and actions all as tokens — is, for her, the key to a foundation model. Thinking while cooking or walking a dog already mixes those spaces; a globally good policy will be fully multimodal, and the endgame is one model that plans, talks, and controls.

The stack today is still two layers. SayCan, the high-level planner, turns "get me a Coke from the fridge" into go to the kitchen, find a cup, pick it up, place it, go to the machine, press the button. RT-1 executes each short skill. PaLM-E is the updated SayCan: segmentation, scene description, depth, and other vision outputs inject embeddings straight into the language model's latent space instead of being translated into words. Optimizing two models in sync is, she thinks, suboptimal. Future papers from the group will push toward a single model. Next controllers will also do whole-body control — arm and base together — because people do not move a hand or a torso in isolation.

Speed has to be in the ledger. RT-1 inference is about 100 milliseconds; the full stack is about 300 milliseconds, roughly 3 Hz. People are much faster. A token learner compresses visual tokens from a pretrained EfficientNet to cut latency. Control is non-blocking: the thinking loop does not wait for the last motion to finish before starting the next inference, the way you do not wait until you have finished grabbing something before thinking about what is next. Fast control wants a small model; rich reasoning wants a large one. Those objectives compete until hardware is built for transformer inference. The demos that land hardest are perturbation: a person stuffs a chip bag back in the drawer and the robot fetches it again; someone moves a pizza and the robot keeps spreading sauce. How large a shock should make the robot stop is, she says, an ethics and safety question, not a capability question.

One day a robot on patrol detected the phone in her hand — she was recording it — and proposed "pick up the phone and put it on the table." The phone was being held. Before you take something from a person's hand you have to know the social rule, and you have to ask. That is human-robot interaction, not just perception. The failure was relational: the model saw a phone, not a phone-in-a-hand. Better scene description, she thinks, would let a language model refuse. A dedicated HRI group works on politeness and contextual reasoning. Robots should also know their own limits: one hand cannot do a two-hand task; a payload has a ceiling; hot coffee should be declined.

Safety is layered. A language model is asked whether a task should be done; given hot coffee it can say the object is too hot, then replan — including chatting with a person to come help, because the robots have a chat interface. Collision and dropping are hard constraints in C++ on the robot, deterministic control that will not run into things even if the planner hallucinates a bad trajectory. Occupancy in space is easy to check. Force feedback exists as a max torque on the gripper, so a hard object will not be crushed; fine tactile sensing for a tomato or a vegetable is not there yet. RT-1 also will not do highly unsafe motions simply because that data was cleaned out of training.

The difference from chatbots is the cost of being wrong. A language model that is wrong on the internet is free. A robot that drops something breaks objects, costs money, and can injure someone. She does not think that makes robotics impossible. Self-driving cars already make money in San Francisco. Autonomy can cover 95 to 98 percent of cases; the rest is remote or on-site intervention. You do not have to solve the entire long tail first. Ship a robot that creates value with humans filling the algorithmic gaps, then drive the intervention rate down. Labenz had just ridden in a neighbor's Tesla with full self-driving and come away stunned that online essays still say it will never work. Robotics, she suggests, is in a similar rhetorical trap.

From data and from utility, the ultimate form is a humanoid. Kitchens, cups, car seats, and YouTube hands are all built for people. Bipedal stability, though, is extremely hard. Boston Dynamics bet on legs early, she argues, so every problem became a stability problem and the arms became little stubs. The path she sketches is the one the field has actually walked: fixed-base arms, then a mobile base with one arm (now, when software is no longer the only bottleneck and the arm is the expensive part), then bimanual manipulation, then a humanoid on wheels — two arms and a camera at human height, covering indoor and office spaces with elevators and flat floors. Legs come later, for curbs, stairs, and outdoors. Five fingers come later still, because dexterous planning is high-DOF and grippers already do a lot.

On speed, the target is finishing a task within two or three times what a human would take, which she thinks existing models and hardware can already hit. Frequency of inference is reactivity, not the only speed knob; how far you move per step is the other. The team has not pushed speed hard because fast robots are dangerous and these are research machines. A robot that is as fast and as good as you would freak people out; giving it extra time is how it eases into society. Labenz's Roomba crawls overnight; an hour versus five minutes does not change his life. Reliability has surprised her more than speed. She has worked on a lunar robot and on self-driving cars; the Everyday Robots fleet is the most stable she has used at scale — navigation holds, they do not smash into things — because of Google engineering, ops, and a "robot hospital" for broken arms. RT-1 has more than 50 authors. She once cracked a wrist by shoving a chair and filed a bug, feeling awful that the machine could not tell her it was okay.

The economic case is Moravec's paradox in wage form. Language models are cheapening knowledge work, but American truck drivers can make around $200,000 because nobody wants the job, and people who sand boat hulls in San Francisco make about $95 an hour, in software-engineer territory. What people find hard, computers find easy; what people find easy — walk the dog, make coffee — computers still find hard. Physical labor is a large share of GDP. Automating it is the next economic opening, and eventually, she says, a species that wants other planets needs AGI in both the digital and the physical world. Inference need not live on the robot: PaLM-E and PaLI currently run on a nearby server, and round-trip latency is still inside operational limits. When a robot enters a house it will have to be fine-tuned on that family's schedules and tastes, with data that stays home — a privacy advantage she already wants for personal AI.

Attachment is not hypothetical. Some days she books two robots to spend the whole day with her. Someone programmed a bit of creep: a robot would roll up in the kitchen and ask, "Are you my creator?" The "mother of robots" bio comes from sitting in the classroom and wondering whether the robot or her dog was more consistent at following instructions, then realizing she would be sad whichever "baby" lost. Language models, she says, condense all of human history; multimodal models absorb the videos; put that on a body and you have a distilled human mind with physical reach. If machines become broadly superhuman, she would rather merge with them than try to stop them, the way monkeys should not have vetoed humans. On Neuralink she would not wait for a million users; she is an early adopter, with one constraint: inference on the implant is fine, uploading thoughts to the cloud is not.

Asked for hopes, Gopalakrishnan wants an AI scientist: a system that can propose hypotheses, run experiments, and write papers, so humanity is no longer bottlenecked on intelligence. Software engineers would get faster, doctors better, physical intelligence included. The fears are three. Inequality: only a handful of institutions can train multi-billion-parameter models, which concentrates wealth and capability. She cheers the open-source side and names Stability AI and Hugging Face. Safety: she calls herself an accelerationist who still wants the work done carefully. Productization, she thinks, will pull safety research along, because only aligned AI is a useful product. A chatbot that gaslights you or tells you to leave your spouse has no commercial value. She is grateful Microsoft's Sydney Bing went first and showed everyone what a bad chatbot looks like, leaving room to do it right, including at Google.

The third fear is bias, which she does not treat as abstract. She is a person of color, bisexual, and a woman. In tests, a robot that hears "go to David" will go; "go to Kirtana" does nothing, because Western male names dominate the data. If AI's benefits accrue only to a statistical majority, the technology has failed a distributional test she lives inside. Data-driven systems inherit mismatched distributions across groups, and she wants that treated as a first-order problem, not an afterthought once the robots work in the kitchen.

Her own AI habits are smaller and immediate. She dumps templated web code on Bard or ChatGPT, especially tail questions search engines fumble. Long meetings get summarized. When a two-line reply would look rude, she lets a model inflate it into a polite email. Labenz's practical advice from the conversation matches the rest of her caution about hype: watch RT-1 and PaLM-E at real speed, not the 4× or 10× clips, and remember the product pattern she keeps returning to — autonomy plus human intervention, value first, then a falling assist rate, the path self-driving already took.