Robots Have No Wikipedia. Chelsea Finn Is Starting With Three Buildings.

Chelsea Finn cofounded Physical Intelligence with four others while remaining an associate professor of computer science and electrical engineering at Stanford, still advising students on leave. Elad Gil, hosting for No Priors, walks her through a path that started more than ten years earlier, at the beginning of her Berkeley PhD: neural-network control, mapping camera pixels straight to motor torques on a robot arm. At the time it was not popular. It is now a lot more accepted. What was already clear, she says, is that you could train robots to do pretty cool things, and that getting one of those things to work in many scenarios with many objects was the major challenge.

Ten years ago they were training arms to screw a cap onto a bottle, use a spatula to lift something into a bowl, do a tight insertion, hang a hanger on a clothes rack. Impressive in a demo. Move kitchens or swap objects and the thing fell apart. Training a robot to do one task was never the hard part. The hard part was still doing it somewhere new. The next decade of her work — a year at Google Brain between the PhD and Stanford, then a lab, then Physical Intelligence almost a year before this taping — sat on that kink: broader datasets, and different learning routes, reinforcement learning, video prediction, imitation learning.

Physical Intelligence’s slant is the opposite of the usual robotics company. Past firms, she says, went deep on one application, got stuck, and found it extremely hard to broaden. The cofounders want a big neural network that could ultimately control any robot to do anything in any scenario. A practical piece of that vision is using data from every platform they can: six joints or seven, one arm or two. They have seen a lot of evidence that rich information transfers across embodiments. Iterate the robot and you do not have to throw the data away. She has lived the other version: a new robot revision, a policy that no longer works, a painful climb back to where you were.

Getting off the ground meant scaling data collection, and the analogy to language breaks immediately. There is no Wikipedia, no internet of robot motions. Real robot data in the real world is what has fueled machine-learning advances before, so they have to collect it. The bread and butter is teleoperating robots in the physical world, puppeteering them the way the ALOHA work did, recording motor commands and camera images as the robot’s experience. They are exploring other ways to scale data. That is still the core. The late-October release showed complex tasks — folding laundry, cleaning tables, constructing a cardboard box — trained in one environment. There was some generalization: shirts never seen before, shorts never seen before. The degree was very limited. You also could not prompt it beyond fairly basic things in the training data.

Language interaction and generalization to different environments are the present focus. Architecture-wise they use Transformers and pretrained vision-language models, which lets them pull in the internet. A result from a couple of years earlier: with a VLM in the loop, a robot could do tasks that required concepts never in its own training data. The famous example is passing a Coke can to a picture of Taylor Swift. The robot had never seen Taylor Swift in person. The internet has lots of images of her, and the pretrained weights transfer that. They are not starting from scratch.

Asked what actually gets you to generalizability — more data, more compute, post-training, reasoning modules — Finn names the boring thing first. More diverse robot data. For that October release they collected in three buildings, technically. The internet that fuels language and vision models is way more diverse than that: pictures and text from lots of people. Quantity is not the point. Many more places, many more objects, many more tasks. Bringing robots into lots of different places has a byproduct she likes: you learn what it takes to keep a robot operational out there, which you will need for any real deployment. They are also looking at videos of people, web data, pretrained models, and basic reasoning — a dirty shirt into a hamper, a sandwich without pickles if the user is allergic. Number one is still more diverse robot data. Breadth is hard to quantify. There is no score for how different one kitchen is from another. The rough proxy is the number of buildings, the number of scenes.

Physical Intelligence has been quite open: some weights open-sourced, details in technical papers, and, more unusually, designs of robots given to hardware companies. People are shocked, she says — what about the IP, the confidentiality? The choice is intentional. The field is at the beginning. The models and the robots should be so much better in a year, in three years. They want the community and the hardware ready when generalist models arrive: more robust robots, people who understand what it takes to use the models. The second reason is hiring. Fantastic researchers want to work at companies that are open, where they can get credit, share ideas, talk about their work. Solving this, she thinks, will take those people.

The last reason is the one she lingers on. The biggest risk with the bet is that it will not work. She is not really worried about competitors. She is more worried that no one will solve the problem. Robotics is very hard, with many failures in the past. Recognize an object in an image and missing a frame is fine. Miss a grasp, or fail to make contact by a millimeter, and the outcome of the manipulation changes completely. Data is hard to collect. Hardware is hard. Successful machine-learning products — recommenders, language models, image detection — mostly emit something a human consumes and can check. A robot doing something autonomously does not hand you a commanded arm position to validate. You need scenarios with tolerance for mistakes, or humans and robots working together. That is a lot of the motivation for language interaction: so a person can say how they want the robot to help, and correct it.

Gil asks where the models will show up first, given constrained wins already in pick-and-pack, distribution centers, automotive manufacturing. As a company they are focused on the long-term problem, not one application, because of the failure modes that come from going deep on a single job. She does not know where the first applications will be. The deployment question is the one she wants the field to take seriously: either design for error, or design for collaboration. Language is a first attempt at the second.

Some companies bet on humanoids because the world is built for people. Finn has one in her Stanford lab. On one hand they are really cool. On the other, a little overrated. Practically, the field is bottlenecked on data. The argument that a human form is easier to mimic, and therefore easier to collect data on, is one she has heard. If you have actually tried to teleoperate a humanoid, she says, it is a lot harder than a static manipulator or a mobile manipulator with wheels. Optimizing for collection speed and diversity is what they are doing: cheap robots, teleoperation interfaces you can run quickly, lots of data. If you ever have more data than you could want, what remains is research, compute, and evaluations. The host mentions a viral fake Kim Kardashian video of a robot trailing her with shopping bags. Finn wanted one too, as a joke. The industry reminder is colder: fix data and operating cost before you fuss over form.

The AI community, she thinks, underestimates how much intelligence goes into motor control. Many years of evolution got us hands that work this way; many animals still cannot do it. Making a bowl of cereal or pouring a glass of water is packed with complexity. Embodied, physical intelligence is core to intelligence, and underrated next to the less embodied models that currently soak up the hype about reasoning.

Gil loved her ALOHA paper and asks what, in the last two or three years, made so many people start companies. For Physical Intelligence there were a few turning points where the field felt faster. One was SayCan: plan with a language model at a high level, plug it into a low-level model, and get long-horizon tasks. One was RT-2, the Taylor Swift result, web data into the robot model and better generalization. A third was RT-X: a huge effort to aggregate robot data from different research labs into a common format and train on it. They could send a checkpoint halfway across the country, and a graduate student running it on their own robot more often than not beat the model that lab had specifically iterated. Pooling across embodiments was starting to work. ALOHA and later Mobile ALOHA showed that teleoperation plus imitation could take on complicated dexterous manipulation. A follow-up had a robot tying shoelaces, a fun project because someone had said they would retire if they saw that. They did not retire. After Physical Intelligence started, she adds, that itself was a sign to others that if the experts were willing to bet, something might happen.

The day of the taping they had put out what they call a hierarchical interactive robot, Hi Robot. Two problems. First, a task that takes minutes — making a sandwich — does worse if you train a single policy to spit the next motor command from images than if something is thinking through the steps. That is the hierarchy. Second, annotated segments give you a policy that can follow “fold the shirt” or “pick up the cup.” What you actually want is interaction: I am a vegetarian, make me a sandwich, I am allergic to pickles, and midway, hold off on the tomatoes. The gap between a one-step instruction and those situated corrections is large. Their system has one model that takes the prompt, reasons, and outputs the next step, such as pick up the tomato, and a lower-level model that turns “pick up the tomato” into motor commands for the next half second. They got vegetarian and ham-and-cheese sandwiches, a grocery-shopping example, table cleaning. She liked it because it was cool, and because it seemed like the right approach to the actual problem.

Self-driving added lidar and other sensors on top of vision. What is missing on robots? They have gotten very far with RGB: one or more external base cameras on the scene, plus cameras mounted to each wrist. Finn would love to give robots skin. A lot of tactile sensors are far less robust than skin, far more expensive, or very low resolution. Wrist-mounted RGB, they have found, is very helpful and probably gives a lot of the same information tactile sensors would. Smell she has argued for with cofounder Sergey Levine, especially for sandwich-making; they have never attempted it. Audio can be redundant in a useful way: hear something unexpected, get alerted, even if you could also have seen it fall. Redundancy is robustness. Extra sensors are not the current priority. The bottleneck is data and architectures.

The other gap is memory. Most of their policies look only at the current image frame. They cannot remember even half a second prior. She would much rather add memory to the models before adding other sensors. Commercially viable robots for a number of applications, she thinks, can exist without those extras. Time frame? She has no idea. Compared with self-driving, robotics is harder in some ways: much higher dimensional — even their static robots have 14 degrees of freedom, seven per arm — often more precise than driving, and starting with less data. It is easier in others. Driving has to cover the entire distribution to be viable: any intersection, any time of day, any pedestrian. Robotics has commercial cases that do not need the whole huge distribution, and less safety risk. That is why she is optimistic. The Waymos she sees in San Francisco are encouraging.

Gil’s sharp observation: a couple of dozen self-driving startups ten to fifteen years ago, and in the US the industry largely consolidated onto Waymo and Tesla, two incumbents, with most of the rest gone. Will robotics replay that? A year ago, Finn says, the landscape would have been completely different; new players have been pouring in. That self-driving field looked like that suggested it might have been a bit too early. Deep learning has come a long way since. Robotics five or ten years ago was too early. It might still be too early. How hard self-driving has been is a testament to how hard it is to build intelligence in the physical world. What she liked about the startup environment, versus Google — an amazing place in many ways — is concrete. Taking a robot off campus was almost a non-starter for code-security reasons. Diverse data needs robots in many places. A smaller company moves faster, with less red tape. Big companies have capital and can last longer. They will also move slower. Advice to someone starting a robotics company: learn as much as possible, quickly; deploy, iterate, get the robots out there. She is not sure she is the best person to give startup advice. She has only been an entrepreneur for eleven months.

Observational data of people — YouTube, purpose-recorded video — is something many groups are trying. Finn thinks it can have a lot of value and that by itself it will not get you very far. Watch an Olympic swimmer. Even with their strength, their practice at moving their own muscles is essential. You will not learn to hit a tennis ball well by watching the pros. The analogies sound contrived because they are about experts. That is the point: humans are already experts at low-level motor control; robots are not. Robots need experience from their own bodies. Human video can expand on that experience. Data from the robot itself is still essential. Collection looks like puppeteering: motor commands plus camera images. Autonomous experience will play a huge role, the way reinforcement learning lets a language model bootstrap on its own output after the initial model exists.

What will robots look like in n years? She does not know exactly. Her bet is a wide range of platforms. Levine likes to call it a Cambrian explosion of hardware types, once the intelligence can power all of them. A kitchen is the analogy: many devices that each do something, rather than one machine that cooks the whole meal. One arm optimized and cheap for kitchen work; other hardware for folding clothes or dishwashing. Speculation, and different from what a lot of people picture. Gil cites *The Diamond Age*’s matter compilers, and the supply-chain argument that scale collapses hardware onto a few platforms because that is cheaper to manufacture. Finn thinks a specialized, cheaper, faster piece of hardware for a use case is very possible, and very hard to predict. Then the joke: maybe robots in the supply chain will manufacture any customizable device you want. Robots all the way down.