The Robot That Brought the Beer Was Being Teleoperated

Lex Fridman opens by asking Sergey Levine, a Berkeley professor working on deep reinforcement learning, robotics, and computer vision, what separates a state-of-the-art human from a state-of-the-art robot. Levine says the question is trickier than it looks. Some things that seem hard are easy, some that seem easy are hard, and the gap between a robot's physical body and what it can do on its own is enormous.

He cites a 2004 Stanford video that robotics-learning researchers love to show. A prototype home-assistance robot called the PR1 tidies a living room, puts away toys, and brings a beer to someone on the couch. It looks like the future arriving early. The punch line is that a person is controlling every move. Give the robot a human brain, Levine says, and the remaining hardware gap is not that large. Spend some money, do some engineering, and you can almost close the body problem. "The intelligence gap, that one is very wide."

The size of that gap, he argues, is measured by how open the world is. Lock a robot in a factory, tell it where every part sits, and rigidly program its motion, and it can be superhuman: faster, stronger, able to lift a car. Let anything vary the way it varies in a kitchen, and the system falls apart. Bodies can still get better. The bottleneck right now is the mind.

Fridman asks how much of human cognition is nature versus nurture — if it is all learned, AI has more hope. Levine is not a biologist and will not pretend to be. He reframes the question as sufficiency: what is the minimum we need, not how humans actually do it. Face recognition may be evolutionarily loaded, but daily life is full of things evolution never prepared us for: cars, spaceflight, a joystick attached to a robot arm you have never seen.

Hand someone that joystick and tell them their life depends on opening a store with it, and they will muddle through after a few clumsy seconds. That flexibility is not an evolved skill, and it is exactly where current robots collapse. Fridman wonders whether it sits on an iceberg of pretrained common sense. Levine agrees the iceberg is real, but thinks it is built over a lifetime of experience, which sounds a lot like what machine learning is supposed to do.

Distilling unstructured experience into common sense is still hard, he says, maybe because the field has been too rigid about supervised input-output learning. The sharper scientific question is where the experience comes from. Do you download the internet and cram it into a giant language model, or does the machine have to try things, observe outcomes, and choose which parts of the world to touch? A system with a bad world model might knock over a water bottle thinking that will raise its utility. Going out and doing it is exactly the supervision that tells it not to.

That perpetual improvement loop — try what your current model thinks will work, see whether it does — is what he suspects will distinguish tomorrow's methods from today's. Classically, reinforcement learning is framed as maximizing a utility, and every action serves that utility. An alternative he likes, though it is not mainstream, is to explore first, build an arsenal of cognitive tools, and only later be told to fetch the coffee.

Asked for the big problem of robotics, Levine gives two answers. The pragmatic one is a system that can do whatever a human user sets, within physical limits, with a little extra training or trial and error — the way a teleoperator would figure it out. The second answer is closer to why he is in the field. He cares less about how good robotics can get than about what robotics can teach us about intelligence.

Robotics forces you to throw away crutches. You have to handle perception and control, and you have to handle their integration. The classical modular move is to solve the parts, wire them together, and hope the whole works. The last couple of decades suggest that studying the thing as a whole yields different solutions than studying the parts. Robotics also puts Moravec's paradox in sharp relief: calculus is hard for people and easy for machines; drinking from a cup is the reverse. When the discrepancy is that blatant, something important is missing.

One reason manipulation is so hard is supervision. Computer vision, up to some abstraction, can be cast as a tightly supervised problem. Robotic manipulation cannot, at least not well. You could label motor commands at every timestep and train on that, but it does not work well and it looks nothing like how people learn. Parents do not tell you how to fire each muscle in order to walk. The low-level detail, Levine says, we mostly figure out on our own.

In 2014 Levine's group published some of the first end-to-end reinforcement learning for vision-based manipulation skills, which he says was inflammatory in robotics at the time. The actual claim was narrower: for perception plus control, treating them together beats splitting them. The task was simple — insert a red trapezoid into a trapezoidal hole. The modular baseline detected the hole with a pose estimator, then moved the arm. The end-to-end baseline mapped pixels to torques.

The end-to-end model put less pressure on perception. It did not need an exact 3D pose. It needed errors distributed so that horizontal mistakes hurt more than vertical ones, because vertically the robot can just push down until it cannot. Combine the two, and you trade off errors between components: each piece can be weaker and the whole can still be better. The same pattern shows up in animals as the gaze heuristic. To catch a ball you keep it fixed in your field of view rather than solving a system of differential equations. Baseball players, pilots judging a collision, and frogs catching insects all use versions of it.

Fridman asks for a canonical robotics problem whose solution would unlock a lot. Levine does not have a clean one, because the difficulty is flexibility, not doing one thing extremely well. Around 2016 he would have said grasping: huge commercial value for warehouses, and different objects need different strategies. People have since gotten good enough at it that he is no longer sure it is the right guiding problem. The methods that worked were not inverse physics. They were learning plus lots of simulation or real trial and error, which meant you no longer had to solve the geometry or the physics by hand.

Grasping is hard because variety is huge. Boxes and plastic bags want different strategies, and properties beyond geometry — flexibility, spill risk, which bits make a secure grasp — start to smell like common sense. Fridman asks whether general robotics therefore requires human-level intelligence. Levine turns the question around: studying robotics may be how we learn to put common sense into AI. An image captioner that calls a person in a fur coat a teddy bear is not living in our world. It lives in a universe of pixels and English sentences, where you never have to put on a coat so you do not get cold. Systems forced to deal with our mess might have to acquire common sense to maximize utility. Current systems can take shortcuts.

Fridman had assumed you must solve intelligence before you solve robotics; Levine has been arguing the reverse. Can the robotics problem be solved purely through learning, without injecting expert rules? Levine thinks the dichotomy is too sharp — a person still writes the algorithm and turns on TensorFlow — but in spirit, yes: many problems that needed meticulous manual engineering can be handed to automated optimization. The idea is not new. In the 1960s you would have said the robot was doing optimal control. Writing differential equations for feedback minimization versus training a neural net is not as large a difference as the branding suggests. It is pushing optimization deeper into the system.

The difference that does matter is that learning systems, in principle, get better the more they do something. That is a powerful break from classic optimal control. As for symbolic AI, Levine's biased history is a straight bloodline. First, rational decisions were logical manipulations of true-or-false predicates. In the 1990s the same inference acquired probabilities. Then people learned the probabilities. Then they stopped specifying the nodes and put a big neural net on top. It is still inference for decisions and optimization for learning.

Train a network to predict what will happen if the robot acts, then invert it with probabilistic inference to find actions that lead to a desired outcome, and you are doing something that looks like logic. You have a model of the world. It just happens to be expressed by a neural net, and you manipulate that model to answer a query.

What symbolic systems had, Fridman says, is a story you can follow. Neural nets do not tell that story. Maybe wanting a convincing explanation is a human quirk. Levine ties explainability to performance: you want the system to explain itself so that when it screws up, you can figure out why. That is a bigger problem than it looks. The system might screw up, then screw up the explanation, or a bug might mean it was never doing what you thought. He would file explainability under verification and validation, of which it is one component.

Fridman wants another dimension: storytelling that is not about debugging, the way people live inside one another's stories in politics and academia. Levine points to work by his former collaborator Jacob Andreas, now a professor at MIT. Andreas trained a policy that reads a natural-language instruction, then treated the sentence not as a human command but as the policy's internal state. Faced with a new task, it tries candidate descriptions — go to the red house, the blue room, the green plant — until one produces reward. You can read the string it settled on. Language becomes internal state.

Fridman pushes further: add the convincingness of the story to the reward. In a world of Twitter and fake news, he says, that is a scary thought — that how convinced you are of the story around the facts might matter more than the facts themselves.

What is reinforcement learning? Levine's working definition is the modern incarnation of learning-based control. Classically it meant learning from reward and punishment. Today the term is broader: any controller that uses data to get better, making sequential decisions that maximize utility. Image classification is a decision problem, just not a sequential one. Controlling a chemical plant is. Recommending YouTube videos is. That breadth is why RL is appealing as a path across a large swath of AI.

Mathematically, supervised learning is a special case: treat the loss as negative reward. Supervised learning assumes someone told you the right answer and the data are clean. RL relaxes those assumptions. In practice, if you actually have a supervised problem, use supervised methods — they are easier. Fridman notes that supervised learning has already been effective on narrow problems, while RL still feels like the dream of AI without the same deployed impact.

The practical gap Levine names is the ability of RL algorithms to use large amounts of prior data. In most settings where you want rational decisions, you cannot deploy a policy that does crazy things and learns purely by trial and error. You have logs from some other policy. If you could get a decent policy out of those logs and then fine-tune a little, applications would grow very fast. That problem is offline RL, also called off-policy or batch RL, and he thinks research is getting closer.

At a fundamental level, RL is about learning models that answer what-if questions: what would happen if I took an action I have not taken? Model-based methods ask what state you would get; value-based methods ask what utility you would get. They are not that different. Answering what-ifs is hard because they are about things that did not happen. On-policy learning works, clumsily, because you make the mistake, observe the counterexample, and stop. Offline data forces you to confront the counterfactuals without that loop. A policy maps observations to actions. On-policy data comes from the current policy; off-policy data comes from someone else. On-policy data is more useful. Off-policy data is usually what you already have.

Offline RL's central trap is that you cannot trust the model for arbitrary actions. If the dataset never shows anyone steering a car onto the sidewalk, asking what would happen if you did will probably get a wrong answer from the value function. You have to judge whether a prediction is trustworthy. Density estimation can flag actions that look nothing like anything you have taken. Regularizers can tell the learner not to answer questions it cannot answer.

Fridman asks whether the bottleneck is data or methods. Levine says for offline RL it is currently an algorithms problem, which excites him: algorithms need smart people thinking hard, not a huge engineering spend on data or hardware. The core issues are close to causal inference — a statistical model predicting unseen things, which generalizes if it found the right structure and fails if it found spurious correlations. You can find the trustworthy regions, try to make the model generalize better, or both.

Mixing is the recipe he likes: most of the iceberg is offline data, then you send the robot out to fiddle with the particular lock it has never opened. That extra sliver of real interaction finishes the new task. Asked for the most beautiful idea in RL, he says it is that you can obtain a near-optimal controller without a complete model of the world. From a controls perspective that is weird. Classically you write the equations, then solve for the thing that maximizes performance. RL theory gives a principled way to optimize a quantity when you do not know the governing equations.

Deep RL is just RL algorithms plus high-capacity neural nets. The combination looks arbitrary until you remember what features used to mean. In the late 1990s and early 2000s, many methods had beautiful math — they reduced to convex optimization — but they needed special inputs, representations where classes were linearly separable. Someone had to write the code that turned an image into those features. Neural nets learned the features, which was great for images and even better for fields that had never had good features. Reinforcement learning was one of those fields.

Try to imagine a linear feature for the value function of Go, or even the optimal policy. People wrote lists of expert hunches: is the knight in the middle of the board? It did not go anywhere. Chess at least has experts who know what to look at. Robotics often does not. Fridman asks about the limits of that representation power. Levine says the question is hard to answer because other bottlenecks hit first. He cannot get a robot to learn dishwashing in a kitchen because the net is too small; he cannot get it because trial-and-error in the real world runs into other problems.

Put the robot in your kitchen and it will break all the dishes, then have nothing left to wash. That sounds like an engineering detail. It is a scientific problem. A person breaks one dish, is more careful with the next, and if they break them all, goes and gets more. That scaffolding — try again, ask for help, look things up — sits outside the classic RL formulation. So does the reward function. How do you know you poured the cup of water correctly? You might need an entire computer system just to score it, which is inelegant. These gaps suggest a shortcoming in the problem formulation itself.

In Go, as with AlphaZero, there are no broken dishes. The bottleneck is compute and the money to buy it. In the real world you need to reuse data and build the iceberg. Hoping a machine in a vacuum masters a complex task in minutes, the way people appear to, may be too optimistic. It also may not have to. It needs a lifetime of doing many things, so that previous experience makes new things efficient. That is the theme under multitask learning and meta-learning.

Fridman brings up Tesla Autopilot, then approaching a million vehicles, with the computer driving perhaps 30 to 40 percent of the time and humans the rest, in a setting where mistakes cost lives. Levine says reliability — knowing the limits of your capabilities — is the same problem as trust in offline RL. If the model cannot predict a behavior pattern accurately, do not use that prediction to change the policy, and do not trust it at deployment either. He is optimistic that a few years of work on that question will help.

Simulation, he says, is a pragmatic tool that can make useful systems work now. In the long run, machines have to learn from real data, because that is the only way they improve perpetually. If they rely on simulated data, the simulator becomes the bottleneck. The rule is general: any human-built piece of the machine that does not improve from data will eventually hold it back. A hand-designed controller is the same kind of ceiling. Simulation will never surface the scaffolding problems of broken dishes and unknown rewards. Those only appear in the real world.

Asked whether we live in a simulation, Levine has no idea. What he finds interesting is the engineering question: how hard is it to build a virtual reality so convincing that you would rather stay? He likes VR and thinks it is far from that point, but the day he would rather live there is the day most people are a few years from living there — and you do not need to simulate quantum gravity to get there. If that were true for training AI, it would imply it is easier to create a universe than a brain, which sounds weird. His hope for humans in the loop is that a multitask robot with lots of prior data and the ability to learn from its own experience might treat people the way it treats everything else. Whether an understanding of others' goals can emerge from maximizing utility is worth testing before inventing a special module.

On gravity, the common pitfall is assuming that because we know something, we should tell the machine. Things that affect many events and show up constantly — objects fall when you drop them — are easy to learn. You might get Newton's version rather than Einstein's, which is enough to act rationally. Hand-specifying what is obvious in the data is often more work than letting the machine find it. Local minima are a real worry; human civilization spent a long time stuck in superstitious local optima about medicine. That is a failure of the approach, and also, he notes, a failure of human intelligence.

Self-play looks powerful in games. Levine suspects you cannot shortcut it into robotics. You still need machines that get better through natural interaction; then they can play with each other, with people, with the environment. Board-game self-play works because the rules mediate the interaction, and the intelligence that emerges depends on that mediator. Classically, RL treats the reward as an external given. He thinks that is a mistake. Rewards can be a communication medium — how a person tells a robot the objective — or an intrinsic-motivation medium: a general objective that produces useful skills even without a task.

That second path is sometimes called unsupervised RL. His group has used information-theoretic quantities such as minimizing a Bayesian measure of surprise, an idea from computational neuroscientists including Karl Friston. Behaving so that you can predict the world accurately can discover stable niches: in Tetris, clearing lines keeps the board clean and lets you play longer. He is more interested in whether discovering new things can be an emergent consequence of some other capability objective than in curiosity for its own sake. A practical guardrail against weird strategies is an objective that says do not enter states that are improbable under the prior — which is also a good way to do offline RL, as long as you venture out slowly.

On alignment, the Stuart Russell question of making AI success match broader human interests, Levine has no great answer but thinks it matters. His worry currently points the other way. Rather than unintended consequences from objectives optimized too well, he is more worried about objectives not optimized well enough, especially in cars and aircraft. Unintended consequences can still appear far from optimality; he is saying the bottleneck for reliability right now is systems that need to work better. Existential risk comes with any powerful technology: nefarious people, fatally flawed systems, systems that are too capable. Among those, he is most concerned about the first two, especially malicious humans. "Just through all of human history, the nefarious humans that have been the problem, not the nefarious machines."

The best he can do is build the best technology he can and push for responsible use. Has RL taught us about ourselves? He is not sure yet. The interesting behaviors may show up once RL is widely used for decision support in healthcare, education, and social media, in settings where systems can influence people. Fridman hopes the companies sitting on that data will be transparent enough that outsiders can tell whether an algorithm was even involved.

Fridman brings up Rich Sutton's essay "The Bitter Lesson": simple, general methods that leverage computation win. Levine agrees with the high-level idea but not with sitting still until computers get faster. The takeaway is to work on general algorithms. If automated general methods plus data lead to good results, then build methods you can deploy so they go collect experience on their own. That last step is easy in a simulator or a board game and very hard in the real world. How do you collect that data? There are seven billion people, each of whom had to learn these skills at some point. That experience is sitting there. We should try to collect it.

Asked for books that shaped him, Levine's honest answer is Isaac Asimov — a future in which robotic systems have a large place in society, and an attempt to imagine the limiting case of technological advance. He fell in love with the field late, in graduate school. Until around 2009 or 2010 it was not a priority, because he did not think substantial progress would happen in his lifetime. The turn was a seminar with Pieter Abbeel. In the opening lecture Abbeel said he used to chuckle when students wanted to work on AI and hand them a math problem; now he thought the field might see substantial advances in their lifetime. Hearing that from someone who had spent a career on it carried weight.

His advice to students is not to be afraid to spend time imagining the outcome they want. A career, a paycheck, or a state-of-the-art number on a benchmark can be outcomes, but they should not be the main driving force. Sit down and picture what you want a machine, a robot, or a language system to do — almost like a commercial for a future product — then backtrack through the steps. A better image-classification score will not carry that role. A robot butler or a healthcare decision-support system might. That habit makes you rethink assumptions and attack bottlenecks other people are not attacking.

What is his own reward function? Satisfaction comes from confidence that he is working on a problem that really matters. Solving it is less important than spending time on things he believes are worth it, and he tries hard to pick those. The concrete dream is machines that get better the longer they exist. We do not have that today, even if it looks as if we do. The universe has unending complexity, and every machine so far hits a wall — a thin simulator, a labeled dataset — before it runs out of things to see. He wants to build a machine that goes as far as possible and runs up against the ceiling of that complexity.