A one-legged robot learned a backflip from "this is better"

Fridman opens with a joke that has a research question inside it. Abbeel has said that if he could meet one person it would be Roger Federer; when will a fully autonomous robot beat him? For most AI problems, Abbeel answers, software is the missing piece. Tennis is not most problems. Boston Dynamics robots are starting to look like they can run, but they are not at human level around a court, and then they still have to swing a racket. He puts the hardware on a 10-to-15-year clock, and he is not even sure clay is easier than grass. Sliding on clay might be harder.

Wheels and a non-humanoid body would bring the timeline in. The swing itself he is more optimistic about. Park a ball machine on one side and an industrial arm with a racket on the other, pretrain a little in simulation, and he thinks reinforcement learning can learn a forehand and a backhand. How much precision, and whether RL can put spin on the ball the way humans hit the lines, is the part that gets him interested. "You got me interested," he says. Maybe they will set it up someday.

The most impressive physical thing he has seen is still Boston Dynamics: the stair run, the parkour. He does not know how much is hardcoded, and he does not much care for the awe. He met SpotMini at a Mars event Jeff Bezos organizes, trailing Bezos from room to room, and earlier spent time alone with a Spot. Knowing there is little or no learning going on does not kill the psychological connection. Pepper, scripted by Fidelity to moderate sessions with a child's personality, jumping into conversation like a person, was 100 percent scripted. He still had trouble not treating it as someone. In the lab, BRETT — the Berkeley Robot for the Elimination of Tedious Tasks — is a he, never an it, and that pronoun does a lot of work.

Could that feeling be a reward? If you can formalize "people like being around this" as an objective, he sees no reason RL would not optimize for a robot that is more fun, more person-like, or more like a pet. The hard part is the ground truth. Asking a person for a 1.5 or a 10.7 is painful. Asking whether the last five minutes were nicer than the five before is easy. Paul Christiano and collaborators at OpenAI used exactly that: a MuJoCo Hopper that learned a backflip from "I like this better than that." Nobody wrote a backflip reward. The person had a backflip in mind; the robot only knew better and worse, and it converged on the flip. Interactive robots, he suspects, can learn the same way.

When Abbeel first opened Sutton and Barto, before the deep-learning revival, reinforcement learning looked like the definition of intelligence. Fridman wants the intuition: how can you learn anything when the reward is delayed and sparse? Abbeel flips the question. The counterpart is why RL needs so many samples. You take a hundred actions and get a 3. You go again and get a 2. That tells you the second hundred-step sequence was worse than the first, and almost nothing else. Good and bad actions are mixed inside both trajectories. You need a pile of experience before you can separate what is consistently present when the score is high from what is noise.

A policy-gradient update is then almost embarrassingly simple. Make the actions that showed up when things were good more likely, and the ones that showed up when things were not as good less likely. On paper that should need even more data than practitioners already complain about. In practice it learns more than the derivation seems to license. Abbeel's original bet, when Berkeley started on deep RL around 2011 to 2013 with Yan Duan pushing first, was a control-theoretic picture of why.

Rectifier networks, he says, give you piecewise linear feedback control. Linear feedback is extraordinarily successful. A hovering helicopter is a messy dynamical system, and a relatively simple linear controller can still stabilize it. The plant can be complex; the controller can be simple. Plain linear is not enough for everything, so people used to tile the space by hand, with finite-state machines switching among local linear controllers. A ReLU net learns that tiling. In a two-layer net, a hidden unit flipping from active to inactive cuts the space along one direction, not necessarily an axis. Adjacent linear controllers share a lot of parameters, which means they share a lot of already-learned expertise. That was the intuition that let him expect deep RL to work: eat the linear-control dividend, then pave the rest of the space with tiles that reuse what they already know.

That intuition hits a wall on time scale. A life is a high-frequency sequence of muscle-fiber contractions. Deciding to do a PhD is almost unimaginably abstract relative to those twitches, and the outcome arrives a decade later through a thicket of forks. Credit assignment of that kind is beyond any current algorithm. What is missing, Abbeel says, is hierarchical reasoning at a level that simply does not exist yet. Tuning will not invent it.

The idea is older than it sounds. Twenty or thirty years ago there were already systems that could reason over long horizons. They were not grounded. People hand-designed logical and dynamical descriptions that never tied to perception or to real objects, so the plans were sharp on paper and blurry in the world. Deep learning is the missing ground: sensors that can see what is actually there. One path is to bolt the new perception onto the old planners, end-to-end, so a network emits a representation that a traditional dynamical system can plan over. Value Iteration Networks, from Aviv Tamar and collaborators, sit on that path.

A second path, information-theoretic, treated a high-level action as a latent variable that tells you a lot about the near future. Decide to go to the gas station and, five minutes later, whether you arrived was already implied by that choice. They spent a long time on it and got little traction. That sent them back to a plainer question: what is hierarchy even for? Better credit assignment, and therefore faster learning. They did not know how to design the hierarchy, but they knew what they wanted from it, so they optimized for that and watched for emergence. That is RL², led by Yan Duan, a meta-learning attack on learning to reinforcement-learn.

In maze navigation, something did emerge. Consistent motion down hallways is already an option: go down this corridor, and only at a junction decide whether to turn. There was even a bias not to revisit places. Abbeel calls that a sign of life, not yet at real-world complexity. He would rather these problems yield to math. A formal result can leapfrog years of experiments, and the research process itself is reinforcement learning: many trials, rare successes. Most of the last years' progress has still been one experiment at a time. The equation that would have saved two years of work has not shown up.

Fridman treats transfer as one of the hardest and most important problems in AI. Abbeel is torn. The results are already impressive. AlexNet in 2012 was a better ImageNet classifier, and then immediately a bigger deal: features learned there could be fine-tuned for new tasks. Machine learning had usually been one model, one scenario, done. Reusable representation, in scope and impact, is still the biggest transfer-learning success on the board. Scale has since amplified it. Language models trained to predict get reused for other tasks. DeepMind's UNREAL work stacked auxiliary objectives on maze navigation so the agent learned more than a walk. Train a big enough model on enough things and something transfers.

The conceptual fog is still there. What counts as generalization, and at which level? He likes a distinction between learning to master and learning to generalize. Current deep learning, asked to predict the relative motion of the planets, would do pretty well. If a massive new body then entered the solar system, it would not know what happens next. Physicists are doing a different job: hunting the simplest explanation that still covers the facts, all the way to a master equation for the dynamics of the universe. Deep learning has not pushed that direction hard. He is not even sure it should, but it is a kind of generalization current methods do not give you.

Vladimir Vapnik dreams of an E=mc² of learning, a general theory. Abbeel finds the pursuit interesting, then recasts it as a search for modularity rather than a single equation. Blind people reuse visual cortex for other functions; parts of the brain can be rewired. Not everything is interchangeable, but neocortex looks fairly modular. If you could design something equally modular and let it grow more capable, that would be a realistic principle to shoot for, not a fantasy theory of everything.

Asked whether he is more excited about imitation or self-play, Abbeel starts with why self-play is beautiful. Ordinary RL starves you: never succeed, never get a signal. In self-play you sit on both sides. One of you wins, one of you fails, and you see the contrast between two versions of yourself. Every episode is a labeled comparison. Whenever a problem can be rewritten that way, learning gets much faster than in a typical environment. So far that trick lives mostly in games with a natural opponent. If someone could turn "build a hut" into self-play, a robot would learn it far more quickly. That would be the mathematical leapfrog he wants: a formalism that turns any RL problem into a game that always emits signal. Reality is that for many tasks nobody knows how.

The fallback is detailed reward for intermediate progress, which is expensive. Once you are spending that much time shaping reward, why not just demonstrate? Teleoperation has high signal-to-noise: the robot actually lives the trajectory. Ten minutes can teach a basic skill, pick up the bottle and put it on a target no matter where the bottle starts. More interesting is third-person teaching, where the robot only watches you and has to translate your hands into its hands. Abbeel puts a 2018 breakthrough, led by Chelsea Finn, on that line: almost machine translation for demonstrations, meta-learned from few examples. It opens a lot of ways to learn faster.

Fridman works on autonomous driving and asks whether third-person watching transfers there. For cars, Abbeel says, first versus third person is not an important distinction. Getting from here to a point a couple of meters ahead is well understood; the only difference is who turns the wheel. Manipulation is the opposite: contact forces are messy, and the viewpoint split matters. Imitation versus RL is still a live choice for driving. Imitation gives more signal and, in its standard form, has no concept of a goal. Purely reactive behavioral cloning will not generalize the way you need. Inverse-RL flavors of imitation that infer an objective get much closer. A policy without a notion of goals is hard to take as far as you actually want.

Simulation can help all of this, and it does not have to be one perfect world. You can chase a single ever-more-accurate simulator, or you can build many, none of them good enough that training in one would transfer. Train in all of them and a policy that is good in every slightly wrong world has a chance in the real one, which is just another draw from the same distribution. "We do live in a simulation," he jokes, then takes it back: if we do, it is a very advanced one.

Safety, for Abbeel, starts as an engineering fact. Robots are physically strong. A mistake can do damage. Cars are robots that happen to go fast. That is a different layer from the long-term worry about systems smarter than us. Simulation helps you test the practical layer. Then he reaches for the California suburban driving exam: around the block, pull over, a stop sign, pull over again, done. We accept that tiny sample as representative of a hugely complicated human skill. A self-driving car that passed the same loop would convince nobody. What is missing is unit tests for robots, especially at software-update time. How do you know the new stack is more capable on everything, and that nothing bad crept in? He says there is no real solution yet, and that it is a research direction worth having.

Fridman asks whether the sea of policies is full of kindness or of exploitation. Abbeel splits the human case in two. Evolution preloaded pain, hunger, and thirst; those were not taught, and they make learning easier. Humans also evolved to get along inside the tribe and to need extra teaching to be decent to the out-group. Steven Pinker's *The Better Angels of Our Nature* is his pointer to the long arc: whatever teams we pick, violence has fallen. Kindness, in that picture, is a slow optimization across generations, plus some innate scaffolding.

Can you teach that to an RL robot, even teach it to love and be loved? He answers with dogs. Their reasoning is sophisticated and still not human, and a lot of human happiness rides on a dog being excited when you walk in. If the dog is away, the house hits different. Strong affection, he says, does not require human-level reasoning, so he does not see why an AI could not reach the kind of bond people have with each other or with friendly animals. Whether that bond is good for us is a separate question. Fridman offers a rewrite of Musk: love is the objective function, and RL is the answer. "Oh, maybe," Abbeel says.