Leslie Kaelbling: You Buy the Ticket Before You Can Plan the Airport

Leslie Kaelbling tells Lex Fridman that she first got excited about AI in high school, reading *Gödel, Escher, Bach*. The book showed how primitives combine, how simple parts make complex things, and what kinds of programs might generate intelligent behavior. Robots arrived later, as a job. She finished a Stanford bachelor's in philosophy, was wrapping a computer-science master's, and got hired at SRI's AI lab to make a robot do something. That, she says, is what actually got her interested in robots.

Stanford had no undergraduate CS major then. Philosophy did have a track that became Symbolic Systems: logic, model theory, formal semantics of natural language. She calls it perfect preparation for AI. In her graduating class, slightly less than half went on in computer science, slightly less than half to law school, and one or two stayed in philosophy. There were not enough people chasing "create intelligence" for it even to be a conversation about which major you should pick.

Fridman asks whether AI researchers should be part-time philosophers. The parts of philosophy closest to her work, she says, are belief, knowledge, and denotation — already quite formal, one step from routine computer science. She is a complete materialist: she sees no reason a robot cannot be behaviorally indistinguishable from a human. Whether it is distinguishable internally, a philosophical "zombie," she does not know and does not care much. The gap between current robots and people looks to her like a huge technical gap, not a philosophical one.

The first robot she worked with was a follow-on to Shakey, built after the original Shakey people had left. By the time she arrived, Shakey itself was sitting in a corner of someone's office, dripping hydraulic fluid into a pan. She still wants everyone to read the Shakey tech report. It is stuffed with ideas the field keeps reinventing: A* search, symbolic planning, learning of macro-operators, low-level configuration-space planning, vision.

Shakey was a mobile robot without arms. It pushed objects with its base. The baseboards were painted black so vision could localize it on a map. It detected surprising objects, planned and replanned from what it saw, and reasoned about whether to look and take pictures. Space was represented at several abstractions: something like an occupancy grid at the bottom, symbolic rooms and connectivity at the top. Multi-level maps were not a deep-learning invention. They were already on a 1960s robot that leaked oil.

At SRI her advisor was Stan Rosenschein, later her thesis advisor. He was pushing situated automata: the tools of logical reasoning matter, but perhaps only for the designers analyzing a system, not for the system to manipulate inside its own head. You can prove a theorem about the robot's behavior in logic while the robot itself is not proving anything. That distinction, she says, is the one they were trying to use to make robots do stuff.

She had zero robotics background — no control, no sensors — so they reinvented a lot of wheels. Advantage or hindrance? She is a firm fan of wheel reinvention. You learn by doing it. You still need the pointers later so you can see what is really going on, but you appreciate a good solution more after you have found a bad one on your own. Along the way she reinvented reinforcement learning and called the reward "pleasure," a name she still thinks is more fun.

Asked for a history from the 1950s, Kaelbling reaches for oscillation. Things become fashionable, go out, something else becomes cool. Early days were cybernetics and control, homeostasis, robots that plugged into the wall when they needed power and then rolled off to work. People decided that was not close enough to "real" intelligence and turned to expert systems: I understand a steel mill, I explain it, you write it in logic, the computer does the job. That did not work out.

The interesting part, she says, is what happens when a thing stops working. The community changes not only methods but problems. We do not have better ways of doing what the expert-systems people were trying to do. We mostly gave that problem up and switched. Some would say you never give up; you shrink the group and put the question on a shelf for twenty years. She thinks that is also right. A third option: you decide the problem is malformed — that you cannot make superficial symbolic reasoning behave like a doctor until you have had the sensorimotor experience of being a doctor.

She is not anti-symbolic. She is anti the expert-systems assumption that humans can articulate their knowledge into logical statements. That is a capability problem, not a cost problem. Everyone is a vision expert; nobody has introspective access to how they recognize a water bottle, and even 1980s researchers knew not to ask for those rules. They retreated to supposedly speakable skills — what to wear, how to set up a microphone — and found post-hoc rationalizations sitting on perceptual judgments the experts still could not define. Fridman tries "so the hard part is encoding it for the machine." She says no, no, no, no. The hard part is getting it out of you. You can tell a story about why you do things. She is not sure that is the mechanism.

Humans failing to describe their own reasoning does not mean computers should not reason. Those are orthogonal. The live question is what kinds of reasoning to run, and her answer is: all different kinds, depending on the problem. She does not even like the word *symbolic*; she does not know what it means technically. She believes in abstractions. You cannot reason completely fine-grained about everything in your life. You cannot make a plan at the level of images and ticks for getting a Ph.D.

To reason about a doctorate, or even about buying ingredients for dinner, you have to shrink the state space and shorten the horizon. Spatial abstraction, temporal abstraction, maybe decomposition of goals. You talk about a room of the house instead of your pose. You talk about doing something in the afternoon instead of at 2:54. You do that because it makes the reasoning problem smaller, and because you do not have enough information to reason in high fidelity about your elbow at 2:35 anyway.

When you are actually reaching for something, you do have to reason about the elbow, perhaps in continuous joint space. Two representations, switched as needed. The dogma is the thing to drop. It should not be "I am for symbolic reasoning and you are for neural networks." Computer science will tell us the right answers if we are smart enough to find them. Automated construction of such taxonomies? Eventually, she thinks, when machine-learning engineers get better at building algorithms that build useful abstractions.

Fridman asks how much of breakfast is an MDP. Kaelbling answers with *stance*. A stance is a position a researcher takes toward a problem: take this messy world, treat it as if it were a formal problem of a certain kind, then apply the matching algorithms. The world is not an MDP, or a planning problem; she does not know what it is. Modeling some aspect a certain way licenses a set of algorithms. An MDP says you fully know the current state — meaning you have all the information that would help you predict the future, and extra history would not help — and that actions change the world probabilistically. Useful for some systems. A bad model for most problems, because in most problems you do not know the state. They are partially observed.

A POMDP still postulates that a state exists, such that if you knew it you could predict well. You do not know it. You get local, noisy observations — images, sounds, touch — and you have to reason from the history of actions and observations about what is going on, then act under that uncertainty. People say they do not use POMDPs because they are intractable. She finds that a funny thing to say. The problem you have to solve is the problem you have to solve. If it is intractable, that is what makes us AI people. Problem-solving is wildly intractable; we will never solve it ultimately. So you approximate in the model and in the solver until something runs in reasonable time.

She wishes the field had better *approximate solution concepts* for huge problems. Right now there is a lot of empirical hacking, training this, reporting numbers, and it is hard to say whether it is good. Theory once chased optimal solutions, then regret bounds and anytime approximations. She likes to say she is interested in doing a very bad job of very big problems — and still wishes she could drop an algorithm on the table and know what she is going to get before running it. Engineering is running ahead of science, the way people once built bridges by trying. Now we often predict a bridge. Can we do that for learning systems and robots?

Instead of controlling the world's state, think about the space of beliefs you can have — a probability distribution over ways the world could be. The control problem becomes controlling your beliefs. Actions matter for what they do to the outside *and* for what they do to your own understanding. Asking a question or looking somewhere may not change the world and still be the right move. Chess has little state uncertainty. A car with lights on all the time gathers information whether you think about it or not. A human driver looking over a shoulder trades not seeing ahead for seeing the lane behind. You cannot make that trade without reasoning in belief space. If you know where you stand relative to the doorjamb, go through; if you have no idea where the door is, maybe do not. Uncertainty is part of the thing you optimize, not only noise to crush.

Long-horizon chores — leaving the house, making breakfast, a robot staring at a sink — need hierarchical planning, temporal and spatial. You divide a long execution into abstract segments: get out of the house, get in the car, drive. If you can build state abstractions and temporal abstractions, you plan at a high level about dependencies without the complete details. In her group's work, a high-level plan still needs a reason to think it is feasible without working it out. That step is the interesting one.

She always talks about walking through an airport. You can plan to fly to New York and later be in an office building. You cannot tell anyone in advance your plan for walking through the airport, partly from laziness, partly because you lack the information: which gate, who is in front of you. There is no point in planning in detail. You make a leap of faith that you can figure it out once you get there. Over a lifetime you have learned to predict how hard some kinds of goals are. You would never buy the ticket without some model of the intermediate steps.

One problem she is thinking about now is aggressive generalization: give an estimate of how long it takes to walk through Kuala Lumpur Airport. It should not be crazy. You need that estimate to make plans that involve the airport even if you never need the details. Abstract models, how we acquire them, then hierarchical reasoning on top. Goal regression — working backward from the goal, Herb Simon's means-ends analysis — matches an intuition that forward branching is overwhelming and backward branching is smaller. In the planning community's usual problem classes and methods, going backward has not been shown to be better. It is still her intuition. She cannot prove it.

Fridman asks how hard it would be to formulate human life as a planning problem — lunch, fuzzy goals, maybe leaving the lab. That would be a mistake, she says. It is not all a planning problem. You have to put together pieces with different styles of reasoning, representation, and learning. Brains are not all one thing; they have parts and substructure. There is no good reason to expect one true algorithmic thing to do the whole job. Just a bunch of pieces designed for styles of problems.

Model-based versus model-free is the same point in costume. You can learn a policy, a value function, or a transition model. Draw a box around a learned model plus a planner and you have a policy again, stored a different way. She reads it as a time–space tradeoff: an explicit policy may take more space and spit out actions fast; a compact dynamics model plus a planner computes the action more slowly. There is no argument to be had, only which form of computation fits which subproblem. Algebraic manipulation wants a different representation than riding a unicycle. Falling in love? No idea how to model that. First solve algebra and object manipulation.

Perception or planning — which is harder? "Perception," immediately. The huge question is representation. Perception has made great strides: classify images, play certain games, predict a steering angle. We do not have a very good idea of what perception should deliver. If you believe in no modularity, you train a giant neural network end to end, and it is hard to argue except on sample complexity. End-to-end on that giant net takes a lot of data and a lot of broken robots.

Machine-learning theory says the only way to cut sample complexity is to cut down the hypothesis space, and you do that by building in bias. Nature built bias into humans. Convolution is a bias, a very strong and critical one, hugely helpful for a kind of spatial reasoning close to images. Her view is that we should look for more things like convolution that address other aspects of reasoning: some amount of recursion, notions of abstraction, the notion that objects exist. A lot of models will not give you objects to start with. Graph convolutions and relational representations are tries in that direction.

She has wandered from perception, she admits, but the next step for perception is understanding better what it should produce and what we will do with the output. It is fine when the output is steer. It is less clear when you are trying to make one integrated intelligent agent. What is the output of perception? We have no idea, and we do not know how it should hook up. The pressing question is what structure to build in — the moral equivalent of convolution — so that learning can proceed efficiently on a real superstructure.

What does it take to build a robot with human-level intelligence? If we knew, she says, we would do it. Self-awareness, consciousness, fear of mortality? She does not think much about consciousness. Even most philosophers who care will grant you zombie robots that behave like humans and are not conscious. At this moment she would be happy enough with that. Technically, some self-awareness is necessary: some part of the system that observes other parts and tells whether they are working. A piece of code counting how many times a procedure runs is a kind of self-awareness. There is a big spectrum. You have to have some of it. You are quite far, on many dimensions, from the human end.

The most compelling research direction for her is what to build in versus what to learn. We are missing a bunch of ideas. Do not ask how many years until human-level robots; she will not participate in that conversation, because she does not know how long finding the ideas will take. Turing tests, benchmarks, DARPA challenges? She resists. Time spent arguing about those would be better spent making the robots work better. Contests motivate some people. They also produce a cycle of motivated hacking; much of it is hacks, and sometimes a cool idea emerges to chew on later. Not her thing. She does not regret that other people do it.

Kaelbling started the *Journal of Machine Learning Research* and served as editor in chief. There was a journal called *Machine Learning*, owned by Kluwer. She was on the editorial board. Annual meetings were spent complaining that library subscriptions were too expensive and authors could not publish easily. Kluwer would sympathize and do nothing. So they decided to make a new journal, following the *Journal of AI Research*, which had been running on the same model for about five years. About 75 percent of the *Machine Learning* editorial board resigned and founded JMLR.

It is completely open access: no page charges, no access restrictions. A postdoc, George, wanted to call it "Free for All." Some people were angry it existed and thought it was a fraud, impossible to run. For a long time she did not even have a bank account. She paid the lawyer to incorporate and for an IP address; it cost a couple of hundred dollars a year. Computer scientists, she says, are competent and autonomous in a way many other fields are not. They already typeset their papers. They have students who can hack a website together in an afternoon. Infrastructure was not the problem. Prestige, it turned out, did not require a paywall.

She had wanted JMLR to be something completely different and did not, because the field needed a traditional journal of record. Publication is barely a sensible word once you can put everything on arXiv tomorrow. Making stuff public has no barrier. We still need curation and evaluation; she does not have time to read all of arXiv. Upvotes are not enough, because when people fight on Twitter and Reddit you cannot tell whether they know what they are talking about. Then you have a second-order problem: whose opinions to value. If she had infinite time — she does not; she wants to make robots work — she would gather people whose judgment she trusts, find papers rather than take submissions, and publish the reviews. "Leslie's friends gave my paper five stars" could sit on a CV the way a journal acceptance does.

Movies separate professional critics from audience scores; that structure is worth stealing. A good review is a big investment, and the flood of papers is out of control — more machine-learning papers in a year than movies. Then the old-person speech, which is solid: research horizons have gotten very short. Students want to publish a lot, get patted on the head, some of which is fine. She worries we are driving out people who would spend two years thinking about something. In her day you did not publish during the thesis. You picked a hard problem, chewed, wasted time, and wrote papers when it was roughly done. Not everyone has to work that way. Some problems are hard enough that a longer horizon matters, and the current structure does not incentivize it at all.

She has been through winters. Cycles are inevitable; each time the high-water mark is higher, like climbing a landscape with a noisy optimizer. Deep learning made deep, important improvements. People are overselling, and eventually investors will look around and say the grand claims were not delivered. She expects a crash of some amount, then it continues. She cannot imagine a monotonic climb from here to human-level AI.

On existential risk and jobs she starts with a conversation with military ethicists about autonomous weapons: smart, well-educated men who did not know much about AI. Their first question was whether her robot had ever done something she did not expect. She burst out laughing. Anyone who has touched a robot knows they do unexpected things daily. Their model of programming was Lego Mindstorms: go a meter, turn left, take a picture. If that were programming, surprise would be weird. In fact we operate one or many levels of abstraction above that. We give a hypothesis class — a space of plans or classifiers — and an objective, then run an optimizer. We do not know what solution will come out. Teaching non-experts that we do not write the behavior is her new educational mission.

You do not have to be freaked out by a robot apocalypse to take value alignment seriously. Everyone who has done optimization knows you must be careful what you wish for: sometimes you get the optimum and discover the objective was wrong. Pragmatically, the field will go from engineering algorithms to engineering objective functions, and that will change the methodology. When you design an objective you wear two hats: what do I want, and what do I know my optimizer can do to that wish. Jobs matter; she does not understand sociology or economics well enough and hopes other people think about it.

The most exciting short-term question is the middle ground for engineering robots that work in the real world. One extreme is introspection and writing the program; that has not worked out well. The other is a giant bunch of neural goo, trained up; she does not think that will work either. What is left is not theological. It is an engineering question: the best combination of learning and not-learning, and what to build in. Favorite science-fiction robot? She does not have one. She loves the process. "I do research because it's fun, not because I care about what we produce."