Maybe people are rational under different assumptions

Lex Fridman introduces Anca Dragan as a Berkeley professor who builds human-robot interaction algorithms that look past a robot working in isolation. She also consults at Waymo, he notes, but in this conversation she is wearing her Berkeley hat. After a long advertising read he asks when she first fell in love with robotics. The answer is gradual, and mostly accidental.

Programming came first, then math, then computer science. In college she found AI and applied to the Robotics Institute at Carnegie Mellon, coming from a small school in Germany that almost no one had heard of. MIT said no. Berkeley said no. Stanford said no. Carnegie Mellon was the only offer, helped by an exchange semester and letters from people already there. She went because robotics looked like a place to put the optimization she already loved into the physical world.

The car story is sharper. In 2014, still a student at Berkeley, she got a ride in a Google self-driving car. Decision after decision, the vehicle made the right call. Manipulation research had taught her how hard it is for a robot to do anything at all; here the thing just worked. She is frank about the anthropomorphism. Nothing in the hardware was magic, she says, but the car came alive in her head. It had a face, a little arm, and it looked at her. Robots, she realized, could be more than tools for moving objects. They could be things that have a human connection. Google handed out a sticker that said she had ridden in a self-driving car, with the company's cute firefly logo. It stayed on her laptop for years.

The appetite for math started earlier. In Romania, she says, the Math Olympiad is close to a national sport, and she was in it from fifth grade. The theory was fun and had no destination. That is how computer science won: it was math she could do in the world. Her first program, in QBasic in fourth grade, drew a circle. She no longer remembers how. The class was an extracurricular you could pick instead of dance or music. Her mother asked why she had chosen programming.

Asked for a favorite robot, real or fictional, Dragan names WALL-E. The reason is specific: the character is astonishingly expressive. Two camera lenses for eyes, a pair of arms, and a sense of timing are enough to play curiosity, hesitation, loneliness, and affection. She used to work more directly on expressive motion, and she still thinks about what a head and a manipulator can say without speech. Pixar, she adds, is part of why the example landed so hard.

Then she tells a private story she sometimes uses in class. Her husband proposed by building a WALL-E. He actuated it: seven degrees of freedom, including the lenses. The robot came in, opened a belly compartment, and revealed a Lego box that opened slowly. They still have the machine. The story is not a detour. It is the research question in domestic form: motion can carry meaning, and a person can be moved by it.

Hand-crafted expressivity, she says, is the easy version of the problem. Hire an animator, design a few trajectories, and a robot can be charming in a narrow setting. The hard version is doing it generally and autonomously, so that a robot picking something up can look confident, hesitant, pleased, or disappointed depending on the task. When robots move they communicate internal state, or what observers take to be internal state. That will matter in homes. It already matters with children who are rude to Alexa because the speaker never gets angry and never changes its tone.

To treat a person as part of the robot's world, Dragan says, you have to enlarge the state. It is no longer enough to track configuration, actions, and a reward. You also need the observer's internal state: what the person perceives, and what they think the robot is doing. Then you optimize in that larger system, which means writing mathematics for how motion changes someone's model of you. That is hard. Lex asks how complicated a human has to be in the model. Mood, aggressiveness, game-theoretic type, or just an object with a heading?

She narrows the field before she answers. She is not talking about making small talk, picking a face, or the finding that a dollar-store robot and a polished one change how people behave. Her slice is narrower. Take a task a robot can already do alone in a lab. Now ask what it means to do that task *for a person*, in the world. Two things change.

First, the robot is no longer the only agent. Cars share the road. An office robot shares corridors. Send it to the cafeteria for coffee and other people will reach into the same space. Even if the robot knows its own goal, its actions have to mesh with theirs. Second, the programmer's objective is the wrong target. In your home you have opinions about how clean is clean, and how close the machine should come. What the robot should optimize is the end user's preferences. Figuring those out, interactively, is the hard problem underneath both coordination and service.

Understanding people, in her account, is two tasks that she treats as almost the same. One is prediction: where will this person be, and how will they drive. The other is inference: what do they want, what is their intent, what are their preferences. If you can read desire from behavior, prediction gets easier, because people are not arbitrary. They act as they do because they are trying to achieve something.

Robotics has gone a long way with inverse reinforcement learning. Forward reinforcement learning takes a reward and finds a policy. Inverse takes behavior and asks what reward that behavior is optimal for. A person can drive, and the system can recover a driving style. A person can show how they want a house cleaned, and the system can recover the tradeoffs. The model underneath is strikingly thin: human behavior is optimal with respect to whatever people want. Invert that assumption and you have a preference.

The lineage is older than robotics. Mid-century economics, via von Neumann and Morgenstern, treated choice as utility maximization. In the late 1950s, Luce and Shepard added noise: people choose somewhat stochastically, with probability that tracks utility. Robotics inherited the noisy version as Boltzmann rationality, and it has been good enough for many tasks. Then the 1970s arrived, and behavioral economics said people are not rational at all. They are messy, emotional, and full of domain-specific heuristics.

The simple models fail on tasks like Lunar Lander, which her group studied as a stand-in for assisting someone who is trying to control a complicated body. People mostly crash hard left and right. A wheelchair-mounted arm, for a patient with a motor impairment, can look just as scattered. On days when that mess feels unreadable, Dragan says she has wanted to pack her bags and switch jobs. The stakes rise as robots get more capable. A stronger machine still needs to be told the right thing to do, and the plan is still to read that thing from human behavior.

If the robot's job is to optimize what a person wants, rather than what a programmer typed, Dragan wants to treat that as a collaboration. Both agents act. The person has better access, at least implicitly, to the thing being optimized. They can talk, demonstrate, and send other signals. The robot does not have to sit still and decode a stream of observations. It can act too, and some of those actions can be designed to solicit a reply that is actually informative.

Dorsa, then a student in the group, worked the idea on the road. An autonomous car wants to change lanes in front of a human driver. It would like to know whether that driver is aggressive or defensive, and whether they will leave a gap. Passive watching is a poor sensor. Drive next to someone who is only going straight, and an aggressive driver and a defensive one look the same. The useful observation never arrives.

So the car learns to nudge toward the neighboring lane and watch. If the other driver slows, the model updates toward defensive, and the merge becomes reasonable. If they do not yield, the model updates the other way, and the car waits. Lex calls it a dance. Dragan says only a handful of people are thinking this way, in part because many roboticists are honestly afraid of humans. People are messy. Her bet is the opposite: mess is not a reason to look away. People also react, and the reaction is information.

That still leaves the tasks where approximate rationality collapses. The usual fork is grim: give up, or accept that fifty years of behavioral economics already called people irrational. Dragan has been trying a third move. Maybe people are relatively rational, just under different assumptions. When roboticists say someone is rational, they quietly assume the person shares the robot's state, transition function, and horizon. If the person does not share those, the behavior looks hectic. The failure may be in the borrowed world model, not in the person.

Lex treats it as a life lesson as much as a robotics paper. If someone looks inexplicable, the first hypothesis should be missing empathy, not a broken mind. On Lunar Lander the robot knows the physics and the person does not. People seem to run a simplified intuitive-physics model of the craft, the sort of thing Josh Tenenbaum and Tom Griffiths study in cognitive science. Dragan's group tries to infer which simplified model best explains the commands. Then, instead of executing the stick input, the assistant asks where the person *thought* the lander would go, and uses the real physics in reverse to make that intended outcome happen.

People landed the thing between the two flags. She will not call the result conclusive. She will call it evidence that giving up and labeling the operator crazy is a form of underestimation. The same spirit, she says, can break the Bellman update in other places. Maybe the person never sees the true state. Maybe they are still estimating the transition function. Maybe they are still learning the reward itself. A recommender that treats a Netflix click as a declared preference is reading a choice made while the person is still figuring out what they like.

The research program she wants is domain-agnostic: foundations of algorithmic human-robot interaction, whether the body is a car, a quadrotor, or an arm. Application work always needs extra adaptation. The shared question is how algorithms change when the robot is no longer emptying a dishwasher alone. Open-domain dialogue, she agrees, is human-robot interaction that happens to be virtual. The boundary she cares about is the principle, not the chassis.

Lex turns the information-gathering idea around. People maintain a world model of the robot, and that model can be shaped. Kids already have a model of Alexa. Drivers already have a model of an autonomous car. Is there value in optimizing how people see you? Dragan says the process is inevitable. See a car cut someone off and you update toward aggressive. See a robot head toward a door and you conclude that is the goal. Humans do to robots the same inference robots are trying to do to humans.

Once that is in the state, the robot can optimize for being informative rather than confusing. In one line of work, part of the state is the person's belief over the robot's internal parameters. Each action is new evidence. The human is modeled with Bayes' rule, assuming the robot is rational, and the robot chooses actions that put as much probability mass as possible on the true parameters. The math is ordinary. The design claim is not: if people will infer anyway, the robot should pick motions that make the right inference easy.

Lex is not only interested in clarity. He is worried about contempt. Children shout at Alexa because nothing happens. He grew up with the Soviet Union and the history of how humans treat a group labeled *the other*, and he thinks robots are next. His instinct is that machines need some attitude, some mystery, some pushback, or they will not be respected. Dragan offers two thoughts. People do respond to assertiveness. They also respond to vulnerability. A robot that can look briefly bummed after being shoved may get better treatment than a robot that performs invulnerability, because people cannot help anthropomorphizing. The classic psychology demo still works: a triangle chasing a square on a blank screen is enough to make viewers angry at the triangle.

The deeper frame, she says, is a general-sum game. The car wants its passenger home; the nearby driver wants the same for themselves. They share a wish not to collide and they do not share a destination. Treating interaction as prediction-plus-avoidance misses the influence. Highway merges stall when a vehicle forecasts that traffic will keep coming, finds no collision-free plan, and waits forever. If the car eases forward, a human may slow and let it in. She likes the older robotics word for this: underactuation. Some degrees of freedom you control. Some you only influence. Human-robot interaction, she says, is an underactuated robotics problem, and there is poetry in that.

Lex has a strange hobby: hundreds of hours of pedestrian video. Crossing a street in America, tens of millions of times a day, is a miniature game of chicken, especially where right of way is ambiguous or the intersection has a personality that changes with the hour. A runner once told him not to make eye contact on the sidewalk; everyone will move. Dragan has a name for it: civil inattention. Communicate that you will not deviate, and you tell others to plan around you. Hesitate, glance, and linger, and you signal that you are still in the game and can be influenced. The robot's nudge is the same game with an accounting step: is this probe worth the information it might buy?

Jim Keller, the chip architect who also led an Autopilot team, has told Lex that driving is still a ballistics problem: do not hit things, learn the dynamics, ignore the human. Lex's intuition is the opposite. Humans make the problem an order of magnitude harder. Dragan answers with a thought experiment. Take downtown San Francisco and remove every person: no pedestrians, no human drivers, no cyclists, no scooters. At that point, she thinks, driving is done. Perception used to be the blocker. A few years earlier it was not safe to say the stack knew what was where. Now, with people gone, she does not see perception as the thing standing in the way.

Planning, control, wet roads, and construction zones are still difficult. Highway speed is what feels dangerous to her, not a slow crawl through a familiar downtown. On sensors she is blunt in a personal register: she thinks it is irresponsible not to use lidar when you have the chance. Elon Musk's line that lidar is a crutch is, she admits, a fun idea. Things that work today do become crutches that stop researchers from looking outside the box. Safety-critical systems invert the ethic. Until vision-only is known to hold, her information comes from stacks that can use lidar, and she would keep using it.

The human piece is not a mystery with no mathematics. Companies are already deploying driverless cars in small areas. Lex describes a Waymo ride in Phoenix as one of the seminal robotics moments of his life: no one in the driver's seat, no option to grab the wheel, speed-limit driving that negotiated people, intersections, and left turns. Dragan will not reduce the remaining work to pure engineering. Success in one place, and absence everywhere else, suggests the method can scale and also that new cities will demand new algorithms, or at least modified ones.

How much of this, Lex asks, can be learned from data, and how much is still heuristics injected by human experts? Dragan will not accept the usual binary. Framing the stack as handwritten rules versus end-to-end reinforcement learning or imitation leaves out planning, search, and optimization. Robotics is sequential decision-making. When a robot searches a space for a path that reaches a goal without hitting things, no one wrote those waypoints as rules. That search is, in her view, indispensable.

Learning is inevitable too, because search needs models of the world. Imitation alone dies off the training distribution. Reinforcement learning is more promising, but trial and error in the real world is a harsh teacher, and trial and error in simulation only helps if the simulated world, including the people in it, is good enough that the resulting policy is safe to roll out and keep learning. Human models for that purpose, she says, have to be a mix of constructed structure and learned pieces. Fit a policy to a batch of human data, compute the robot's best response, and the robot will leave the support of the data. Outside that support the fitted person is worthless. Many policies are consistent with a finite dataset. The data do not tell you what the person does next.

Lex dislikes the phrase *out of distribution* because it smuggles in a tidy statistical model of everything already seen. People walk into novel situations and still make mostly decent forecasts. Dragan's reply is about priors. You have to regularize the huge space of mappings from history to action. History may mean the last few minutes or the last few months; the right horizon depends on the task. A candidate inductive bias is that people have intentions and that those intentions drive what they do, at least until nearby data says otherwise. Whether that prior fails, and whether you need a richer belief structure, she calls a completely open question. Common-sense reasoning, in the grand AI sense, may not have to be solved first.

Lex wants an explicit fear of death in the model. Pedestrians playing chicken are playing with mortality. Dragan thinks the utilitarian frame already contains it. Treat an agent as having objectives, and you have given it a reason to stay alive: a dead agent cannot finish the job. She quotes Stuart Russell: you cannot fetch the coffee if you are dead. That is why she stays inside a rationality framework she knows is broken. The alternatives she sees are packing up, or going fully black-box and having no idea what to assume off the data. She would rather keep the useful fiction and stay ready to notice when it fails.

The conversation turns from full autonomy to the Level 2 systems already for sale, including Tesla Autopilot and Cadillac Super Cruise, which watches the driver's face and allows hands-off driving only while the eyes stay on the road. Lex asks whether a dumb robot plus a supervising human is a sound way to keep both alive. Dragan's caution is about a hidden assumption: that a person who is no longer driving, only intervening, will be as safe as the same person with their hands on the task.

Technically, a system that takes over will visit states the person would not have produced. Those are off-policy from the driver's point of view, and it is not obvious that people perform as well there as in the states they know. On the human-factors side, she experiences active control and passive observation as different even when she tries to stay engaged. Students who generate an answer learn more than students who are told one. Aviation has treated the gap as established for decades.

Lex has taken heat for the opposite observation. Decades of human-factors work predicted disengagement; large-scale data and a lot of conversations have surprised him. On sufficiently dumb semi-autonomous systems, he says, people can be more energized as observers than they are as drivers. Dragan does not shut the door. Naive designs implicitly bet on an assumption that may be false. Designs that deliberately keep the person in the loop, and treat the goal as empowering a still-present driver, are a different philosophy. Wanting someone to barely drive and stay ready to grab the wheel is not the same as wanting the pair to outperform the person alone.

Then Lex asks the question that has been sitting under the whole hour: how do we design good reward functions? Dragan's answer is that we do not. She used to think the hard part was customizing a reward to an end user. She now thinks specifying a reward is hard even after you strip the interactive piece away. Write an objective, deploy the agent, and you want the induced behavior to be the behavior you intended in every new situation. For an arm, that means a cost that balances clearance from people against efficiency, and still looks right in scenes she has not tuned. She has no idea how to guarantee that. In practice she invents a few representative situations and tunes until the optimum looks right on those. The reinforcement-learning revolution promised that we would stop writing behavior by hand. The work moved onto the reward. She has been tuning cost functions for ten years, and for a long time she looked away from how much of the old craft was still there.

Lex lifts the failure mode out of robotics. You respected the specification and still did not get what you wanted. That is a human problem, and it has a name: Goodhart's law. The moment a metric becomes the target, it stops being a good metric. Dragan takes the mapping. A robot will optimize what you wrote. That may not be what you meant. The collaboration she first studied between a robot and an end user applies just as well to the robot and the programmer who tried to write the task down. The programmer is not a perfect oracle. The written reward is not the letter of the law.

Once you stop treating the spec as sacred, other signals come into view. Imagine a home robot with a usable initial objective from its programmer. It comes too close. You push it away. Classical control can treat the force as a disturbance to reject. A more compliant mode can go limp and follow the push. Her third reading is that the push is about the reward. If the robot was moving optimally and you intervened, you disagree with that notion of optimality. The cost is wrong. Added to the robot's original command, the external force is a clue to the action you actually wanted, and the system can learn that you prefer more distance. One shove will not settle a million alternative explanations. A longer acquaintance might. The robot can also move on purpose to tell those explanations apart, the same information-gathering idea, now in physical contact.

An emergency stop is another leak. The robot should halt. It should also update: whatever it was about to do was worse than sitting still. The environment leaks too, without a demonstration. A robot that walks into a house and sees shoes laid in a line can infer that the line matters, because shoes do not arrange themselves. She is careful with the inverse. A messy house does not mean the owner wants mess. It means they spent their limited bandwidth on something else, and that something else ranked higher. The current state of the world is a sediment of choices. Read it too crudely and you learn the wrong preference.

Lex asks about Asimov's three laws, the folk ethics his relatives reach for when they imagine robots. Dragan does not want three laws. She does not even keep the wording in her head, and the deeper problem is that they do not translate into mathematics. What does it mean, as an objective, not to harm someone? You want machines to do what you meant, not what you said literally. In work with people in her group and collaborators including Pieter Abbeel and Stuart Russell, a designer-specified reward is treated as evidence about what the person wants in the situations they actually had in mind, not as a universal function to optimize forever. The robot should remain uncertain about its true reward, and keep updating from corrections, demonstrations, stops, and the room itself.

Asked for a book that turned her life, she goes back to twelfth grade in Romania and a PDF of Russell and Norvig's *Artificial Intelligence: A Modern Approach*. She knew nothing about AI except *The Matrix*. The idea that captured her was simple and, to her, beautiful: you can have a goal, and from a messy situation you can find a sequence of decisions that reaches it autonomously. That is still the lab's signature. They write mathematics about human behavior, combine it with data and learning, and hand the result to a robot to plan with, instead of writing rules for how to move around people. The thrill is the same one she felt watching a search algorithm find a path on a graph.

If a doctor gave her five years, she says, that is her nightmare, because she likes living and does not like being told she will die. She also does not want to live forever. *The Good Place* left her with the moral that even an afterlife has to be finite or people lose the plot. She is not religious; she thinks death is the end of existing, and existing is a privilege. The sadness she cannot shake is that we love being here partly because it ends. She finds almost everything beautiful, including the silliest mundane things, and she knows the finiteness is part of why. Lex wants that fact in our reward functions. She laughs that people accuse her of Russian melancholy, then declines to redesign her life. She would take more trips to the Caribbean, do less departmental service, and stay off admissions. The work, the students, the colleagues, and the family already spark joy. For the most part she already refuses work that does not.

The teacher who opened the door is a smaller story and, she makes clear, a larger debt. While classmates paid for tutoring ahead of the baccalaureate, she had neglected physics. She started working with a teacher she names Bîcă, who told her to sit contests, apply to college abroad, and improve her English. Her parents could not afford the extras. The teacher tutored her in physics for free and sat with her to train for olympiads. Lex notes the obvious: a person with no institutional reason to do that is part of why she is at Berkeley. She remembers it as one of those small, unaccountable kindnesses, someone who simply wanted a student to go further.

The meaning of life is a question she cannot ask with a straight face, and she answers it anyway. In a planetarium the camera zoomed out until people were dust, then the presenter reached multiverse theory and the entire unfathomable universe became a blip. At that point she was done: there is no cosmic meaning. What remains is to affect the local piece you can reach, your community, friends, and family, and to be there for other humans. The immensity does not inspire her so much as frustrate her. Her brain, she feels, lacks the capacity, and even the physicists understand only up to a line. Lex closes on the AI dream of expanding that capacity, then leaves the listener with Asimov: your assumptions are your windows on the world; scrub them off every once in a while, or the light will not come in. The conversation has been that kind of cleaning, starting with the assumption that people are optimal, and ending with the possibility that they are only living in a different model than the one we wrote down.