Michael Littman isn’t waiting for a runaway god. He’s watching us.

Lex Fridman introduced Michael Littman as a Brown University computer scientist who works on teaching, machine learning, reinforcement learning, and AI, and who likes to keep a conversation silly. After a long warmup — *Robot and Frank*, Billboard pop, a TurboTax commercial, parody songs with Charles Isbell — Isbell’s reported claim that the super-nice professor actually holds some cold opinions finally landed. Littman said he tries to live in a Calvin-and-Hobbes cubist world, seeing every side at once. One view he would be hard to talk out of is the superintelligence story.

He put it without drama. He is willing to hear other arguments. He is “not particularly moved by the idea that if we’re not careful, we will accidentally create a superintelligence that will destroy human life.” Fridman, playing prosecutor, offered the usual cautionary Microsoft quotes about the internet and smartphones. Littman did not flinch. Nobody knows the future; you use the best intuitions you have. His is that we are quite far from that being the serious concern, relative to the other ones we already have.

He has done the reading. The story, as he understands it, is that we will build computers smart enough to design the next version of themselves, bootstrap past us, and then sit at the mercy of a more powerful intellect whose goals we do not control. If those goals omit the continued existence of humanity, we are toast. Very smart people have signed on. It is a compelling story. He once wrote an op-ed answering Elon Musk, who had come to Providence and told the nation’s governors they were worried about the wrong thing.

Littman’s read of Musk is a character study, not a technical rebuttal. What makes Musk remarkable, he said, is that he believes in the power of ideas. A good idea for getting into space gets you into space. A good idea for how people drive can change how people drive. You just have to do it. Apply that same faith to the remarkable computational tricks on display in AI, push the idea all the way to the limit, and of course you end up with something that could kill us. “His strength is also his undoing,” Littman said. That does not make the conclusion true. It makes it natural for Musk to think it.

Fridman found the zoomed-out version just as hard to argue with. Sam Harris and Nick Bostrom ask whether anything in the physics of the universe prevents this from eventually happening. Humans live in minutes and days; history lives in centuries. If the laws of physics do not forbid it, then either it arrives or we destroy ourselves some other way. Littman agreed that the argument is difficult to punch through without sounding as if you are rolling your eyes at science fiction, or ducking out to pick up the kids. Especially in a COVID year, shorter crises always feel more real.

He even granted Harris this much: you do not have to put a date on it. A thousand years, a hundred, two. As long as this kind of progress continues, it ultimately has to become a concern. “I kind of am on board with that.” What he thinks the extrapolation is missing is the process. We do not now know how to build technology that really gets around in the world. In the course of learning that, we will learn a lot about what it means — knowledge we do not have yet because we cannot do it yet.

The scary picture is a deep-learning network switched on with enough compute, climbing the wall of complexity with nobody in the loop. That picture, Littman said, feels like it is against how these systems actually work. They need help. They need to be pushed over the difficulty of arranging the world into a form that will function. If you believe you can just turn the network on and wait, the fear is rational. He does not believe that.

He reached for the anti-evolution slogan that a 747 cannot be assembled from random parts. Fine: 747s do get assembled. They get assembled by us. Superintelligence, if it arrives, will be that kind of process, not something that springs into existence fully formed while we stand around unprepared. Along the way we will have more ability to control it, shape it, or build it in our own image. On the long arc of human history the “springing” might still look sudden — twenty years is a blink, as nuclear weapons showed. It will not, in his view, be a surprise with no one in the room.

Fridman put a number on the nuclear shadow. Think through the crisis moments of the twentieth century and you could put the chance that humans knocked themselves out with nuclear weapons at 30 or 40 percent. We walked close to the edge more than once. Survival was not a clean calculation. Littman’s larger fear, Fridman added, is not that AI “runs away” so much as that while it is running we will not have enough time to think. Takeoff, FOOM. Littman did not deny the historical luck. He still thought the nearer algorithm problem was already here, and dressed as something other than a movie villain.

Fridman argued that an AGI situation is already happening with social media: our minds, the collective intelligence of civilization, are already being controlled by an algorithm. Wikipedia, he joked, is the AGI you should donate to. Littman trusted Wikipedia’s values more than Facebook’s or YouTube’s, and noted that there are smarter and less smart ways of exposing ourselves to each other online. The vision of AGI as a separate superintelligent system is one thing. The paperclip is another: a super-dumb, super-powerful optimizer. Social media is closer to the second. Very simple algorithms, at the scale of tiny interactions with billions of people, can start to steer how those people think.

Fridman wanted to believe the chaos is the same argumentative mess that ran from the founding through the Civil War — tension over core values, people being awful on Twitter, the algorithm managing it toward something better. Littman believed in social media’s power to screw us up royally and also to benefit us. It got dropped on top of the culture, and now the culture is trying to cope. College students who were in middle school when the platforms really took off may have been damaged in ways we do not yet understand. They are also, he said, the generation that can fix it. If they keep the good of the technology without letting it eat them alive, we level up. If they fail, we wreck each other.

His daughter lives in that weather. She is tapped into social stuff and is always hunting for the balance — sometimes she ventures too far and is consumed, sometimes she gets distance. Enough people like her, and the choppy water is navigable. Fridman had learned a related skill the hard way: a mini-army of trolls, despite talking endlessly about love and kindness, and the discipline of not looking too much. His father, who runs a large institute at Drexel, never learned how to shut off email and walk away from the drama waiting in the inbox. Young people are being bullied in ways you cannot escape, and they are also being forced to learn disconnection. Littman has become a believer in human design: flawed, somehow still able to level up when the next stage of the game appears, in a way his reinforcement-learning agents never do. Train an agent through the first stage of a video game and the second stage sends it back to the drawing board. Humanity keeps assembling the skills for the next level. He hopes we can keep doing that.

Asked for a history of the field from a front-row seat, Littman went back to college in the early 1980s, when neural nets were stirring and he was taking piles of psychology and computer-science classes. Teaching a home computer to play tic-tac-toe in BASIC ate his vacations. The love affair had started earlier. He saw machines in a mall RadioShack, tried to stump them with parenthesized arithmetic, and they never missed. Algebra had taken him years; the box seemed to know it natively. Instead of a bar mitzvah party, his parents bought him a computer, around 1979. Ages 13 to 15 are a blank: a room, a machine, one Billy Joel album on vinyl, later a cassette so he did not have to use his parents’ hi-fi. At 15 he walked out ready to talk to people again.

Princeton had computer engineering, not computer science, and he did not want to touch hardware, so he went to Yale. He remains a cognitive-psychology groupie. Brains are wacky; cognitive scientists ask bigger questions about intelligence and perception. In a seminar with a favorite professor, Richard Gerrig, the class did a close reading of a thick Steven Pinker and Alan Prince paper — that generation’s Gary Marcus and Ernest Davis, he said, the takedown that grants neural nets their tricks and then lists everything they cannot do, insisting the circle of impressive results is drawn too small. The paper brushed reinforcement learning. Behavior was what interested him in psychology; programs behave; he wanted programs that learn to behave. Neural nets then were supervised maps from inputs to outputs. He tried, in a sense, to invent RL himself.

After graduation he joined Bellcore, spun out of Bell Labs after the 1984 breakup, in a group with Dave Ackley, first author of the Boltzmann-machine paper that showed neural nets could learn XOR and helped start the second wave after the perceptron winter. Ackley became his mentor and handed him Rich Sutton’s 1984 TD paper. Littman did not get all of it. He got the idea: backpropagation-like machinery, pointed at predictions over time. When he said he wanted the details, Ackley offered to invite Sutton down to give a talk. In Littman’s head the authors had been words, ideas that magically seeped into papers. “No, I know Rich. We’ll just have him come down.”

Sutton arrived excited about Q-learning. Chris Watkins had visited the Sutton–Barto lab at UMass, and the new algorithm resolved problems the earlier TD paper could not. Temporal-difference methods predict over time; you can try to choose actions from those predictions, but the theory wanted a consistent process. Q-learning was off-policy: you could learn the value of actions in the environment while separately figuring out how to behave optimally. That was a revelation. A single equation, one line of Lisp if you squinted, looked like a sketch of all of intelligence. It is not quite true, and it needs a lot of supporting machinery. As a way to create behavior that gets better over time, he still thinks it is a very reasonable idea. Worst-case proofs matter to him more than they do to many practitioners, because they buy modularity: plug the piece into another piece and still know what will happen. Meet Sutton, he joked, and everything after was downhill.

Through the 1990s Littman was inside enough to hear results from the people doing them, or from the people talking to those people. Jerry Tesauro’s TD-Gammon was the one that felt remarkable. Tesauro had first trained a more classical backprop network on expert backgammon moves, not to mimic actions but to learn an evaluation function: if the expert chose A over B, adjust the weights until the net also values A more. That is learning preferences from human preferences. Then he took the trainer away and let the program play itself. The field did what the field always does. If you can solve that hard problem, surely you can solve these other hard problems. Almost none of them became practical. Neural nets of that era were inconsistent. Training them in an RL setting was a mess.

Generation after generation of Littman’s master’s students wanted to do value-function approximation with neural nets. Over and over they failed to get Tesauro’s results. “I now believe that Jerry is a neural-net whisperer.” He has a particular ability to get networks to do things other people find impossible. It is not the technology. It is the technology and Jerry together. People are drawn to the idea that power comes from the artifact and lose the human. Nobody looks at GNU Emacs and concludes that computers are about to take over the world. They conclude that Stallman is a hell of a hacker. He could not have done it without the computer; the computer could not have done it without him. Tesauro, in that analogy, is the discounted expert.

Self-play as a term still felt slightly ridiculous to Fridman, like “Big Bang” in its early days, a silly name for a concept that might be among the most important in AI. Littman had used it in his 1996 dissertation; Tesauro’s paper was already talking about training a backgammon expert through self-play, and “rollout” is another backgammon word that migrated into AI. After TD-Gammon, not much more of that architecture worked for a long time. An elevator-bank controller looked promising in simulation. Cell-tower handoffs looked promising. None of it, that he knew of, made it into production. Neural nets as a whole started to implode, and there was not much air in the room for RL. They found their way back more than a decade later.

When Fridman was learning AI, Go was treated as the limit test. Beat a world champion there and you have brushed AGI; the chess-killing techniques could not beat a typical Go-playing teenager. AlphaGo felt magical. AlphaGo Zero, then AlphaZero, felt more so: no supervised expert games, only self-play. Littman’s ranking is the reverse of some of his peers. “AlphaGo knocked my socks off.” They integrated a pile of ideas into one giant system. The software engineering alone was mind-blowing. David Silver, like Tesauro, is a neural-net whisperer, coaxing new architectures into problems nobody knew how to solve. In a very short number of years the field went from losing to teenagers to trouncing champions. That blew him away.

Fridman had taken the engineering for granted and wanted the philosophy: persisting through long odds, Silver’s unbreakable hope mixed with enough skepticism to hunt the bugs. Littman granted compute, data wrangling, distributed supervised learning, the Deep Blue lesson that a well-funded company can pour millions into ideas already in the air. He also granted that Silver was not cocky. The 1990s had to happen for the 2010s to happen. What still surprised him was how little beyond TD-Gammon got done on that RL front for so long, given that his own students had tried the architecture on other problems and consistently failed.

Satinder Singh, another Barto student, was more impressed by AlphaGo Zero’s removal of the human-game bootstrap. Littman was not. AlphaGo already did a tremendous amount of self-play. Expert games kick-started the policy network; then they trained for a long time with more traditional temporal-difference methods. Once it works, it works. Why wouldn’t dropping the extra information work? He does not think AlphaZero is merely better at implementing human strategies. It developed its own, more effective ones, at least one quantum leap in strategic knowledge. The first leap — getting the networks to hold value reliably — was the doozy. Random play becoming basic play is less mysterious in Go than it sounds: the agent is told who won, so credit for being in certain parts of the board can start to percolate quickly. Basic play, then search for the next increment.

Fridman quoted David Silver’s calm, ominous line: they have not yet discovered a ceiling for AlphaZero. Throw more compute at Go or chess and it keeps improving. Continue Moore’s Law in some broad sense and you can imagine an AlphaZero that swallows the world. Littman’s first correction was narrow. It is getting better at Go. That is a big leap from “therefore it is better at other things.” How much of the game of life can be turned into that kind of game is a real question, and he does not think the community knows the answer.

Chess history is his depth gauge. For a thousand years humans were the strongest chess-playing things on the planet. Then computers passed the best person and have stayed ahead. For a stretch, computers and people together could still beat the best computer programs. Elo math says you move up a band if you beat the band below you about 62 percent of the time. Human ability caps at the best humans; computers went a step beyond; human–computer teams, last he had checked, had not gone a full step beyond that. The estimates were starting to asymptote. Chess has finite strategic depth. At some point you cannot get better.

Go’s strategic depth, from the experts he has talked to, is substantially greater — more steps of improvement. There is still no reason to think it is infinite. Finite games have a minimax-optimal strategy. You cannot beat that strategy. It may be incompressible, a giant mass of hard-coded rules we can never achieve, and that still puts a cap on how many levels of improvement you can make. What Silver is seeing could be an asymptote: keep getting better with diminishing returns until you hit optimal play. Self-play’s real gift, Fridman noted, is that past a point it does not need human labeling. The live research question is whether we can formulate problems that matter more than chess — simulation for vehicles and robots, or the internet as unlabeled text — so that self-supervised methods can work on them.

The transformer family, Littman said, is really, really cool. Anyone who once played with Markov models for generating text and saw the gibberish they spat out has to be amazed at what comes out now. It does not take long interacting with one of these systems to find the holes. It is not smart in a general way. It is very good at a bunch of things. It seems to understand the statistics of language extremely well, and that is powerful enough to answer many questions. It does not make a good conversationalist or a good storyteller. It makes a good imitator of things it has seen.

The same complaint, he noted, is what Trump voters say about Biden voters and vice versa: they are not intelligent, they are following what they have seen. He has had a similar thought about GPT-3’s humanlike stories. It does not necessarily mean computers are smarter than we realized. It partly means people are dumber than we realized, or that much of what we do day to day is not that deep. We go with the flow. We say whatever feels like the natural next thing. Not a lot of it is creative, meaningful, or intentional — enough is that we get by, talk each other into things, vote for reasonable people sometimes. The rare interesting bit is hard to see in the statistics because so much speech is rote, and the metrics we use to score the systems do not reveal it.

Scaling from GPT-2 to GPT-3 was a noticeable jump. Is there no ceiling, only more compute? Here he was willing to call himself speculating. He does not believe you can get something that really uses language without interacting with people. Everything we have written down is not enough. You need to be pushed back at. Even smart people get most of their benefit from talking to each other live; a story you write for someone to read later is a different object. Systems that only learn from our stories, not from losing arguments with us, are fundamentally limited. They have to get shut down. They have to argue and lose a couple of times before they notice that some nuance actually matters.

The limitation inside that claim is brutal. Bandwidth to humans is low and expensive. Self-play works because it can run a very large number of interactions. You cannot deploy an RL system, or a language model, into the real world to spar with people and expect enough data relative to the cost of their time. The research program he wants is therefore sample efficiency: keep the spirit of reinforcement learning and make the algorithms need fewer trials. That, not a bigger scrape of the internet, is the bottleneck he takes seriously.

Fridman summarized Rich Sutton’s essay “The Bitter Lesson”: over seventy years of AI, the algorithms that improved most in practice were the simple, dumb ones that could eat compute while researchers congratulated themselves on tricks. Wait for the hardware people to finish. Are ML academics just a subfield of computer architecture, procrastinating on the hardware they do not want to do? The argument reminded Littman of Fred Jelinek, the computational linguist who said that every time they fired a linguist, performance went up about 10 percent. Building knowledge in was less successful than throwing the scholars out and adding compute. Vision is the modern version: part-based models that feel obviously right, versus a lot of data, and the data wins.

His own analysis starts from two explosions. Compositional complexity grows explosively as you consider more possibilities. Moore’s Law has also been growing exponentially. Two exponentials side by side make it look as if you never have to think hard about algorithms: the best you can get from cleverness is exponential, the best you can get from waiting is exponential, so wait. That has to end. There are hints of friction. A counter-Moore’s Law, which he said many people do not know, is that the development cost of each successive chip generation is also doubling. Money per cycle is roughly constant. At some point you run out of money, or you need an entirely different development process.

He has always been skeptical of “it is an exponential, therefore it has no end.” Soon the number of people at NeurIPS will exceed the population of the Earth, therefore we will discover life on other planets? No. It means we are on the front half of a sigmoid, which looks like an exponential; the back half looks like diminishing returns. Fridman added that Moore’s Law itself is hundreds or thousands of S-curves stacked, constant breakthroughs, and the useful question is how team size and resources required to find the next curve are changing. Sitting in the present, exponential still feels like the reasonable thing to say. Software keeps seeming happy. It is still hard to disagree with Musk when he talks as if fully autonomous vehicles are a year away, a learning problem, cameras and data. Then you put on the computer-vision hat, the skill-training hat, the psychology hat, and driving is much harder, with a much larger number of cases, than the exponential story wants.

Littman used to be much more skeptical of neural nets. He now leaves a sliver of possibility that we will be surprised, including in social networks, where a very good transformer could learn human behavior and manipulate it for ads — which, they both agreed, the current systems already do, just not for self-preservation. If they started manipulating us to keep themselves from being shut off, he would be freaked out. He does not see a path to that from here. Corporations, though, are funny organisms with self-preservation designed in, and the broader system of algorithms-plus-companies is more complicated and more powerful. On books, he first dodged. He is not a voracious reader; Tom Landauer, who ran his Bellcore group and pioneered latent semantic analysis, once told him most vocabulary is learned from reading, and Littman did not believe it. Audiobooks later closed the gap.

The driving section is the one that most clearly shows what his kind of intelligence looks like from the passenger seat. He is teaching two children to drive because his wife does not want to be in the car. They start better than he imagines a neural net would: they get that they are seeing the world, that there is a road, that steering angle maps to the vehicle. Staying in lane at speed comes reasonably fast. What is remarkably hard, and what he thinks makes self-driving hard, is that driving is a social interaction. He had been completely unaware of that until he watched his son, who has always had social-communication challenges, send mixed signals to the cars around him. Those cars drove weirdly. Littman had no doubt there would be an accident, because nobody could read him. Speed and lane position are theory of mind, nested: what the other driver thinks you think. Pedestrians are worse, an octopus of degrees of freedom, except that humans know what they will do because we model their model of our model. He has heard little of this in self-driving talks.

He leaves room for the possibility that it is learnable and that he is being speciesist. A ride in an aggressive, fast South African autonomous car felt good; Tesla might yet change his mind. The way this technology enters the world will be messy, as democracy is messy. Years ago he predicted a gradual rollout in closed communities so humans and vehicles could coevolve. Capitalism instead plopped the cars in, rewards risk, and — looking at quality-of-life data rather than at academic DARPA-challenge nostalgia — often lets competent people rise. Academia, he and Fridman agreed, still treats making the thing work in practice as less sexy than a theory paper on a story dataset.

The three books, once he stopped dodging, were specific. Douglas Rushkoff’s *Program or Be Programmed* analogizes coding to literacy: readers receive, programmers put information out, and a world where everyone can write a script is a world where other people’s software has less sway. Rushkoff has backed away from the prediction; Littman still believes the premise. Brian Christian’s *The Alignment Problem* he was in the middle of: a strong first third on fairness and present-day social impact, built from interviews with the people whose names are on the ideas; a second third on RL; a third on superintelligent alignment that he suspected would be less fun. Stuart Russell’s book, he added, does a remarkably good job explaining how the technology actually works. The fiction pick was Ted Chiang’s *Exhalation* — Chiang studied computer science at Brown — short stories driven by a sharp bit of science pushed to an extreme conclusion.

Fridman closed with the absurd question: what is the meaning of this whole thing, life on this spinning rock. Reinforcement-learning researchers, Littman said, think about lifetimes more than supervised-learning people do, because their agents have little weird lifetimes and it is hard not to project yourself into them. When he turned 42 — the Hitchhiker’s Guide answer — he threw a meaning-of-life party. Guests made slides. They sat down and presented to each other.

His own slide was balance. The party activity was every balancing toy he owned: a board, a pogo stick, a ripstick, a unicycle he could not yet ride and later learned, badly but actually. His wife’s answer is probably more accurate: healthy relationships with people you love, and working hard for good causes. His word is still balance. Not too much of anything, because too much of anything is iffy. Fridman offered a Rolling Stones parody: you can’t always get what you want, but if you try, sometimes you can strike a balance. After two hours of superintelligence, recommenders, Q-learning, and GPT-3, that was where Littman put the risk — not nowhere, just back in proportion, on a unicycle.