Humanity Cannot Survive on the Right Person Showing Up at the Right Time

Lex Fridman opens by noting that Tegmark was episode one of the podcast, and that a lot has happened in both physics and AI since. Tegmark, an MIT physicist, AI researcher, co-founder of the Future of Life Institute, and author of *Life 3.0*, wants to take stock of 2020’s real breakthroughs — not the ones that filled headlines — and ask what they mean for science, for people, and for new failure modes. He is especially fired up about a new NSF center, the Institute for Artificial Intelligence and Fundamental Interactions, a five-year, $20 million collaboration between MIT and neighboring universities.

About five years earlier he had shifted his own MIT research from physics into machine learning. Colleagues were polite, he says, but the look on their faces asked what Max thought he was doing. That look is gone. The institute’s bet is two-way: AI can help physicists do better physics, and physics can help AI become something you would actually trust. Hardware is part of it. His colleague Marin Soljačić is working on optical chips that compute with photons instead of electrons, promising less energy and more speed.

The deeper gift, Tegmark argues, is attitude. AI has been an engineering culture that tries to make things work and is often more interested in selling them than in proving they will always work. Contrast that with a rocket to the space station. SpaceX did not train a booster by firing a bit left, then a bit right, and hoping. People used Newton’s laws and a deep understanding of the hardware. His vision is that every machine-learning system with a real effect on people’s lives will be understood at that depth, so we trust it because it earned trust, not because a sales representative told us to.

The year’s dazzling demos share a trait. Boston Dynamics robots dance. AlphaFold 2 crushes protein folding. GPT-3 writes English that sometimes blows Tegmark away. DeepMind’s MuZero beats humans at chess, shogi, Go, and Atari without even being taught the rules. All of them are powerful, and none of them are fully understood. That is acceptable, he says, if the worst case is a face-plant or a bad move. It is less acceptable if the same style of system is driving a car or running a nuclear plant.

Hollywood trained people to worry about machines turning evil. Actual harm from automation, Tegmark says, has come from over-trust in systems we did not understand as well as we thought. The Boeing 737 MAX’s simple automation killed people; the system was not evil. Knight Capital deployed a shiny trading system and lost $10 million a minute for 44 minutes until someone shut it off. Factory robots that have killed workers treated a human like a part. In every case the pattern is the same: misplaced trust, and not even the humility to notice the gap.

Some colleagues, he says, have started to treat inscrutability as the source of neural-net power, a kind of mysticism. He calls that nonsense. You send in numbers, multiply by matrices, get numbers out, and twist knobs until a definition of “good” is maximized. The tables of millions of parameters do not tell you how the thing works. The real advantage is differentiability: the output changes smoothly when you tweak a knob, so you can use scientific optimization. Change random letters in a Python program and you get a syntax error, not a gradient. Every neural-net parameter setting is still a program, and you can tell whether it got better or worse.

Tegmark’s group’s slogan is intelligible intelligence: do not settle for something that merely seems smart. Step one is still to train a mysterious network that does a job well. Step two is more AI, aimed at turning that black box into something equally capable that a person can actually read. The flagship example is AI Feynman. They took about a hundred famous equations from *The Feynman Lectures on Physics*, the book that got Tegmark into the field, filled a spreadsheet with random inputs and computed outputs, and asked whether a machine could recover the formula from the table alone.

That problem is symbolic regression. Linear formulas are easy; logs, cosines, and mixed expressions explode combinatorially, like passwords growing with length. Their trick is to train a network that approximates the unknown formula, then probe the network with data that was never in the original sheet, hunt for simplifying properties, and break the formula into pieces. The code discovered all hundred textbook equations plus others. Anyone can type `pip install ai-feynman`. Kepler spent four years staring at Mars data before seeing an ellipse; the program does it in about an hour. It also rediscovers Planck’s blackbody formula.

What excites him is new formulas nobody has seen. The method matches how human scientists already work. A child catching an apple, or a dog catching a tennis ball, is a trained neural net predicting a parabola. Galileo went further and wrote *y = x²*. Newton’s laws, simple enough to tell a friend and accurate enough to send rockets to a space station, are the same distillation. Humans already combine inscrutable neural nets with symbols and language; eagles see well and dogs catch balls, but they do not write equations. If we are information-processing entities, he says, we should be able to make machines that do both.

History inverted the order of discovery. Humans first built GOFAI, logic-based “good old-fashioned AI,” then neural nets, because the latter needed bigger computers. Evolution ran the other way: animals already train networks for vision and catching; what humans add is the symbolic layer. Tegmark sees two industrial strategies for putting the two together. One scares the heebie-jeebies out of him: make ever larger systems we still do not understand until they are as smart as people. If that succeeds, he puts at least a 50 percent chance of extinction before long — an “utter epic own goal,” the Knight Capital blow-up in seconds rather than minutes, long before anyone needs a sci-fi story about machines deciding to harm us.

He asks listeners to picture their least favorite world leader, then imagine that person holding incredibly powerful AI and imposing their will on the planet. Fridman asks whether the bigger fear is misuse or accidental destruction by people who think they are doing good. Tegmark is concerned about both: Hitler, Stalin, and Mao, he notes, believed they were doing good. The alternative strategy is intelligible intelligence. Use networks for intuition, then spend serious resources demystifying the box, converting it into something we understand, maybe even proving theorems about it. DARPA-backed work on symbolic verification, and the seL4 operating system that can be mathematically shown to be unhackable, are the existence proofs he wants for laptops and drive computers.

A music player in a car can be sloppy; the driving computer should be physically decoupled, barred from over-the-air updates while moving, and eventually proven. Companies should apply that standard only to the few systems whose failure threatens the firm’s life, and save money on the rest. Alignment, though, is not only a technical property of the autopilot. Germanwings pilot Andreas Lubitz told the computer to descend to about 100 meters over the Alps. The jet had maps and GPS, and no one had taught it the kindergarten rule: never fly into a mountain. If the pilot issues that order, Tegmark says, go to autopilot, email the police, and land at the nearest airport. Any car with a forward-facing camera should refuse to accelerate into a human. No manufacturer wanted the Nice truck attack; they simply had not thought to do the alignment.

Tegmark zooms out to evolution. Genes wanted copies of themselves, built brains, and aligned those brains with food and sex. That was successful value alignment: they created something smarter than themselves and tried to keep it on-mission. Then humans hacked the setup with birth control and diet soda, enjoying the reward while subverting the intent. Humans now play the gene’s role, creating other powerful non-human entities. Corporations give us supermarkets and hospitals. Institutions are supposed to keep them in check: sell poisonous food, go to prison. Powerful firms, like humans with the genes, have an incentive to hack the institutions that govern them.

Nineteenth-century Massachusetts railroad companies had bigger budgets than the state; Theodore Roosevelt’s trust-busting was a realignment. Tegmark does not want a fairy tale about whether corporations or robots are good or evil. A robot is a tool for surgery or for harm. A firm with good incentives opens a hospital; with bad ones it markets addictive drugs and you get an opioid epidemic. He notes that a new Biden defense secretary had been on Raytheon’s board — not because he is a pacifist opposed to Raytheon, but because the conflict of interest is obvious. Another Roosevelt, he suggests, may be due.

Fridman wonders whether collective intelligence always finds a last-minute realignment, a kind of evolutionary karma. Tegmark is not that optimistic, which is why the 50 percent extinction number is still on the table if we take the dumb AI route. History is full of fiascos, but they were local. Easter Island could collapse and other people could move in later. If Elon Musk does not get us to Mars and we go extinct globally, nobody is coming back. Stone-age rage with a rock took out a handful of people; an accidental U.S.–Russia nuclear war, which he says we have almost had about a dozen times, could take out six or seven billion. There is no law of physics that says technology will never be able to wipe out the species, and a designer pandemic as transmissible as COVID and as lethal as smallpox is not unfeasible in a lifetime.

Fridman offers an optimistic intuition: the most brilliant people he has met want to do good, so the cutting edge of technology may be staffed by people who rally in time against an engineered pandemic. Tegmark would love that to be a law of nature. He prefers not to gamble on it. Too much faith in last-second luck makes people skip smoke detectors. If you assume the government will handle every future crisis, you abdicate. The upside of getting this right, he says, is not the next election cycle but billions of years of healthy, wealthy life. That is worth working for rather than waiting on fairy-tale karma.

COVID, in his view, was a wake-up. Ask whether governments handled 2020 flawlessly; the answer is a disappointing no. South Korea, with more than 50 million people, had on the order of 500 deaths when they spoke, because it tested and contact-traced fast enough never to lose the thread — and because SARS, about 17 years earlier, had left the memory that pandemics are real. The lesson is to spend a tiny fraction of GDP on horizon scanning: which things could go wrong, which are likely, which are actionable, and then prepare, rather than waiting for the next pandemic or the next AI failure.

Fridman is also heartbroken that 2020’s discourse was less about understanding what happened than about competing “truths,” red versus blue, even science politicized. Tegmark agrees that most people are fundamentally good, and that nastiness tracks the information they are fed. Five hundred years ago, tell Swedish farmers the Danes are terrible and you get an invasion. Today the same pattern runs geopolitically and inside parties: the other side is not merely wrong, they are bad people. We evolved to judge 30 people we actually knew. Most people on the planet we have never met, so the quality of the feed is the whole game. Propaganda is old; Machiavelli already wrote the playbook. What is new is machine learning meeting propaganda.

In Tegmark’s liberal university bubble, many people blamed Donald Trump for the information mess. He sees Trump as unusually skilled at a game created by Facebook, Google, and others deploying machine learning to maximize time spent watching ads. Good people inside those firms, he says, underestimated how well the black boxes would work. The systems discovered that the best glue is strong emotion, especially anger, and that truth was optional. Print journalism could not compete; he believes the U.S. has less than half as many journalists as a generation ago. Most people now get news inside a bubble. Trump played that game. He did not invent the board.

Machine learning is a tool. So far, in news, it has mostly helped big tech keep people watching ads and, as a side effect, fragment democracy. Algorithms are basically free, Tegmark says, and can run on a phone. There is no reason they must only help the big guy manipulate the little guy. His nights-and-weekends COVID project is Improve the News, at improvethenews.org: a Google News-like aggregator with sliders to break the bubble. Slide left or right on the political spectrum and a different set of articles, a different “truth,” appears. Immigration coverage makes the split obvious. The first defense against manipulation is simply seeing the techniques.

A team of MIT students used machine learning to catalog bias. Fake news exists — “Hillary Clinton just got divorced” — but omission is far more common. Trump family finances and Hunter Biden’s finances do not get equal coverage; each side loves one story. A citizen who dislikes all corruption can slide both ways and see both. Noam Chomsky, a former MIT colleague, said propaganda is to democracy what violence is to totalitarianism: you need it when fear is not enough, which is why Western propaganda is higher quality and more believable. Tegmark finds Russian and Chinese scientist colleagues more alert to their own media than many American colleagues are to theirs.

David Rand at MIT, he notes, found that people will hear contrary views if they are presented respectfully. Opening Breitbart as a Democrat, or a left outlet with an ugly Trump photo as a Republican, fails in ten seconds. A “nuanced” slider surfaces more academic, less insulting versions. An “establishment” slider is, for Tegmark, the fun one: when almost every powerful entity wants you to believe something, both mainstream left and right will say it. Tobacco companies once buried coverage of smoking’s harms. After the Iraq invasion, journalism research showed CNN and Fox both pushing weapons of mass destruction while smaller papers called the evidence sketchy — voices most people never found. Hearing them, he argues, would have been better for U.S. national security.

On a one-to-three-generation horizon, Tegmark sees only two geopolitical endings: great for all of humanity, or terrible for all of us. Technology has no borders, and if people keep fighting while weapons get more powerful, luck eventually runs out. Since 1945 Western Europe has had no war and everyone got richer because of tech, not because they seized land. The optimistic century is one where America, China, Russia, and everyone else win together. Powerful forces, he says, are pushing the opposite way, demonizing other countries so that the new tools get used for aggression. That is the bridge into military AI.

He loved the dancing robots. One of the biggest growth areas in robotics, though, is autonomous weapons, and 2020 was their best marketing year. In Libya’s civil war and in Nagorno-Karabakh they made the decisive difference, and everyone else started wanting a copy. In Libya, the UAE flew Chinese-bought autonomous weapons while Turkey flew drones; neither outsider had skin in the game, and Libyans paid. In Nagorno-Karabakh, Turkish drones came from a company founded by an MIT AeroAstro graduate. MIT, Tegmark says, has a direct responsibility, and many civilians died. Because the systems worked, the push is now for ever more autonomy.

People obsessed with a future Terminator, he argues, should look at two nearer threats: machine learning manipulating the information flow that democracy needs, and an arms race in autonomous weapons that 2020 either starts or stops. The systems in those wars were still mostly remote-controlled. The next step is something the size and cost of a smartphone, given GPS coordinates or a face, that flies off and kills. The reason the United States banned biological weapons, Harvard’s Matthew Meselson told him, was simple: you do not want a $500 weapon of mass destruction that every enemy and non-state actor can afford. Nuclear weapons are expensive; that is the point. Cheap killer drones are the bio-weapons pattern, not the nuclear one.

He is not a pacifist. Current Pentagon policy keeps a human in the loop; algorithms should never make life-or-death decisions. Cross that line to beat jamming and light-speed delay, and you have a machine hovering over a country killing whoever matches a “bad guy” definition. ISIS will not use the Pentagon’s definition. Human soldiers, with all their faults, still have wiring that says do not kill children; an autonomous weapon is “the perfect Adolf Eichmann on steroids.” Meselson’s other insight: you do not need 100 percent chance of catching cheaters. One percent, even five, is often enough, because stigma plus the risk that the other side goes full tilt is a huge cost. Fifty years of a bio-weapons ban produced, in Tegmark’s telling, two superpower own-goals and no real bioterror. That is the association he wants for AI: cures and vaccines, not a reason to be afraid to go outside.

The Future of Life Institute’s first award went to Vasily Arkhipov, a man most people have never heard of. Tegmark calls his contribution the greatest of any human in modern history, then tells the story so it does not sound like hype. During the Cuban Missile Crisis, Americans dropped practice depth charges on a Soviet submarine they did not know was nuclear, did not know carried a nuclear torpedo, and did not know had authority to launch without Moscow. Batteries were nearly dead, oxygen was low, sailors were fainting, and the temperature was 110 to 120 degrees Fahrenheit. The captain decided World War III had begun.

Launch required three officers. The captain said yes. The political officer said yes. Arkhipov said no. Tegmark’s line is the episode’s moral: he does not want the species to rely on the right man at the right time. The Institute later found Arkhipov’s family living in relative poverty outside Moscow, flew his daughter to London after his death, and tried to honor them. The next award went to Stanislav Petrov, who saw five incoming U.S. missiles on flaky Soviet radar and, mostly on gut instinct, refused to escalate. Tegmark is glad Petrov was not replaced by an AI automatically following orders. A later award went to Viktor Zhdanov and Bill Foege, who helped eliminate smallpox — about 30 percent fatal, versus COVID under one percent — and, Tegmark estimates, spared some 200 million lives.

Arkhipov combined loyalty, experience, and cool analysis. He had been captain on another boat during a reactor meltdown, later died of cancer, and had watched crew members suffer, so nuclear war was not abstract. When explosions started and the captain screamed to nuke, Arkhipov noticed the charges alternating left and right and guessed the Americans were signaling them to surface, not trying to sink them. Fridman hears hope in the power of an individual inside a centralized state. Tegmark wants the opposite lesson: treat people like this as a wake-up, not a panacea. Because of Arkhipov we are alive to learn that playing Russian roulette with accidental nuclear war is not a long-term strategy. The survival probability drops exponentially if you keep pulling the trigger.

Fridman asks how Elon Musk’s AI fears map onto the conversation. Tegmark pushes back on the word “evil people”: people who do terrible things usually think they are doing good, because of bad information. Musk, in Tegmark’s model, thinks on millennial and cosmic timescales. The universe looks mostly dead; the opportunity is for it to come alive, with Mars as a first stop. From that view, nuclear Russian roulette and building ever more powerful AI we do not understand are both reckless. Musk wants humans to control the machines, not become cogs. Consciousness, Tegmark defines later, is subjective experience; beauty, meaning, and purpose are experiences. If Earth-originating life goes extinct and nothing else has telescopes, it is game over for those things in this universe.

Musk’s “summoning the demon” line, Tegmark says, was a single answer at the end of a long MIT AeroAstro Q&A packed with students, Buzz Aldrin in the house, after an hour on rockets and Mars. Media stripped it for clicks and cast him as a doom merchant. Stuart Russell of Berkeley, author of the best-selling AI textbook, gets the same Luddite smear; the joke is on the people who say it. The public still thinks the worry is dancing robots with AR-15s. It is not malice. Why did humans drive the West African black rhino extinct? Not because we hated rhinos. Goals failed to align, and tough luck for the rhinos. Click-through optimizers that polarized social media were already poorly aligned with democracy, and they were dumb by future standards.

On Neuralink versus Stuart Russell’s uncertain, self-doubting machines, Tegmark does not know which idea will work. He is convinced that if none of them work and we barge ahead, the species is probably extinct this century. The analogy is the Manhattan Project: they did not know which uranium-enrichment method would work, so they tried all four. Alignment research should be funded the same way. He splits the technical problem into three unsolved pieces: make machines understand our goals, adopt them, and retain them. Lubitz’s autopilot failed the first. Self-learning systems are like children, except the window between “smart enough to understand” and “too late, they are teenagers” may be very short. Recursively self-improving systems also have to keep the goal, unlike kids who outgrow Legos. Of the billions going into more powerful AI, only a minuscule fraction funds this work. NSF even awarded an institute labeled for AI safety to ocean and climate researchers. Speed up the safety research, he says; do not try to slow the technology. And treat alignment as a problem for political scientists and economists too, not only for people who own robots.

Tegmark treats both consciousness and intelligence as kinds of information processing. It should not matter, to a physicist, whether the processing runs on carbon in neurons or silicon in a gadget. Consciousness is “the way information feels when it’s being processed.” Most of what the brain does never becomes an experience: heartbeat regulation, the footwork of jogging. He hopes for an equation and a detector — in an ER, to tell locked-in syndrome from true unconsciousness; with a helper robot, to know whether shutting it down should feel like guilt or like pausing a tape recorder. Giulio Tononi has stuck his neck out with equations; maybe they are wrong, but the subject should be measured, not only mused over with beer.

Once that science exists, we would probably build some machines that are deliberately unconscious for boring, repetitive jobs, and perhaps conscious companions for the elderly so the sadness and happiness are not a fraud. Tegmark would be creeped out to form a real emotional bond with a robot he knew was faking it, the way a talking doll can harvest a child’s secrets onto the internet. If Ray Kurzweil uploaded into a robot that talked and laughed like him, and there was nobody home, that would be the wrong kind of immortality. A future of unconscious machine descendants throwing intergalactic rock concerts would be, in his phrase, a play for empty benches: the ultimate robo-zombie apocalypse.

Fridman worries that love will turn out to be a trick. Tegmark tries to cheer him up. He knows he enjoys food because genes do not want him to starve, and that does not make pistachio ice cream one bit less enjoyable. Feynman answered the artist who said science ruins a flower by turning it into quarks: the scientist sees the same beauty and more. The tree is made of air, carbon bound by the sun; a fireplace undoes the trick, and the flame is sunlight that had been stored in wood. From a hot plasma, through nuclei, atoms, planets, natural selection, and ever more goal-directed behavior, you eventually get love. Evolution is MuZero’s self-play on a planetary scale: quark blobs competing until you get giraffes, elephants, humans, and ice cream. Understanding that, Tegmark says, adds to the experience. It does not subtract.

Will AI find a theory of everything? Tegmark thinks a huge boost to science is already underway, AlphaFold being one example, and that unless we go extinct our understanding of physics will get good enough that technology is limited by the speed of light rather than by human invention. Near term, machine learning already hunts exoplanets, particle-physics signatures, and gravitational waves in data no graduate student can pour over by hand. Computational physics is shifting from GOFAI-style human-programmed solvers toward networks that learn a faster path to the same answer, the way MuZero outplayed both Kasparov and hand-crafted chess engines.

Three supercomputer hogs are the test cases. Lattice QCD already contains the theory that should spit out the periodic table — which atoms are stable, how heavy, what spectral lines — and Frank Wilczek has a Nobel for the underlying physics, but the math is too hard; Phiala Shanahan at MIT is using normalizing flows to speed the lattice. Black-hole collisions are a two-body problem Newton solved for classical gravity; in full gravity the objects radiate waves and merge, and mapping the waveform as a function of masses and spins can take months of cores. Cosmological simulations, at the other extreme, grow galaxies. Then comes the theoretical end: discovering equations from data, which is what AI Feynman is for. A proof is a string of symbols; checking it is easy, finding it is a search in a space that, at 10,000 symbols with ten choices each, is 10^1000 candidates, more than atoms in the universe.

AlphaZero punctured the story that intuition is uniquely human. Deep Blue had human-coded piece values and brute force; AlphaZero learned a value function that looks at a board and scores it. MuZero learns “what is good” across chess, Go, shogi, and Atari without being told the rules. If a net can get intuition for a good position, maybe it can get intuition for a good direction in a proof. Tegmark doubts a machine will be handed a Nobel; for a long time people will call it a tool, and a physicist who knows no machine learning will sound as odd as one who uses no computers. Fridman points to the Magnus Carlsen feeling: AlphaZero as another entity, not a calculator. Tegmark agrees that moment will come, and that it will roughly coincide with AGI, at which point who gets the prize is not the main worry. Influence arrives earlier. Recommenders that glue us to screens already hacked society, and they were, in the grand scheme, incredibly dumb. “The robots are coming,” he says, mostly not to kill us. They are coming to hack us — and if puppies can hack us into sharing a house, we should not be shocked that weaker algorithms can too.

Tegmark’s SETI intuition has two claims that could separately be wrong. First, civilizations going from bacteria to space colonization may spend only a tiny fraction of their lives in our present stage. Earth is the n=1 data point: 4.5 billion years of life, dinosaurs stomping around for over 100 million years without smartphones, then about 400 years from Newton to now, and no internet in Tegmark’s own childhood. This century we either start spreading life into space or we wipe out. If that compressed window is typical, a random civilization we stumble on should be bacteria or already post-biological, doing galactic construction — not mid-ham-radio. Quiet inward-looking “Buddhists” in virtual reality could exist, but this is a numbers game: among millions or billions of worlds, one expansionist is enough to notice, and a billion-plus Earth-like planets in the Milky Way, many a billion years older, makes “they all chose not to expand” a stretch.

The second, more controversial claim is that we may be the only civilization in the observable universe that has gotten as far as telescopes. The probability of life on a random planet could be 10^-1 or 10^-40; every order of magnitude is about equally likely, so the nearest neighbor could be 10^10 meters away or 10^40. Data cuts the range: closer than about 10^16 meters, the Sun’s neighborhood, we would know; farther than about 10^26 meters, the edge of the observable universe, light has not had time to reach us. The chance that the nearest neighbor sits in the thin remaining band is, he thinks, significantly less than one. The moral is not to lull ourselves with a Star Trek fleet that will take over if we fail. If the future of intelligent life in the observable universe is on us, that is terrifying and empowering. Nick Bostrom’s Great Filter cuts the same way: another null result on Mars is good news, because it suggests the hard step may already be behind us. Finding mice on Mars would be depressing. Rarity of life is something we should hope for.

Does he think about his own death, and the discoveries he will miss? Of course it sucks. His father once said life is fundamentally tragic; years later Tegmark understood. He is not afraid of something terrible after death. It will be game over. Finiteness makes pistachio ice cream and the rest of it more acute. What “dies” is not a skin cell, which dies next week anyway, nor the atoms, which are replaced. From modern physics, he says, you are the information processing: memories, values, dreams, passion, love. Not all of that dies with the body. Feynman’s ideas live on; Tegmark tries to keep a bit of Feynman alive, and even quoted him in this conversation. Teaching lets some of him persist in students. Equating the self with quarks is old-fashioned materialism without scientific basis, and it is uninspiring. Post-biological minds could copy all of the information, not a sampling, which would change both mortality and individuality — a hive-mind science fiction he does not pretend to feel from the inside. For us, finiteness still matters. The conversation, like all good things, ends anyway.