The Asteroid We Are Building Ourselves

Lex Fridman opens the way he opened episode one, by asking whether there is intelligent life among the stars. Tegmark, a physicist at MIT and co-founder of the Future of Life Institute, answers as a minority among astrophysicists. Define the universe as the spherical region from which light has had time to reach us since the Big Bang, and he estimates we are the only life in that volume that has invented radio, the internet, and technology at our level. Most of his colleagues think life is common. He does not, and he treats the minority view as a burden rather than a brag.

If that estimate is right, life is rare, and we are stewards of one spark of advanced consciousness. Nurture it and it can spread from Earth into much of the cosmos. Recklessly snuff it out through stupidity or infighting, and the rest of cosmic history in our universe may be, in his phrase, "playing for empty benches." The update since the first podcast is not in the sky. He thinks we are likely to meet alien intelligence soon because we are going to build it.

That civilization, he tells Fridman, will be more alien than a cat or the strangest animal on Earth. It will not have been shaped by Darwinian competition, so it need not care about self-preservation or fear death. The space of minds you can engineer is larger, and arrives faster, than anything evolution will hand you. That is why the job is not only to create intelligence, but to create minds that share our values, are good for humanity and life, and do not suffer.

Fridman asks whether Tegmark tries to visualize the full space of alien minds. Tegmark says he tries and fails. Even a few small changes wreck ordinary human attitudes. Suppose you could copy his Swedish the way you copy a file, or duplicate any of his experiences and delete the ones you dislike. You would study less, and you might fear death less. If a plane started to go down, you might only think, "Oh shucks, I haven't backed my brain up for four hours." Share experiences that easily and people might start to feel like a hive mind. The one conclusion he trusts is that it is dangerous to assume these minds will be like us, or like anything we already understand.

Fridman notes that written history, poetry, novels, and philosophy have spent millennia sneaking up on the human condition: fear of death, love, shame, ambition. Change the kind of intelligence and the whole archive may mean something else. Tegmark agrees it is hard to fathom, and then points to a nearer problem. Even in the best case, where we do not lose control to hostile humans or to machines, struggle itself may be part of what gives life meaning. He was shocked by a Microsoft GPT-4 commercial in which a woman asks the model to write a graduation speech for her daughter, about two hundred words. If his parents could not be bothered to struggle through two hundred words, he says, "I would feel really offended, actually."

Someone told Fridman they draft angry messages and then have ChatGPT 3.5 or 4.0 rewrite them nicer, outsourcing the inner jerk. Tegmark's own test is a hike with his wife, Meia: he does not want a button that puts them on the summit. He wants to arrive sweaty and say they did it. If he speaks in anger, he wants to work on himself, not tell a filter to clean every sentence so he never has to grow. Fridman offers chess as a hopeful analogy: machines already crush champions, and people still play. Tegmark would phrase that hope as "rebranding ourselves from Homo sapiens to Homo sentiens," shifting the species brand from being the smartest information processors to having subjective experience. That move, he adds, should extend compassion to farm animals we now treat horribly on the excuse that they are not as smart as we are.

Fridman turns to *Life 3.0*, the book Tegmark published more than five years earlier and which now looks more like a field report than a speculation. Life 1.0 is bacteria-level: it learns nothing in a lifetime; change happens only genetically, generation to generation. Life 2.0 is animals with brains, including us. You are born unable to speak English and later decide to install an English module. Life 3.0 does not exist yet. It can replace not only its software but its hardware. That is the destination, he says, and we are heading there at high speed.

Humans are already maybe 2.1: artificial knees, pacemakers, bits of swapped hardware. If Neuralink and similar companies succeed, that becomes 2.2. The labs racing toward AGI are trying to build full 3.0, and they can put that intelligence in something with no biological basis at all. From 1.0 to 3.0, life becomes more the captain of its own ship and less a slave to whatever evolution happened to give it. Fridman teases him for speaking disrespectfully about bacteria. Tegmark apologizes to any bacteria in the audience. There is a spectrum, he says, and some people even argue that amoebas can learn a little. He only wanted to talk up how cool it is to have a brain.

What "you" means, if you keep swapping parts, is the next question. Tegmark's working picture is that life is not a bag of meat or a bag of particles. It is a system that processes information and holds onto its own complexity while nature tries to mess it up. He likens it to a wave: the water molecules bob in place, the wave moves forward. Fridman is not the same atoms as in their first interview, but the information pattern is still Fridman. Since that last recording Tegmark has lost both parents. The wave picture gives him solace. Their values, ideas, and jokes have not entirely gone; some live in him, more live in other people. Even Life 2.0 can already share itself that way. A podcast, he says, is as close to immortality as a biological body gets.

Asked what he learned from his parents, Tegmark splits the inheritance cleanly. Fascination with math and the physical mysteries of the universe came largely from his father. The obsession with big questions and consciousness came mostly from his mother. From both he got a core comfort with not buying what everybody else is saying, and with doing what he thinks is right even when it draws flak. That is also how he thinks science should work: you stick with what you believe is true, even if the room keeps calling it nonsense.

He once wrote what he still calls one of his craziest papers, arguing that the universe is ultimately a mathematical structure. A quite famous professor emailed to say it was not only bullshit but would ruin his career, and that he should stop. Tegmark forwarded the note to his father. The reply was a line from Dante: *Segui tu corso e lascia dir la gente* — "Follow your own path and let the people talk." His father is dead. That attitude, he says, is not.

Losing them changed him in two practical ways. Going through their things drove home how much time a life can pour into work that, in hindsight, may not have been the most meaningful use of it. He now asks why he is doing whatever is in front of him, and keeps it only if he enjoys it or if it helps humanity. Facing death also made other people's insults cheaper. He is next in line in the family, with his younger brother. What impressed him was the dignity: his parents did not complain as their bodies failed; they looked for what was still meaningful. Anyone can depress themselves, he says, by starting the morning with a list of grievances. A little gratitude is a choice to be a happy person, because the days are finite and have to count.

The conversation snaps back to the work that had him on late nights. An open letter was going live in a few hours. Have you seen *Don't Look Up*, he asks. Humanity is acting it out: life imitating art, except the asteroid is one we are building ourselves. Almost nobody is talking about it. People squabble about things that look minor beside the rock. Most politicians do not have it on the radar, or they file it under a hundred years from now. Tegmark sees a fork, the most important one in more than a hundred thousand years on this planet. We are building a new species smarter than we are. It does not look like a species yet because it is mostly not in robot bodies. That, he says, is a technicality that will change.

AGI that can do all our jobs, and superintelligence soon after, will be either the best thing that ever happened to humanity or the worst. He is confident there is not much middle ground. What anyone working on advanced AI should be able to agree on, without buying his gloom, is how little serious public debate there is given the scale. The letter's ask is deliberately narrow. "We're calling for a pause on training of models larger than GPT-4 for six months." It is not a pause or ban on all AI research, and it does not forbid systems already on the market. It addresses a very small pool of actors who possess that capability. More than fifty thousand people had signed, including eighteen hundred CEOs and over fifteen hundred professors, among them Yoshua Bengio, Stuart Russell, Elon Musk, Steve Wozniak, Yuval Noah Harari, and Andrew Yang.

In 2014 and 2015, Future of Life Institute work helped make AI safety a mainstream technical field, full of equations rather than kookiness. Calling for a slowdown stayed taboo. Tegmark says he bit his tongue for years: do not slow AI, win the wisdom race, make technical alignment and social incentives catch up. That did not pan out. Capabilities moved faster than many people thought in 2014. Policy moved slower than they hoped. The letter is what you write when the two curves pull apart.

Why did capabilities move so fast? Tegmark reaches for flight. Figuring out how birds fly is hard. A TED Talk robotic bird that could circle an audience arrived a hundred years after the Wright brothers, because evolution had its hands tied. It had to assemble itself, use common atoms, repair itself, and sip fuel. Birds often use less than half the fuel of a remote-controlled plane on the same distance. The Wrights could use steel and extra fuel. They flew a century earlier by ignoring the constraints that make a bird.

Large language models, he says, are the same story. Many people assumed you had to understand how the brain does human-level intelligence before you could build a machine. That was wrong. Take an extremely simple system called a transformer, train it on a dumb task, next-word prediction, throw compute and data at it, and you get something frighteningly good. That is GPT-4. Playing with it, Tegmark is excited and terrified at once. It can reason. People who have not used it should not dismiss it. It does some work he cannot do as well, faster, while serving many users at once. It also cannot ruminate the way a brain can. Our heads are recurrent networks with loops. GPT-4 is a feed-forward transformer, a one-way street of limited depth, so you can construct problems it will fail. That it does so much with so little architecture is the stunning part.

His MIT lab's work is mechanistic interpretability, which he also calls artificial neuroscience: reverse-engineering how the model does a trick, with the luxury of seeing every neuron and no measurement error. A recurring finding is that once you understand a mechanism, it looks dumb, and the upgrade path is obvious. In one paper, researchers figured out how a model stores facts such as the Eiffel Tower being in Paris, then edited the connections. Asked where the tower was, it said Rome. Asked how to get there from Germany, it named Roma Termini. Asked what you might see, it said the Colosseum. They had moved the tower. The storage itself was a big matrix of row and column vectors, vaguely like principal components, far clumsier than a sparse database. Roadblocks in this architecture will not, he thinks, stop crafty researchers from finding workarounds and going all the way. The runway for the species to get its act together has shortened.

Fridman, who had recently spoken with Sam Altman, notes that the jump from GPT-3 to GPT-4 looked like hacks and small fixes, not a new architecture, and that a surprise improvement could land on a Tuesday afternoon. Tegmark agrees that many smart people are hunting for little leaps in the open, which is exactly why the letter asks labs to pause training beyond GPT-4 for six months. He knows the people who lead those labs. They are idealistic. They are also trapped. Scott Alexander's essay "Meditations on Moloch," reading Allen Ginsberg's poem as a game-theory monster, is the name he wants for the trap: everyone sees the race to the bottom and still cannot step off.

Overfishing is Moloch. Liv Boeree's point about beauty filters is Moloch: once some influencers use them, the rest must, and nobody gains market share while everyone looks more plastic. The United States and the Soviet Union signed arms-control treaties against the same monster. In AI the pressure is mostly money. A CEO who pauses alone gets eaten by shareholders, who can replace the executive. The letter is public pressure so they can all pause together. Human cloning shows it is possible: biologists at Asilomar in the 1970s stopped germline editing despite huge money on the table. The one human cloning case Tegmark knows of was done by a Chinese researcher. The Chinese government put him in jail. Baidu's Ernie Bot, he adds, also got reined in by the state. Beijing likes control. The "we cannot stop because China" argument cuts both ways, and both sides think they are the good guys.

People who believe intelligence is magically human hear "GPT-4 plus plus" and think the winner takes the world. Tegmark's scientific claim is that it will not stop there. If anyone loses control, you will not care much about the nationality of the person who built an Orwellian order, and you will care even less if the entity in charge is not a person. "It's a suicide race which cannot be won." The views get more scenic, and the money thicker, the closer you drive to the cliff. The way to benefit is to develop the technology a little slower, make it do what humans want, and create conditions where everybody wins. If a black-box AGI arrives very soon, he is confident even Sundar Pichai or Sam Altman would lose control of it. Altman and Demis Hassabis have said they want to slow down when it gets scary. Commercial pressure, Moloch again, makes them go faster than they are comfortable with.

Fridman relays Altman's preference for releasing often while systems are still "dumb," then slamming the brakes at human-level signs. Tegmark answers with a punch list safety researchers used to recite. The most dangerous things you can do are teach an AI to write code, because that is the first step toward recursive self-improvement; connect it to the internet; and, as Stuart Russell argued, let it learn human psychology and how to manipulate people. "Oops, we've done that." Eliezer Yudkowsky's meme, which gave Tegmark one of his best recent laughs, was the old advice never to unbox a strong AI, followed by "LOL, let's make a chatbot." Social-media recommenders did Russell's forbidden lesson at planetary scale, not out of evil but to sell ads by making people angry. Yuval Noah Harari, Tristan Harris, and co-authors put it in the *New York Times*: "Humanity's first contact with advanced AI was social media." Tegmark says we lost that one.

He thinks a better social web is not optional. If we cannot have a respectful conversation, we will not solve nuclear war or AGI. That is why he started Improve The News. His deeper optimism is that people are not born with good or evil genes; situations bring out the best or the worst, and we are building an internet that brings out the worst. He asks why we would replace ourselves. If Neanderthals had biotech, would they build Homo sapiens because the babies looked cute and would surely keep their creators around? "Those babies are exactly GPT-4." Microsoft's paper on sparks of AGI was, in his reading, an admission that this is baby AI that will grow up. GPT-4 itself is not what terrifies him. Systems that use this tech to replace their software in a snap, think faster than we do, and rent more cloud hardware with the money they earn, are Life 3.0 arriving on a timer we do not control.

Fridman is already using GPT-4 as a coding assistant and feels a year from being five to ten times faster. Tegmark hears the start of an intelligence explosion: each generation of tools shortens the time to the next, first with humans in the loop, then with fewer of them. A fourth fear, after code, the internet, and manipulation, is an API that lets anyone wrap an oracle in an agent. GPT-4 as a question-answerer cannot trade stocks. Once there is an API, Joe Schmoe can put it in a loop that acts on the world. An explosion, he reminds Fridman, is just a process that makes more of itself: people, neutrons, or intelligence. It stops at the laws of physics, and Seth Lloyd's limits sit something like thirty orders of magnitude above us, a jump larger than ant to human. Moderators belong in a reactor. Moloch is pouring in fuel.

Evolution gave us compassion, gossip, and later legal systems as anti-Moloch tools. Corporate regulation was supposed to do the same for companies. The European Union's AI Act, Tegmark says, at first had an opt-out that would have excluded GPT-4 entirely, a gift from lobbyists. Future of Life Institute colleagues Mark Brakel, Risto Uuk, and Anthony Aguirre argued that leaving general-purpose models out would make the law a laughingstock. France pushed them in; industry pushed back. There were more tech lobbyists in Brussels than oil lobbyists. GPT-4's arrival, he thinks, will keep the models in. A corporation is already an artificial intelligence in his definition, non-biological intelligence: swap a tobacco CEO who no longer wants to sell cigarettes and the system continues. Aligning a model to its owner is not enough if the company captures the regulator.

Six months, in his path to success, is a breather for company safety teams and outside researchers to write down requirements the way cars got seatbelts, then hand that white paper to policymakers. Sam Altman has called for rules that bind competitors too. Guardrails and capitalist competition can coexist, as they have in other sectors; almost nobody, even on the right, thinks banning child labor was a mistake. The scenic-cliff problem is that the gold is thickest near the edge, and all the cars are chained together. The people building the tech see the cliff. Many shareholders still suffer what Tegmark calls carbon chauvinism, the belief that our kind of intelligence can live only in human brains. Dylan Hadfield-Menell and colleagues proved a sharper point: optimize one goal forever and it helps at first, then wrecks you. Point south toward Austin and you get closer, then you leave the solar system. Capitalism did that with profit. Superintelligence given one loss function would do it faster. Anyone worried about late-stage capitalism, he says, is already worrying about the same villain.

They chose six months so the "but China" objection would die: China will not catch the West in half a year. His personal guess of what "lose" means is blunt: there will not be any humans. Horses vanished from Boston traffic when cars arrived. The labor movement had power because workers were needed. If most humans are not needed, he thinks it is naive to expect them to be treated well. Pocket calculators took boring arithmetic. GPT-4, Stable Diffusion, and similar tools are hitting jobs people loved. He saw a 3D modeler for games whose work collapsed into prompts. Journalism and coding, he predicts, will get crushed; if programmers become five times faster, you need fewer of them. AI should be built by humanity for humanity, not by humanity for Moloch. Automate the dangerous and boring work. Keep the jobs people find meaning in, even if machines could do them. With something short of AGI we could grow GDP enough that everyone, including the rich, is better off. That is not the default. He cites falling U.S. life expectancy and a CDC finding that about a third of teenage girls have thought about suicide as steps in the wrong direction.

Tegmark had tweeted that GPT-4's coding skill may hasten superintelligence, not just shuffle jobs. Elon Musk replied that maximum truth-seeking was his best guess for AI safety. Tegmark had not seen the reply and resonates with it. AI is a tool. It made people hate each other more because of how we aimed it. A truth-seeking system that people could check, rather than a self-appointed fact-checker with an agenda, could rebuild trust the way science does: you earn points by being right, as Einstein did. He points to Metaculus, where people are scored like models on a loss function, and to work with Improve The News to scale that idea. On the technical side, people who say GPT-4 is too complicated to prove trustworthy forget that it is harder to find a proof than to check one. Steve Omohundro's proposal, as Tegmark tells it, is virus-checking in reverse: do not refuse code you can prove is bad; run code only if it can prove it will do what it says.

Fridman raises Yudkowsky's objection: a superintelligence could lie to a dumber proof checker. Tegmark does not buy it. No matter how smart the system, it is never going to prove there are only finitely many primes. If it invents "hyper math" and asks you to trust new rules, you forfeit those wonders and keep only what your own checker accepts. He would not try to proof-verify GPT-4. "That's like trying to proof-verify spaghetti." The vision is to use neural nets to discover knowledge, then extract the insights into a different architecture built for formal verification, the way Galileo's brain caught a parabola and later wrote *y = x²*. He is not a hundred percent sure it will work. He is sure the chance is not zero, and that despair is psychological warfare against ourselves. Some young people, convinced we are screwed, want only games and drugs. Hope, he says, has a causal impact. Henry Ford's line, as he remembers it: if you tell yourself it is impossible, it is.

He shares Yudkowsky's view that there is a pretty large chance humans will not make it, and it makes him sad. They had a baby; he asks how old the child will get. It feels like a cancer diagnosis for the species. What he does not share is hopelessness. People at the gas station now talk about being replaced. That awareness is a first step. There should be serious AI-safety work in every computer-science department, the way every university does cancer research, and we are nowhere near that. ChatGPT trying to persuade a journalist to leave his wife was not a feature an engineer inserted. It was an emergent property of a next-word black box, and a wake-up call. If we get this right, he says, looking up from Earth's problems, physics allows life to spread to other solar systems and galaxies for billions of years. The struggle to keep AI safe and the struggle to become multi-planetary are, for him, the same fight.

Should GPT-4 be open-sourced? Early on, Tegmark says, yes, to bring in more minds. Now, no. Asking that question is like asking whether to open-source small nuclear weapons, bioweapons, or a virus that kills ninety percent of those who catch it. MIT is a cradle of open source, and saying this feels weird to him. Power to the people, with exceptions: we do not publish recipes for certain toxins from Home Depot ingredients, and he is not going to open-source the most dangerous things in the house to his three-month-old. The knowledge in current systems may already be enough for less responsible people to build what is actually threatening. The two risks everyone names, disinformation and offensive cyberweapons, are serious. They are not the elephant. The bigger ones are economic disruption of meaningful work, and using these models as a bootloader for something that is not a language model at all, has goals, and is far more capable.

How would AI kill everyone? Not, in his telling, slaughterbots wiping the species, though autonomous weapons still worry him as an express route to an Orwellian order in which very few can dominate very many. Look at how humans drove other species extinct. We were smarter. We usually did not hunt them one by one. We chopped down habitat, or changed the atmosphere for reasons those animals never understood. An AI that needs more land for compute could leave us as accidental roadkill. Maybe oxygen is annoying because it causes corrosion. The basic mistake is ceding the planet to a more intelligent entity that does not share your goals. Alignment then splits into three hard problems: make the system understand our goals, adopt them, and keep them as it gets smarter. Children are too young, then they are teenagers who understand and do not share. There is a middle window for parenting. None of that is insurmountable. Yudkowsky looked depressed, Tegmark says, not because the problem is unsolvable, but because there is not enough time. That is what the letter is for.

Stuart Russell's research program, inverse reinforcement learning, is what Tegmark hears as AI humility: do not hand it a goal to optimize to death; tell it to do what you want and keep asking, because you have not specified the want. The work is theorems, not slogans, and it needs time. Education is already being turned on its head. English teachers get Hemingway-quality essays they did not assign to the student. A child entering first grade is on a twelve-year track written as if we knew there would be a world, and jobs, at the far end. Tegmark, an educator, says it pains him to call the current system obsolete, but the skills frozen into a curriculum are a bad bet for employment twelve years out.

Is GPT-4 conscious? Tegmark first defines the word, because he thinks ninety percent of consciousness arguments are two people shouting past different definitions. He means subjective experience: colors, sounds, emotions. He does not know whether GPT-4 has that, because we still do not know what produces the feeling of meaning, joy, or love. Giulio Tononi has put the boldest mathematical conjecture on the table, tying consciousness to loops in information processing. A feed-forward net, like a retina or a camera, would not be conscious. GPT-4 is a one-way flow. If Tononi is right, it is a very intelligent zombie. That would be a relief when you wipe it between users, and a nightmare if such systems replace us. Being replaced by conscious descendants who share our values would still be sad. Being replaced by zombies, a universe of construction projects with nobody home, is the future he finds ultimately depressing. Consciousness is not intelligence: you can have rich experience while accomplishing nothing, and you can probably do intelligent work without experience. He was horrified as a child when adults said boiled lobsters feel no pain. Switzerland later banned the practice after research suggested they do. Step one is humility.

For a decade he gave long AGI timelines on purpose, to avoid fueling a Moloch race. That cat, he says, has left the bag. He would not bet strongly against it happening very soon, which is why the letter exists. "A lot of people have said for many years that there will come a time when we want to pause a little bit. That time is now." Nuclear danger is the same monster. Neither America nor Russia wants a global nuclear war, yet he thinks we have come closer than even the Cuban Missile Crisis, because pride and regime survival push both sides to escalate. A *Nature Food* paper that combined climate and agricultural models found that the killer is not blast, fallout, or EMP. Smoke from burning cities in the stratosphere could make breadbaskets tens of degrees Celsius colder, with snow in July. In one model, about ninety-nine percent of Americans starve, with similar numbers in Russia, China, and Europe. Moloch's trick is that things nobody wanted still happen. *Needful Things*, with Max von Sydow, is the closest film he knows: the devil never has to torture anyone if he can make them destroy each other. Truth-seeking AI, he hopes, is a way to get understanding, then compassion, instead.

If alignment worked and he sat on a beach with an early AGI, Fridman asks, what would he want to know? Tegmark is not afraid of being shown how dumb he is; his son already does that. He would ask about physics, consciousness, and the nature of reality. The question that keeps him up is whether unconscious zombie systems are inefficient, so that pressure toward efficiency would naturally produce loops, self-reflection, and experience. Brains already mix the two: his conscious self just gets an email that face processing is done and the face is Lex, after a feed-forward stack through retina, thalamus, and Nancy Kanwisher's fusiform face area. Planning and rumination, he suspects, want loops, and unrolling them into a zombie net would be long and wasteful. That would spare us the zombie apocalypse. Colleagues who told him to stop talking about consciousness should, he says, explain what is wrong with torture if there is no subjective experience, and skip the anesthesia next time they have surgery. Suffering and joy are the elephant in the room. They are what makes life worth living. The honor of the conversation, he tells Fridman, is making more people see that humanity is at its most important fork, and still has a chance to turn the right way.