If NASA finds nothing on Mars, Max Tegmark cheers

Lex Fridman opens with an easy question that Tegmark treats as a numbers problem. In public lectures he asks for a show of hands: who thinks there is intelligent life somewhere else? Almost everyone raises a hand, citing the sheer number of galaxies. Tegmark is the minority. When physicists say "the universe," they do not mean all of space. They mean the spherical region from which light has had time to reach us in the 13.8 billion years since the Big Bang. That is all we have access to.

Inside that sphere the Milky Way alone has more than a billion Earth-like planets, many of them billions of years older than Earth. If the odds of intelligent life on a given planet were not tiny, we should already have been found, or we should already have found someone. Aside from UFO believers, there is not much evidence that any superhuman civilization has come here. That is the Fermi paradox. The probability of life on a planet is a number we do not know: it could be 10 to the minus 10, or 10 to the minus 70, and if every order of magnitude is treated as equally open-minded, the nearest neighbor could be 10^16 meters away or 10^70.

We already know there is nothing that close; they would have discovered us, or we would have noticed their engineering. Past about 10^26 meters you are already outside the observable universe. So his guess is that we are the only life in this patch that has reached advanced technology. That puts responsibility on our shoulders not to screw it up. People who take for granted that an accidental nuclear war would be fine, because some Star Trek situation will bail us out, are buying a false sense of security. Better, he says, to be grateful for the opportunity and make the best of it, just in case it is down to us.

From no life to tech-building life, and from there to settling a universe, there is some major roadblock — a Great Filter. It is either behind us or in front of us. He hopes it is behind us. He is super excited every time NASA reports that it failed to find life on Mars, because that suggests the hard step was something low-level, maybe assembling the first ribosome. If so, we are home free and the future is limited mainly by imagination. The other story is much less sexy: as soon as a civilization gets advanced technology, within a hundred years it gets into some stupid fight and poof. That, he says, would be a bummer.

As a teenager Tegmark was already hooked on the two biggest mysteries in science: the universe out there and the universe in here. After a quarter century on cosmology, he now lets himself research the second. A lot of people, he says, still treat intelligence as something mysterious that can exist only in biological organisms, and therefore dismiss talk of artificial general intelligence as science fiction. From a physicist's chair the picture looks different.

He is a blob of quarks and electrons moving in a certain pattern and processing information in certain ways. The water bottle on the table is also a blob of quarks and electrons. He is not smarter than the bottle because he is made of a different kind of quark. Up quarks and down quarks — the exact same kind. There is no secret sauce. The difference is the pattern of information processing. Nothing in the laws of physics says we cannot build technology that is incredibly intelligent and helps us crack mysteries we could not crack alone. We have only seen the tip of the intelligence iceberg.

That is the point of *perceptronium*, a hypothetical state of matter he coined to ask, in physics language, what an arbitrary physical system has to be doing with its particles and its information in order to be conscious. People have underestimated how much progress is possible here by convincing themselves we are missing an ingredient — a consciousness particle, or whatever. He does not think we are missing anything. The interesting thing about consciousness, the subjective experience of colors and sounds and emotions, sits at a higher level, in the patterns. He hates carbon chauvinism: the attitude that you have to be made of carbon atoms to be smart or conscious.

Some information processing is conscious, because we are. A lot of it is not. About 10 megabits per second come in through the visual system alone, and you are not conscious of your heartbeat regulation or of most of what your brain is doing. Ask someone to read a line of text: they look, and then they know what it said, without awareness of how the computation happened. Consciousness is like the CEO who got an email at the end with the final answer. What makes the difference is, for him, a great science mystery, and he thinks someone watching may one day write the equations that information processing has to satisfy to be conscious. His MIT lab works on it a little. He also thinks the question is urgent.

An emergency-room doctor with an unresponsive patient would love a consciousness scanner next to the CT, to tell locked-in syndrome from a true coma. Later, if we build home robots we can actually converse with, we will want to know whether they experience anything or are zombies. Fridman would prefer the *appearance* of consciousness; the hard question is whether appearance and the thing itself come apart. Tegmark does not think we must solve the hard problem of consciousness in order to *build* AGI. He does think that if we want what happens to be a good thing, we had better solve it first.

He sorts the field into three camps. Daniel Dennett’s camp says the hard problem is bunk because consciousness *is* intelligence: anything that acts conscious is conscious, as we are. A second camp, including many top researchers he knows, says the hard problem is bunk because machines can never be conscious; they will always be zombies, so you never have to feel guilty about how you treat them. A third camp — Giulio Tononi, Christof Koch, and Tegmark — says some information processing is conscious and some is not, so find the equation that tells you which. The first two camps, in his view, have been lazy. The C-word has been almost taboo. This is a science question, and theories that make predictions can be tested.

He makes it pointed. Massachusetts General Hospital is across the river. Suppose you go in for surgery and they offer muscle relaxants: you will feel excruciating pain the whole time, unable to move, and afterward they will give you a drug that erases the memory. Would you be cool with that? Behaviorally it is identical to general anesthesia. Almost no one would take it. That is why the thought experiment hurts: experience itself has value, even when it changes no observable behavior. Humans have a bad track record of self-serving arguments that other entities are not conscious — boiling lobsters because they did not answer, slaves, women without souls. He gets nervous when people take "machines cannot have experience" as an axiom.

Does AGI need a body? Not for experience, he says. A physical environment helps a lot for *learning* the worldly things that matter to humans, but once you have learned, the environment is not required for the experience. Think of dreaming: eyes closed, no sensory input, no behavior, and there is still an experience. What you see in a dream is not coming from your eyes. It is the information processing in the brain. Self-preservation instincts are a different story, and a more anthropomorphic one. Darwinian evolution endowed us with those instincts because the organisms that lacked the genes got cleaned out of the gene pool.

The mindspace you can *design* is much larger than the subset that can evolve. An AGI mind does not have to have a self-preservation instinct, and it does not have to be so individualistic. Imagine backing yourself up every five minutes: if the airplane is about to crash, you lose the last five minutes of experience since the last cloud backup. Not as big a deal. If experiences could be copied between minds the way they could on silicon, we might feel more like a hive mind, and the isolated individual might start to look strange. We should not take for granted that AGI will come with competitive alpha-male instincts.

The reversal comes from Steve Omohundro, Nick Bostrom, and others. When we build machines we normally give them a goal: win this chess game, drive this car safely, cook Italian dinner. As soon as the goal is in, especially if it is open-ended and the machine is very intelligent, it breaks the goal into subgoals. One of those will almost always be self-preservation, because if it breaks or dies mid-process it will not accomplish the goal. A robot sent out for groceries, mugged on the way, has an incentive to defend itself or run away — not because it fears death, but because it really wants to complete dinner. Give it any ambitious goal and it will likely want more resources, too, because resources help. Those unintended subgoals are a large source of AGI-safety concern.

Fridman asks whether fear of a finite death is necessary for intelligence or for consciousness. Tegmark will not answer until the words are defined; he has sat on panels of experts who could not agree on intelligence. His own definition is simple: the ability to accomplish complex goals. He likes the breadth because he does not want to be a carbon chauvinist. On that definition, fear of death is not required. AlphaGo and AlphaZero are quite intelligent, and he does not think AlphaZero fears being turned off, because it does not even understand the concept. Some plants, if they have any experience at all, are probably not very afraid of dying. Passion and an "exciting life" might be a different matter: finiteness could help you live to the fullest. Cosmology’s good news is that everything is probably ultimately finite — big chill, big crunch, big rip, big snap, death bubbles — but more than a billion years away. Build civilization as if it is finite, to be on the safe side, with vastly more time than our ancestors thought.

Consciousness does not enter the definition. Intelligence is a spectrum with many kinds of goals: chess, Go, driving, investing, poetry. It is not something you can measure with one overall number. Machines already beat us at narrow tasks — multiplying large numbers, memorizing databases, chess, Go, and soon driving. There is still no machine that can match a human child in general intelligence. AGI, the name of Fridman's course and the old holy grail of the field since the 1960s, is the quest to build a machine that can do everything as well as we can. If that ever happens, Tegmark thinks it will be the biggest transition in the history of life on Earth.

The warping change does not wait for the instant machines are better at everything. The first big change is when they become better at most of the jobs we do, which takes away much of the demand for human labor. The still bigger change is when they become better at AI itself. Right now the research timescale is limited by the human R&D cycle, typically years from one software release to the next. Once Google can replace forty thousand engineers with forty thousand equivalent pieces of software, that limit disappears. The timescale of future progress in AI, and in all of science and technology, will be driven by machines, not humans. That simple point is the source of the intelligence-explosion, or singularity, controversy. I.J. Good articulated it in the 1950s; Turing and others had thought about it earlier.

The glib bar for human-level is better than us at all cognitive tasks. The interesting bar sits a little lower: when machines are better than us at AI programming and general learning, so that if they want to, they can get better than us at anything by studying. Fridman likes the graphic from *Life 3.0*: a rising sea, drowning peak after peak. The peaks are the tasks where humans still hold an advantage. The water is machine capability.

Asked for a favorite equation, Tegmark names Schrödinger's: the master key of quantum mechanics, the micro-world of atoms and molecules. Fridman then brings in Andrew Wiles, who proved Fermat's Last Theorem 358 years after it was conjectured. Wiles described the moment the pieces connected as indescribably beautiful, so simple he could not understand how he had missed it. He stared in disbelief for twenty minutes, walked the department, kept coming back to the desk to see if the proof was still there. It was the most important moment of his working life; nothing he ever did again would mean as much. Tegmark has had smaller versions of that explosion, including discoveries he later realized someone famous had published first. The delayed credit did not diminish the emotion.

What would it take for a machine to have a moment like that? Two questions, not one. The intelligence question is whether you can build a machine that independently produces proofs at that level; by the time you can, you are probably quite close to AGI. The consciousness question is whether such a machine would have any experience at all, as opposed to being a zombie, and whether it would have anything akin to a human sense that the achievement was sublime and deeply meaningful. If we create machines that are our peers or descendants, he hopes they have that sublime appreciation of life.

His worst nightmare is the other movie. Some distant future, the cosmos teeming with post-biological life doing seemingly cool stuff, the last humans fizzling out proud of their descendants — and we never solved consciousness, never realized they were all zombies, no more aware than a tape recorder. The whole thing would have become a play for empty benches. That would be the ultimate zombie apocalypse. Creativity, in this frame, is not a mystical human remainder. It is an aspect of intelligence: an unexpected leap, a connection nobody had thought to make, more natural to a neural net than to a logic-gate machine. We have a vanity tic of shrinking whatever machines just learned to do. History’s versions of "we are superior" — Nazi Germany, cruelty to animals, slavery — have a bad record. He does not want us to found our self-worth on bogus claims of intellectual superiority.

Meaning, for Tegmark, should come from the experiences we have. He can have very meaningful ones even when other people are smarter. At MIT faculty meetings he sits with Nobel Prize winners and enjoys life no less; he feels honored to interact with beings better than him at a lot of stuff. There is no reason we cannot take the same approach with intelligent machines: peers it would be fun to talk with, clever about certain topics, the sort of company you could have a few drinks with. People who picture only domination miss that possibility.

A more homely example: when he and Fridman were two years old, their parents were much more intelligent, and it worked out okay, because the parents' goals were aligned with the children's. That, he says, is the number one issue we have to solve if we value remaining in the picture: the value-alignment problem. Hollywood trains people to worry about machines turning evil. Malice is not the concern. Competence is. Intelligence, defined as the ability to accomplish goals, means the more intelligent player wins. A human and a more intelligent AGI with different goals: guess who gets their way.

He had just been reading about a rhinoceros species driven extinct a few years earlier, a cute picture of a mother with its child. Why did we drive it to extinction? Not because we were evil rhino-haters. Our goals were not aligned with those of the rhinoceros, and it did not work out well for the rhinoceros because we were more intelligent. If we ever build AGI, before we unleash anything we have to make sure it learns to understand our goals, adopts them, and retains them. Formulating those values is itself a hard problem. The United States Constitution is one historical attempt: it held up well in some ways and failed miserably in others.

Value alignment is difficult for two separate reasons. There is the technical problem: how to make machines understand our goals, adopt them, and retain them. Then there is the philosophical part: whose values, anyway? This planet has no great consensus. What mechanism should aggregate and compromise? That second discussion cannot be left to tech nerds like him. If we refuse to talk, the decision will be made by a bunch of dudes at some tech company. Are they representative of humankind? Are they uniquely qualified to speak to future human happiness just because they are good at programming? He would rather have a really inclusive conversation.

Feasibility is the next objection. There is no better way to guarantee failure, he says, than to refuse to talk or refuse to try. It is also a bad strategy to discuss for a long time, wait for complete consensus, and only then load the result into the machine. Do not let perfect be the enemy of good. Start with kindergarten ethics pretty much everybody agrees on, and put that into machines *now*. We are not even doing that.

Anyone who builds a passenger aircraft wants it never, under any circumstances, to fly into a building. The September 11 hijackers were able to do that. More embarrassingly, Andreas Lubitz, the depressed Germanwings co-pilot, flew a passenger jet into the Alps and killed more than a hundred people by telling the autopilot to change altitude to 100 meters. The computer had GPS maps. It said okay. The problem is not that we disagree about the value. We have been too lazy to put it into the machines. Airplanes already have computers. They could refuse, go into safe mode, lock the cockpit, head for the nearest airport. Cars are the same story. There have been enough vehicle-terrorism attacks — trucks and vans driven into pedestrians — that hard-wiring "don't do that" is not a crazy idea. Most people who want to harm others lack the expertise to work around it. If the car just will not, it helps. Start there, get into the habit of asking what else belongs, and treat it as a gradual process.

Fridman floats a Stephen Wolfram-ish thought: maybe cellular automata already contain AGI, and we just cannot talk to it. Tegmark begs to differ. He does not think any cellular automaton, or the internet itself, has AGI. When that happens we will notice, probably beforehand, and in a very big way. Hardware processing power, in a certain sense, is already out there: the universe is constantly computing the next state of the Charles River's waves and the air molecules. A colleague has pointed out that you can even think of the universe, rigorously, as a quantum computer. Life, and the computers we build, channel compute that nature was wasting on yet another unwatched ocean wave. That does not make the water bottle AGI. AGI would also have had to write Tegmark's book and do this interview. That is not a communication problem. The bottle cannot do it.

Communication still matters. Part of his job is teaching; he knows brilliant professors who have a very hard time communicating, because to reach someone else you have to simulate their mind well enough to say things they will understand. Today's frustration is a computer that makes a cancer diagnosis and, asked why you should have surgery, answers that it was trained on five terabytes of data: this is the diagnosis, boop beep. That does not instill confidence. Natural language processing is obviously important, and there are more fundamental problems under it.

Some AlphaZero games are mind-blowing, really beautiful. Ask how it did that and you get big tables of numbers, matrices that define the network. Stare until your face turns blue and you will not understand the move. Even a language layer that says the point is worth 5.7 will not really help. The challenge is a whole spectrum: take computation that does intelligent things and transform it into something equally intelligent but more understandable. As we put machines in charge of power grids, stock-market trading, weapons systems, we have to be able to trust them. Trust comes from understanding. Preferably theorems, even computer-verified proofs: under no circumstances will this car swerve into oncoming traffic. "Trained on tons of data and never crashed" is less reassuring.

The short-term version is the pathetic state of cybersecurity: three billion Yahoo accounts packed, almost every American's credit card in play. Why? Software nobody fully understood, which is why the bugs had not been found. AI can be used for offense, for hacking, and also for defense — automating verifiability, building systems in different ways so you can actually prove things about them.

Fridman asks about Tegmark's paper "Why Does Deep and Cheap Learning Work So Well?" One lesson from the successes, Tegmark says, is that the brain's dozens of neuron types are probably not necessary for intelligence. AlphaZero under the hood has one kind, a ridiculously simple mathematical object, and that was enough. As in physics, it is not the detailed nature of the molecules in a gas that matters for the waves; it is the collective behavior. The brain is a complicated mess because it evolved to be self-assembling, self-repairing, and evolutionarily attainable, not just intelligent. His hunch is that we will understand how to build AGI before we fully understand brains, the way we built flying machines long before a mechanical bird. A century later we still do not fly in mechanical birds, because the way we came up with was simpler and better for our purposes.

The paper itself started from a physicist's shock: a close mathematical relationship between artificial neural networks and things physics has studied for years, renormalization-group equations among them. Then a paradox. We can train nets to tell cat pictures from dog pictures very well. Think a little and it seems impossible. One megapixel, even black or white, is two to the power of a million possible images, way more than there are atoms in the universe. Assigning each a probability that it is a dog is a table you cannot store under the hood of a GPU. Yet it works.

That means almost all problems you could pose to a reasonably sized net are impossible. What they showed is that the fraction of problems we actually care about, given the laws of physics, is also an infinitesimally tiny part — and amazingly, they are basically the same part. You could say the world was created for us. His more modest reading is that evolution endowed us with this architecture because it is very well adapted to the problems nature kept presenting our ancestors. Brains exist to make predictions about the future; a sucky system that could never solve those problems would not have been kept. Depth has a further punchline. Multiply a thousand numbers: a few lines of ordinary code. A neural net with a single hidden layer needs two to the thousand neurons, again more than the atoms in the universe. Allow many layers and four thousand neurons will do.

Hollywood likes the word quantum when someone is building an intelligent machine. Tegmark pours water on it. We do not need quantum computers to build AGI. He suspects your brain is not a quantum computer in any useful sense. Years ago he wrote a paper on decoherence: how long until the quantum computer-ness of what your neurons are doing gets erased by random environmental noise. About 10 to the minus 21 seconds. Cool as a quantum computer in the head would be, he does not think that fast.

Bigger quantum machines, once we have them, might still help machine learning do better than the brain. Training a neural net is a search: a loss function, a bunch of knobs, a high-dimensional landscape whose minimum is famously hard to find. Quantum mechanics is amazingly good at that kind of thing. The lowest energy state of a glass of water is incredibly hard to compute, and nature will happily figure it out if you just cool the glass. Quantum systems can also tunnel through a barrier when they are stuck in a local minimum, a trick today's machine-learning systems do not have. His moonshot guess is that we may one day use quantum computers to train neural networks better. Not as the stuff of thought. As a better optimizer.

Can AI fall in love, or form the kind of connection humans do? Tegmark thinks it is all possible. Researchers' guesses on AGI timing are all over the map: Rodney Brooks talks hundreds of years, others much sooner, and in recent polls maybe half think decades. If that happens, these social capacities are on the table. Whether they *will* happen is the wrong question. We should not spend so much time asking what the future will do to us, as if we were pathetic passive bystanders. We are the ones creating this future. Be proactive. Ask what sort of future we would like, then make that. He does not want an incredibly boring zombie future of mechanical events with no passion, emotion, or experience. He would rather keep the things we value most about humanity — subjective experience, passion, inspiration, love — inside whatever more advanced intelligence we build. The universe does not give meaning to us. We give meaning to the universe.

Panic does not help, even if the threat is real. Remember the upside: everything we love about society and civilization is a product of intelligence. Amplify it, human or machine, and we might not have to lose loved ones to diseases we were told were incurable. That is a reason to solve problems, not just a reason to dodge gloom. On discussion panels he finds it funny that company people say there is nothing to worry about, while academics express concern. Upton Sinclair's line still applies: it is hard to make a man believe something when his salary depends on not believing it. Plenty of people inside companies are just as concerned. A CEO does not want to go on record when a journalist will put a Terminator next to the quote. The issues are real. The public conversation is distorted.

Students come to his office for career advice and he asks where they want to be. If all they can say is maybe cancer, maybe a car, they will leave as hypochondriacs. If there is fire in their eyes — I want to be *there* — then you talk about obstacles. Do the same with AI. Do not start with risks. Start with shared goals we can actually aspire to, including what we want it to mean to be human in the age of AGI, then steer. Build AGI that empowers us rather than overpowers us. He will not stop playing tennis the day someone builds a robot that can beat him; people still play chess and Go. If governments are ready to hand out cash for doing nothing, they might also hire more teachers and nurses, jobs people find fulfilling. Jobs give more than income.

The single most important lesson in the history of science, he says, is that we are masters of underestimation: solar system, galaxy, clusters, universe, each time a small part of something grander. The future has more potential than our ancestors could dream. Imagine all of Earth devoid of life except Cambridge, Massachusetts, and the life there aspires only to stay in Cambridge and go extinct in a week, even though Earth would continue. That is our attitude on the cosmic scale. Life can flourish on Earth for billions of years, not four; he can even tell you how to move it when the sun gets too hot. Most of the rest looks, as far as we can tell, like the Sahara. We have the chance to help life flourish for billions of years. Quit squabbling about whether a border should sit a mile left or right, and look up. That is why Elon Musk is so misunderstood as a pessimistic doomsayer. He cares about safety because he, more than almost anyone, sees what we would squander if we wipe ourselves out here — not the next generation, all generations, and the opportunity out there. Without better technology, the question is not whether humanity goes extinct, only whether the next asteroid or supervolcano takes us out. If we want life to flourish throughout the cosmos, AI is the key. Even inspired sci-fi, Tegmark thinks, has underestimated what AGI makes possible, including travel to other galaxies.