The Most Valuable Moment in VR Is Taking Off the Headset

Asked whether people will one day spend most of their lives in virtual worlds, Jaron Lanier does not sell immersion. The most valuable moment in virtuality, he has always found, is taking off the headset. Senses refresh. You perceive physicality as if you were a newborn, except with enough experience to notice how strange, delicate, peculiar, and impossible the real world is. The magic, on his take, remains in the physical world. He will not tell everyone else how to think, and younger generations have already liberated him from having to worry about it.

Even mixed reality, which he used to call by that name and which now goes as augmented reality, works the same way. One of his favorite uses of a HoloLens is to augment a forest, not because the forest needs help. Put a virtual object next to a real tree and the tree pops out as astounding: interactive, changing slightly all the time, hard to attend to until the comparison teaches you how. Media technology's gift, in this view, is that its defects make reality easier to see.

The practical version of that jolt goes back to the 1980s. Working with Dr. Joe Rosen at Stanford Medicine, Lanier prototyped the first surgical simulator. Fake anatomy is valuable for designing procedures and for training. Then the surgeon walks to a real person. Surgeons, he says, really get woken up by that transition. He thinks the transition is more valuable than the simulation. Fridman compares it to travel making home feel precious. Lanier takes it that way, and adds that all attitudes are welcome.

Why is he still drawn to meat space? Virtuality as we have it is made of software, and software, he says, is terrible stuff. It is always the slave of its own history and legacy, infinitely arbitrarily messy. Working with it brings out a nerdy personality in people, or at least in him, that he is not fond of. The physical world is not something we understand in the same way. That is a different kind of object than a pile of legacy code.

He is mystified when people ask whether the universe is a computer. If it were not some kind of computer, it would not follow principles consistently and would not be intelligible. We have physics and technology; we can program it. Of course it is some kind of computer. Trying to understand it as a Turing machine is probably foolish. Turing machines, he calls teases. They are universal only given an infinitely smart programmer, infinite time, infinite memory, and infinite clock speed. None of that can exist, so they are not universal in practice. They are a very particular machine inside the conservation principles of any reality worth being in.

Fridman's technical question is different: how hard is it to build a world realistic enough that we would not know the difference, or would know and still prefer to stay? Lanier first rejects a hidden premise in how that question is usually asked. It suggests people have a fixed ability to perceive reality. People are always learning, evolving, forming themselves. They are fluid, self-programmable, adapting. The more important answer, he thinks, is historical, not speculative about some future headset that finally fools everyone forever.

A peer-reviewed paper once showed that an early wire recorder, playing an opera singer behind a curtain, was indistinguishable from the real singer. To us the recording would be horrible and blatant. To people without the experience, it seemed plausible. An extremely crude video teleconference between New York and Washington in the 1930s was viewed as absolutely realistic. Civil War-era itinerant photographers sold pictures of people who merely looked like a few archetypes, because actually photographing a loved one was inconceivable and hiring a painter was too expensive.

Those episodes are how media first feel great, and then how we evolve through them. Maybe photography's greatest gift is that we can see the flaws in a photograph and appreciate reality more. Maybe audio recording's greatest gift is that we can now tell that opera singer from the wire. We should not limit ourselves by an assumption of stasis. Someone will still ask whether technology can go so far that it surpasses us. Lanier thinks that is a different, interesting question. The more important one is the one he just answered.

Will large numbers of people unmask and prefer to stay inside? His hope is that virtual worlds help people feel tender toward this physical world and keep it from getting too wrecked. He is not anti-digital tech. He spent years trying to make beautiful virtuality happen. He is anti the manipulative economy of social media, which he thinks is making everybody nuts at a level that might threaten survival. Fridman loves walking Skyrim; Lanier thinks loving artistic creation, play, and other people is wonderful. Humans are political and tricky. Classical music is beautiful and also a backroom status ritual. The search for purity, he says, will not get you anywhere.

Lanier once published in the *Journal of Consciousness Studies* on how fear and denial of death shape philosophies of mind. Sentimental dualism—the feeling that there is a soul apart from the physical brain—is obviously motivated by a hope that whatever that is will survive death. That hope sits at the core of many world religions. The hardcore opposite, that the brain is a computer and that is all, is motivated in a remarkably similar chain: I will figure it out in my lifetime, upload myself, and live forever. People who look like opposites in character and cultural references are, on this point, remarkably similar.

Hardcore computationalism about the brain has, to an incredible degree, turned into medieval Christianity. There are people afraid that the wrong thought will piss off the super AIs of the future, who will come back and zap you. Ernest Becker's *The Denial of Death* argued that mortality-fear is the worm at the core of civilization. Fridman asks whether that fear is a prerequisite of consciousness. Lanier says the question crosses a vast cultural chasm that separates him from most of his colleagues, and he cannot answer it without a huge deconstruction.

He does not believe in AI. He does not think there is any AI. There are just algorithms; we make them; we control them; they are tools, not creatures. The claim rubs people the wrong way, and he knows it. His main mentor when he was young was Marvin Minsky, the principal author of the computer-as-creature rhetoric that the culture still uses. Minsky populated AI with most of its tropes. Lanier always rejected it, and they argued for decades. The last time he saw Minsky, the older man was frail in the Brookline house. A mutual friend said not to start the AI fight. Lanier showed up anyway. Minsky looked up and asked if he was ready to argue.

Nobody can claim absolute knowledge about whether somebody or something else is conscious. That is faith. Lanier used to call the practical version the circle of empathy: you draw a circle, and what is inside might deserve consideration; a rock sits outside. The circle is based on faith, but it cannot be pure faith. Make it too big and you suffer incompetence—if you will not kill bacteria, you cannot brush your teeth. Make it too small and you become cruel. People are clannish and always ready to shove some designated group out.

Is he shoving AI out? If you include AI, he says, you exclude yourself from being a good engineer or designer; you face incompetence immediately. Algorithms need to be subordinated and treated with much more skepticism. Fridman asks why we cannot grow with systems the way we grow with dogs, cats, and partners, with transparency and the right to leave. Lanier reaches for Alan Watts: morality is like gravity, unreal in some cosmic frame and very real here among us. Identification with a remote frame, he suspects, cannot be separated from wanting to feel separate from and superior to other people. We are not competent yet to talk about those absolutes.

He is not against humane technology. Musical instruments are technologies that help people connect. During the pandemic he designed Together Mode, so people see themselves seated in a classroom or theater instead of in squares, with something like proper eye contact; it seems to promote empathy. The AI idea, he says, is not new. It was born with Adam Smith's invisible hand: we build an algorithmic thing, it gets a bit beyond us, and we decide it must be smarter. Everybody eventually became a Keynesian because the hand was not infinitely smart. People who really believe it is infinitely smart screw up their economies. Recognize the economy, and the algorithm, as a subservient tool.

Fixing social media, Lanier says, forces everyone into economics, a field he always thought boring. He has sat in the meetings where platforms sell themselves to advertising holding companies. The fiction is that the algorithm will get really good at getting people to buy something. Returns suggest otherwise. It is more like a cognitive-access blackmail payment: to be connected and known as a business, you put money in. Stanford used to run a program called engage-and-persuade. The engage part works. The persuade part is iffy. The industry plays make-believe that persuasion works more than it does.

He is not anti-advertising in the old sense. It can be demeaning, banal, and dishonest, and he thinks that, looking at the long arc, it also helped bring people into modernity. The damage from social media is a different mechanism. Algorithms present a stimulus and have to see if you respond. The measurement is extremely crude: a click, maybe a stare if a camera is on. Crude enough that it only catches dramatic responses—scared, pissed off, aggressive, horny. Those are the old fight-or-flight circuits, helpful in survival once in a while, not people at their best.

Once you are engaged on those channels, you get incrementally more paranoid, xenophobic, and aggressive. You get a little stupid. You become a jerk. It happens slowly. People who get hooked get drawn further into being at their worst. He wrote a book of arguments for deleting your accounts and does not have one himself; reaching him means an old-fashioned address on a crappy website. Even so he gets a flood of mail from people who quit and feel better—a flood that would be a trickle to Facebook. He does not think you should try to engineer personal growth or happiness. Design a system that is respectful, subordinate to people, and free of perverse incentives, and there is at least a chance of something decent.

People have to be responsible for curation, he says. Algorithms in this era do not understand enough to recommend. He is not saying that as an outside critic. He is in the middle of it, knows the math and the corpora, and his office is funding GPT-3 and other systems at the edge of what is possible. It is still statistical emergent pseudosemantics, not a deep representation of anything. Fridman says YouTube discovery has been helpful to him. Lanier asks whether he has ever started from an anonymous place, or from a known person, and gone fifteen or twenty hops, always taking the top recommendation.

Lanier has done that experiment many times, including with high-school kids, dozens at a time. YouTube is so large that a statistically solid result is hard, but in the majority of those runs, after about seventeen or eighteen hops you end up in weird, paranoid, bizarre territory. That is the stuff the algorithm rewards most, because of the crude feedback. He is not saying YouTube never recommends something cool. He is saying its fundamental core promotes a paranoid style, irritability, xenophobia, fear, anger, selfishness, and separation. A large citizen-science version would be better, except YouTube would detect it and change.

Fridman wants platforms that strive for healthy conversation without a dogma of what healthy is. Lanier looks for places online that already have it. GitHub, for the most part, has had a relatively low poison quotient. People have a stake: actual code, actual stuff being done, not empty status games. As soon as you have a real-world stake you have a motivation not to screw up the thing. Dumping on someone on TikTok has no price. On a shared project you have to get decent. Fridman plays the warning back: if you have a stake, a platform can use that stake to manipulate you, because you want to preserve it. That, Lanier says, is why the conversation has to become economics.

Lanier has studied data dignity for a long time and wrote an earlier version in *Who Owns the Future*. He lives on a hill in Berkeley that climate change might burn. The city does some things; the electric company sort of does; property owners tend their lots. There is also a middle layer that fascinates him: groundskeepers, many not legally present, many without English, who cooperate so trees do not touch and fire does not jump lot to lot. He started talking to them during the pandemic. They are a small web keeping the hill alive.

Future one is the one being built. A van with a kindergarten paint job and cameras observes the gardeners. Years later, robots that can shimmy up trees arrive. The robots are good and scale to more land. The people are out of work, dignity gone, maybe on basic income as wards of the state. Every time in history a centralized dispenser of benefits has existed, he says, it gets seized; that has happened to every communist experiment he can find. People will seize the algorithm too. Things we think are decentralized often are not: something like sixty people own a quarter of all the bitcoin.

Future two: the gardeners see the van and form a data union. California even has a baby data-union law emerging. They sell their data, make it better than silent grabbing would have been, improve the robots, and become a respected expert class. Pride and agency rise. What the robots do also changes. Instead of a purely functional keep-the-hill-from-burning job, a creative community starts asking about climate mitigation, fresh water, healthier food, waves of cultural groundskeeping. Future one and future two have the same robots and the same algorithms. There is no technological difference, only a human one. That second future is data dignity.

On social media, that looks like massive data unions for people putting content in, smoothing winner-take-all a bit, and users paying the way they pay for Netflix, with an allowance for the poor. An honest measuring device yields a bell curve: a few do really well, many do okay, an expanding economy of creativity lifts more people out than put money in, and society still supports the lower wing with a safety net, not universal basic income as the whole design. Not everyone will live on TikTok videos or groundskeeping-robot data. Many niches will evolve. That large number is the economic potential for new creative classes.

Getting from here to anything in that zone is hard. Lanier knows Jack Dorsey and thinks he is striving to make Twitter better, a good human being whose mind is in a lot of good places, under enormous business pressure, trying to turn a ship already moving. Almost every social platform that gets big has a honeymoon when it is sweet and cute: early Twitter, Snap, TikTok. As they scale, algorithms become more influential than the early people, the paranoid style rises, and they get dark. TikTok's shift, he says, has been recent. Scaling reveals the flaws in the incentives.

He imagines a future where many people would love to live on TikTok dance videos, know that is hard to get into, and still work like programmers on GitHub: crews with rights, negotiating returns, joining artists' societies to soften the randomness of who gets the network-effect windfall. A person might join a thousand data unions, even ten thousand, a hedged portfolio, some of them trickling money for health studies. People find the GitHub-like community where their taste and correctives feed the algorithms and robots of the future. That is an alternate economy that rewards productivity, creativity, and value as perceived by others—a genuine market, not a center doling out worth, not the cheapest grab at attention, which is how you get the lizard brain.

Fridman wants complete control and transparency over data, a one- or two-click exit with your data, and individualized control of the ranking algorithm. Lanier is in wholehearted agreement, with a caution. Years ago he co-founded an EU advisory board with Giovanni Buttarelli, a former Sicilian prosecutor who used to jail mafiosi and was not scared of Mark Zuckerberg. Their path of transparency and consent fed into GDPR. Empirical feedback: most people get stymied by that kind of management. It differentially benefits those with a technical mindset. Incentives have to help creative people of the future make money because you get a cut, growing an economy, not only serving programmers. Money itself hides information; Adam Smith's charitable reading of the invisible hand is that you could never follow the whole chain. Incentives that reduce complexity can work, and can be manipulative or not.

Lanier had advocated some kind of digital currency for a long time. When Bitcoin and the original blockchain paper appeared, his heart sank. Fancy thought and careful distributed security were being applied to recreate the gold standard: retro, dysfunctional, useless from an economic point of view. Using computational inefficiency at boundless scale as security is, he says, a crime against the atmosphere. People knew that when the first paper came out; they said if this thing scales it is a carbon disaster. There are people trying to be thoughtful—he suggests talking to Vitalik Buterin—and he is not against a cryptographic currency that actually helps. He has watched Bitcoin for years and has not seen one example of it creating economic growth.

The obsession with government as the problem is, in his view, a way of refusing politics: if I can get rich, nobody should bother me. His theory of freedom is that the only authentic form is perpetual annoyance. Annoyance means you are actually dealing with people, because people are annoying. Perpetual means the annoyance is survivable and does not destroy everyone. If you have that, you have freedom. That is politics. He will invite you to a Berkeley city council meeting to feel what perfection is. If you are not annoyed by other people, you are not free; you are trapped in a temporary illusion that will fall apart. An abstract thing that protects us from having to deal with each other is always an illusion.

The problem is not always an overbearing government. It can be criminal gangs, weird cults, abusive clergy, failed infrastructure, poisoned water, a failed grid, a crappy education system. Obsess over one kind of problem and the others pop up. He grew up adjacent to Soviet catastrophe: his wife is Russian, her father was sent to the gulag, and on his father's side almost everyone was killed in a pogrom in Ukraine. Unlike his anarchist friend Michael Malice, he believes government can do a lot of good. Asked whether cryptocurrencies do more to prop up corrupt regimes or to resist them, he notes early unknown Bitcoin whales who do not look like people you would want to support, and transactions that have tended to support criminality more than productivity. Old-fashioned currency is still an engine of growth. He is pro digital currencies and anti economics wiping out politics.

In the 1980s someone used "viral" as a metaphor for network effect in order to talk about how bad it was: distortions that ruin economic incentives. After a pandemic, people still think of viral as good because they imagine themselves as the virus, on the beneficiary side. You are not likely to be there. Money blinds people from basic morality. Lanier argued with some of the initial Bitcoin people that it is intrinsically a Ponzi: early people have more than later people, and further down the chain you get gambling-like dynamics and weird network effects that wave around the little people.

The conversation turned to gambling. One defense of Las Vegas is that losers paid for entertainment. The argument made to him was different: they get a chance to experience hope, and a lot of people do not get that chance. He finds it heartbreaking. He has talked to young people for whom crypto was the first thing that gave them hope. Money, in the long term, will not be a source of deep meaning. It is good to have enough. It should not be the source of hope. That is the psychology of trying to base a civilization on semi-random peaks of network effect. The peaks need to be softened.

He co-authored a paper with Lee Smolin, *The Autodidactic Universe*, describing a universe that learns its own physical laws. The technical core is a triple correspondence: theories of physics as matrix models, already established, and those models as a machine-learning system, so you can move among descriptions and get more tools in a field that has felt stalled. The storytelling part, which Smolin loves, tries to push the origin story further back: start with really simple things that evolve, instead of a finely tuned initial condition. There are already dozens of papers noting eerie equivalences between physics ideas and machine learning. They want an actionable framework, not another metaphor. He emphasizes baby steps and does not want to over-claim. On whether we are alone, he says we are not yet in a privileged enough position to know much. You can view that as frustrating or as charming, like the first year of TikTok.

If he sits quietly and plays music, it sometimes feels as if there is a lot more life flowing through the universe than the scientific story captures. He is speaking as an artist then. Music towers above him; he has no overview. You can say people evolved it to coordinate brains on a pattern level, and some of that is probably true. It also has a substantiality that is philosophically impossible. The understanding he feels when he hears the right song at the right time is not comparable to anything on Wikipedia. That mystery makes him think panpsychism could be true. He is not a panpsychist. He will proudly remain a dualist for people and cats, because there is a pragmatic imperative to treat people as special. Listen to anyone locate consciousness in a soul or a special brain widget, though, and you pretty much say forget it and go panpsychist.

The moments that defined him were not the good ones: his mother's death, being under the World Trade Center in the attack. Hard lessons make a world that avoids them stupid, and he does not wish hardship on the young; they deserve a grace period of naivete. Advice, unoriginal over a thousand years: pay attention to what you are fundamentally feeling, find cover from constant social sway, believe experience is a real, mystical thing rather than squirting magic dust on programs. Kindness requires genius; facile kindness is not kindness; effective kindness takes skill and hard lessons and is the most rewarding thing. When you are young, broken hearts and status pain feel total; remember what a fragile crust that is. If he said this to his daughter she would already be out of the room.

His mother was a Viennese piano prodigy who survived the camps and died in a car accident in the United States. He still connects to her through music, and learned Beethoven sonatas for her in an exaggerated, drippy way. He has always been against clinical precision; a great pianist should feel the music and make it messy, maybe the way a blues pianist plays. Pauline Oliveros, in whose band he played, loathed Beethoven as all ego, emotion turned into an enemy, self-importance at others' expense. Jon Batiste, by contrast, can sight-read the sonatas perfectly, slide from jazz into the *Moonlight*, and talk to Beethoven as if he were James P. Johnson—another musician in conversation, not a god. That relationship looks freeing.

He used to be unable to think about his own death. Age makes it familiar: you know people who die. He is still afraid, less so, without a revelation, just familiarity. He believes in the future of people, or of this whole thing of life on Earth. You cannot both believe in the future and want to live forever. You have to make room. Optimism has to come with humility; you have to make yourself small. Life seems short. He cannot find the complaint department. Meaning, he says, is a word we use as if we knew what it meant. He grew up in southern New Mexico, where Clyde Tombaugh, who had discovered Pluto, taught him to grind telescope mirrors and look at a globular cluster as actual photons, a three-dimensional vastness. Meaning lives in a context. Using a thought experiment defined by an outer frame—a Turing machine with its clock and observer, quantum mechanics with its external context—to talk about ultimates is a category error. It may be aesthetically valuable. Treating it analytically is not sensible. Music, maybe, escapes any particular context. That, he adds, is the crazy artist talking, not the scientist.