Hundreds of Miles Apart, Zuckerberg and Fridman Sat Close Enough to See the Wrinkles

Lex Fridman is in a dark room in Austin with a Quest headset on. Mark Zuckerberg is in another conference room, hundreds of miles away, with a headset on too. Nobody else is in either room. What each man sees is a photorealistic Codec Avatar of the other, sitting close, with spatial audio. Fridman apologizes for crowding Zuckerberg's personal space. Zuckerberg tells him he had already moved the avatar back a few feet before Fridman put the headset on, and that they have choked each other in jiu-jitsu from farther away than this feels. Fridman calls it the most incredible thing he has ever seen. His voice catches. He says his heart is racing. He is almost getting emotional.

Nothing on the wire is a video of the room. Both of them sat for scans in a Meta research project Zuckerberg calls Codec Avatars—the transcript's "Kodak" is the usual ASR slip. They made a large set of expressions. The system built a computer model of each face and body, then collapsed it into a codec. With the headset on, cameras read the live expression and send an encoded version of how you are supposed to look. Beside being photorealistic, Zuckerberg says, that is much more bandwidth-efficient than transmitting a full video, let alone a 3D immersive scene. The counterintuitive point is the one product people miss: the more the face is a local model, the less you have to stream.

What sells Fridman is not smoothness. It is the flaws: freckles, small color shifts, wrinkles, asymmetry, the corners of the eyes, what the eyes do when you smile. "Perfection isn't the thing that leads to immersion," he says. Zuckerberg supplies the word *wrinkles*, then *asymmetry*, then notes that eyes are a huge part of it. Studies keep showing that even when people are speaking, most of the communication is not the words. Cartoon-style expressive avatars can wear some of those expressions. Photorealism, Zuckerberg thinks, is what makes the presence feel magical.

That is the core of the virtual- and augmented-reality bet as he tells it: deliver a sense of presence, as if you are there together, no matter where you actually are. They are in two different states, halfway across the country, and it looks as if Fridman is sitting right in front of him. Fridman's mind goes straight to loved ones. Having this kind of conversation with them, he says, would change everything. He also thinks they have crossed the uncanny valley. It looks like Zuckerberg. It feels like Zuckerberg. It does not feel weird.

Fridman flew to Pittsburgh for the full scan. The hardware and software are elaborate, and the sitting is long. Zuckerberg's near-term picture is not that everyone wakes up with a Codec Avatar on the same day. A small number of people do these very detailed scans first, partly because the team is still over-collecting expressions until they know how far they can extrapolate. The project already in motion is a phone scan: wave the camera in front of your face for a couple of minutes, say a few sentences, make a bunch of expressions, finish in two to three minutes, and get something at the quality they have in this room. Right now they can scan you if you have hours to sit. Driving the result on a Meta headset that is already for sale is solved. Efficient production of the scans is one of the last pieces they still need to nail.

A dark room is fine for a podcast. It is not the point of the metaverse. You can already video-call on a phone. What you cannot do on a phone, Zuckerberg says, is be physically there together—play games, take meetings, and, once mixed and augmented reality arrive, walk into a room where some people are flesh and some are photorealistic overlays. He expects a progressive rollout over the next few years: more scans, then integration into Meta's apps, starting with work and remote meetings. Workrooms already makes people feel seated around a table; the complaint is that the cartoon avatars do not feel realistic enough. Something like this, he thinks, would matter there. Quest 3 is about to be the first mainstream mixed-reality product. Codec Avatars are not a mainstream product yet.

There is still tuning. People emote to different extents. How wide is your smile, and how wide do you want it to be? Some people may prefer an avatar that is more emotive than their actual face. Zuckerberg gets a lot of grief for a relatively stiff expression. He may feel pretty happy and still make a pretty small smile. For him, the interesting possibility is an avatar that expresses how he is feeling better than he can physically. They still want to start from how people actually emote. Fridman predicts the meme: the two most monotone people, talking in the metaverse. He thinks that makes the demo harder, not easier, because the signal lives in the eyes. He does not do remote podcasts for a reason. This one, he says, breaks all of that.

Quest Pro shipped last year at $1,500. Quest 3 is coming out at $500, and in some ways Zuckerberg thinks its mixed reality is better than the Pro they are wearing for the interview. The mixed-reality cameras and sensors are multiple times higher resolution, a bottleneck they only fully saw after living with Pro. Qualcomm's next-generation VR and MR chipset, in a custom version Meta worked on, was available this year for Quest 3 and was not available for Quest Pro. So Quest 3, even though it is not the Pro product, has a stronger chipset than last year's Pro line at a third of the cost. It also does all the VR work Quest 2 did, with a better display and chip, and it is 40% thinner.

He draws a contrast with phones. Companies ship a new smartphone every year and you can barely tell it from the one before. Each new headset, so far, has a major new capability. This generation's is mixed reality: digital people and objects superimposed on the world. In a category that is still moving that fast, the SKU that ships a year later on a new chip can beat the expensive SKU that had to launch on the old one. That is a product-rhythm fact, not a branding accident.

The industry pitch for mixed reality is holograms in your living room. Zuckerberg's homelier pitch is safety. A lot of today's VR does not need to be fully immersive. If you are playing a shooter or doing fitness, people worry about swinging an arm into a lamp. Put that experience in mixed reality and they get the 3D immersion while still seeing the physical world, which just feels safer. Fridman tried Quest 3 with friends and furniture in the room. He could see the couch, see the people, and see them trying not to laugh while he swung his arms. Zuckerberg's version: someone is shooting at you and you hide behind your real couch. In a room with no windows you can add a window, enjoy the view, and still have zombies running at you. The physical room stops being the enemy of the game and becomes part of it.

Passthrough is not just cameras on the front of a small headset. Multiple cameras have to capture the three-dimensional space, then an AI and compute stack has to remap that to your perspective, because the cameras are not where your eyes are and no two people have eyes in the same place. That warp has to run in real time and look natural before you even drop digital objects into the scene. Hand tracking, running on those same cameras, has gone further in a shorter time than Zuckerberg expected. Fridman mentions the piano demo: keys appear on a real table, fingers come down, sound comes out, no controller, no obvious latency, while the system also tracks other people in the room. Codec Avatars are the parallel track: ultra-realistic copies of people standing in that same mixed space.

Zuckerberg is moving toward glasses that look like glasses, where you still see the physical world and also see holograms. His thought experiment: maybe by the end of this decade, you walk into a room and there are as many holograms as physical objects. People call the physical world the real world. He thinks the physical world is super important, and that the real world is actually the two coming together. Until this technology they were divorced: you entered the digital through a small screen in your pocket or a bigger one at a desk. Screens, books, art, board games, cards, ping pong—many of those, he says, work as well or better as holograms, because you can snap your fingers and instantiate a ping-pong table in the living room, then snap again and have it gone. Physical objects carry sunk cost and floor space. Digital ones do not.

Stack that on Codec Avatars and a meeting looks different. You show up in a conference room wearing a headset now, glasses in something like five years. Some colleagues are physically there. Others sit in the other chairs as photorealistic holograms that feel like them. Then Zuckerberg adds a third occupant. You may also embody AIs. The meeting is you in the flesh, a couple of people as holograms, and Bob the AI, an engineer on the team, present as a realistic avatar. A chatbot in a text box is a tool. A face with a seat and a voice is a presence, which is efficient and, though he does not linger on this, a different trust problem.

Why photoreal instead of the cartoon avatars Meta already has? Motion of eyebrows, eyes, and cheeks could go on a cartoon too. Zuckerberg thinks photorealism is the bridge that lets people take hybrid meetings seriously, starting from the physical world they already trust. Longer term he is genuinely unsure. People may stick to photoreal and tweak it, or, once they are used to these rooms, they may prefer selves not so tied to physical reality. Internal tests keep surprising him: a cartoon background behind photoreal people feels normal; a photoreal person next to an expressive cartoon avatar also feels normal. He expects both styles to survive. He does not know the mix. Fridman, for now, wants a friend on a beach with a beer, no fake hat, as real as what he is looking at in this dark room.

Fridman says he is already forgetting that Zuckerberg is not real. Zuckerberg objects: he is real; this is just an avatar version of him. Then he offers the small example that makes identity slippery. He put the headset on that morning and noticed the avatar's hair was a little shorter than his physical hair. He probably needs a haircut. He happened to shave, but if he had not, the photorealistic avatar could still be clean-shaven. The avatar is realistic in the sense of what you looked like at capture. It is not necessarily temporally accurate to this moment.

Fridman pushes it. People will diet, do their hair, show up as the summer beach-body version for the scan. If others mostly meet the digital you, which one is the person—the driver or the avatar? Zuckerberg does not treat that as a brand-new glitch. Physical bodies fluctuate too. Weight moves around. He suspects a lot of people think of themselves as the best version, not the worst, and that day-to-day life maps to neither. Social scientists and psychologists, he says, are going to have a lot to understand about how this shifts our perception of ourselves and others. The discussion does not resolve. It may be the most durable question in the hour.

The first feeling Fridman had in the headset was that he would love to talk to loved ones. The next was the people who are no longer here: a father, a grandfather, a grandmother, a mother, sitting in this form, speaking. Zuckerberg's answer is not polished. There is probably a balance. If someone has lost a person and is grieving, being able to interact or relive certain memories could help. It could also become unhealthy. He is not an expert. They would have to study it.

Meta already handles death every day. Families want the profiles. There are protocols: some of the account is memorialized so it does not vanish; private pieces, including messages, are not handed over. Those practices, he thinks, will carry over. Codec Avatars still change the category. A smiling, answering double is not a static archive. It generates new lines and new faces that the dead person never produced. That is closer to continuation than to memorial. Zuckerberg then pivots to the living version of the same hunger. He does not want a single AI superintelligence. He wants tools that empower people. A creator with more demand than hours—Fridman and his audience—might build an AI version of himself, not after death but while he is here, on WhatsApp, Messenger, Instagram, and eventually as an avatar people know is an AI. They are not ready to ship the creator-builds-himself product. They are starting by releasing AIs people can talk to in other ways.

Zuckerberg's early AI personalities are a stack, not one bot. Meta AI is the plain assistant: no face, vanilla, factual, sitting in your threads. Then come business cases. A small shop does not want to staff the phones; you do not want to wait on hold. A natural-language agent that can take a product issue, a reservation, or an order is better for both sides than a website or a queue. Then come the fun ones: a jokester in a group chat, sports, fashion, trivia, better NPCs in games. Internally the most-loved prototype has been a text adventure dungeon master. They have gotten cultural figures to play some of these. Snoop Dogg as dungeon master, Zuckerberg says, is hilarious. That Snoop is an actor. He is not Snoop Dogg the person. They also made a Jane Austen and a Marcus Aurelius, and he is curious how those go.

New personas are easier because they do not have to stick to how a specific human wants to be represented. An AI that embodies a real creator has to not say things the creator would not say, and to know things the way the creator would know them. You do not program that like ordinary software. You train it. You need it predictable enough to set parameters, and even if it is not perfect, you need it generally inside those bounds. That, more than the avatar face, is why a true creator double is a much harder problem, and why he expects to start releasing that sometime next year, not this year. Characters, the assistant, games, and some small-business experiments they are ready to do this year, rolling out right after Meta Connect.

They will start with a set, then open AI Studio so that, over time, anyone can create one of these AIs almost the way they create any other user-generated content. Zuckerberg does not think anyone will see the full potential until the whole community can build. There is a lot to get right, so he wants stages. He also thinks nobody else is doing quite this combination: avatars and expressiveness, interaction across the apps, profiles, engagement on Instagram and Facebook. Fridman is already entertaining a podcast with assistant-Lex, and the surreal moment of freezing his own Codec Avatar and walking around it, like an ape seeing a mirror.

Fridman has lived through pre-internet life, voice, and video, and had assumed there was a ceiling. This session makes him think the blend of physical and digital might be the actual future. People who say the internet is not reality, that you should turn it off and go outside, might like this, because it feels real, flaws and all. Zuckerberg says a big part of the design is that the new computers should be physical. Phones and TVs leave you sitting still, and people are not meant to be that way. He and Fridman share a taste for sports and martial arts. Humans move. "We're not just like a brain and a tank." The human experience is a physical one. There is too much digital now to silo it on a small screen, and the physical is too important to spend all day at a desk.

What should be allowed in a metaverse that can do things illegal in physical life and routine in games? Zuckerberg starts from harm. Society should let people do as much as they want unless they damage others. Damage looks different digitally. He and friends get into a world and immediately shoot each other, which they would not do in a room, and even in a game lobby it is just a joke. He does not think that is a problem, because they are not causing harm in that world. Some physical restrictions should loosen. New harms appear. The one he flags is anonymity. At a restaurant, norms rest on the bill at the end, because you have one identity. If you stiff them, life is a repeat game. Online you can be anonymous and show up as someone else. The incentive to act like a good citizen drops. That causes toxic behavior, and it needs to get sorted out.

There is also the physics dividend. Flying to sit together takes hours. Putting on headsets is a teleport into a space that feels shared. That is why he thinks this gets closer to the remote-work dream: live where you want, still feel together enough to build culture, relationships, and trust—the things video calls do not fully restore. He does not pretend first principles will list every implication. They will build it a year at a time, watch what developers do, accentuate what is awesome, and try to mitigate the rest.

On their last podcast they were still arguing about whether to open-source Llama 2. Zuckerberg is glad they did. In his view the value of open-sourcing a foundation model like Llama 2 is significantly greater than the risks. They spent a lot of time on a rigorous assessment and red-teaming. Downloads and usage came in above even his optimistic guess. Llama 3 is not a ship date. There is always another model in training. Right now the priority is turning Llama 2 and the branches off it into consumer products—the AIs people will actually talk to. Llama 2 itself is infrastructure, not a consumer product. He wants hundreds of millions of people on those products, billions someday.

They will need a similar debate and red-team process for the next version, to make sure it is safe. His hope is that they can open-source that one when it is ready too. They are not close to doing it this month. They are still somewhat early. Fridman thanks him for the release and for talking in public; putting personas in people's hands, he thinks, is how a society learns the failure cases, the weird directions, how much personalization is good. Zuckerberg agrees he does not know what will work—text adventures, a comedian in a thread, historical figures—until people try. That is why the launch set is diverse, and why AI Studio comes after.

They have been talking about AI for minutes, and Fridman is still stuck on the fact that he is talking to Mark Zuckerberg, who is not in the room and feels as if he is. He has done a lot of intimate one-on-one conversations. This feels like those. For long stretches he forgets they are not in the same place. The ability to snap into that with anyone, the way we now place a shallow 2D call, is the civilizational change he cannot quite predict. Zuckerberg's thesis for the metaverse has always been presence. From his side, he is definitely here. They are not physically in the same place. Fridman is not talking to an AI. The novel part is conveying, through technology, a sense of almost physical presence. The thing that is not physically real is sharing a location. Kind of everything else is.

That is his answer to the nature of the modern real world: the physical world and the presence we feel, combined with an increasingly capable digital world. The digital side is gaining power. The physical side is a lot of what makes us human, which is why he does not want people to run away from it and spend all day on a screen. Even though the current version is a headset, he thinks this will be by far the most human and social computing platform that has ever existed. Fridman tries to shift "real" toward conscious experience rather than spacetime. Zuckerberg brings it back to the social facts that used to require a plane to Austin: intimacy, trust, nonverbal communication. They are short-cutting those laws of physics.

He closes with the company's arc. Facebook started as almost all text, one photo, lists of interests; then photos; now mostly video. The trend is richer and more vivid. Immersion and presence, he thinks, are clearly the next frontier. It happens to be incredibly hard. It needs new computing platforms and new software stacks. That, he says, is what they are here to do as a company. Fridman thanks him and the Meta engineers, still unable to measure how high his heart rate has been, and jokes that Zuckerberg might actually be on a beach. Zuckerberg is not. He is in a conference room. Fridman is, he claims, at a beach and not wearing any pants. Then they say see you later, still looking like two people in the same dark space.