A 2040 Siri That Passes Every Turing Test Still Wouldn't Be Conscious

Lex Fridman introduces Christof Koch as one of the seminal figures in the scientific study of consciousness: more than 100,000 citations, a Caltech professor from 1986 to 2013, then president and chief scientific officer of the Allen Institute for Brain Science in Seattle, and author of *Consciousness: Confessions of a Romantic Reductionist*. The conversation that follows runs from alien minds and a childhood dachshund to a one-millimeter structure under the cortex. The through line is an attempt to put "what it feels like" into science.

Asked whether there is intelligent life elsewhere, Koch says the numbers overwhelm any other bet: on the order of 10^11 galaxies, each with 10^11 to 10^12 stars, and planets around more of them than we used to think. That does not, for him, cancel the strangeness of being a subject. He still wakes up inside a world. If those creatures are products of natural evolution, he expects they will experience their worlds too. Consciousness, he says, is not a human extra. What humans uniquely do is talk about it — and even that is not universal: babies cannot, and some stroke patients cannot.

He does not remember a childhood moment of stepping outside himself. The mystery arrived as an undergraduate in physics and philosophy: nothing in the Schrödinger equation, general relativity, the periodic table, or the genome explains the jump from brain physics to feeling, and yet he wakes up every morning inside an expensive world. Consciousness, as he uses the word, is any experience — subjective feeling, phenomenology, qualia. In Thomas Nagel's famous phrase, it feels like something to be a bat, or an American, or angry, or in love, or in pain. Sitting in a chair counts. A mystical moment in deep meditation counts. Those are different forms of the same thing.

Fridman asks about the gap between a future Watson, smarter than any living human, and actual experience. Koch traces the question to Alan Turing, who approached thinking with an operational test: lock the candidate away, converse, and guess person or machine. Alexa, Siri, or Google will pass, he says, now or very soon. In Fridman's generation there will be systems that speak with complete poise, remember every email, remember everything you ever said — Samantha in the movie *Her*. The live question is whether it feels like anything to be Samantha, or to be Watson.

Two concepts have been glued together by history: intelligence, natural or artificial, and consciousness, natural or artificial. We fused them because we grew up among kin and other species, watching nervous systems scale from a worm with a few hundred neurons to a blue whale with hundreds of billions, and we watched behavior get more complex in parallel. Ancient cultures located the mind in the heart; we still say we love with all our heart, when the accurate Valentine, Koch jokes, would be a piece of lateral-hypothalamus chocolate. Now we locate it in the brain, and we infer that bigger, more complex brains host richer experience.

Engineering intelligence breaks that inference. Intelligence, however you define it, is function: adapting to a new environment, learning fast, seeing who the actors are and what happens next. Consciousness is not about completing a task. It is about being. In organisms the two rose together because evolution stacked complexity and causal power in the same tissue. In digital machines, Koch argues, we have no idea whether the intelligence we are building has anything to do with experience — not even the sign of the correlation.

Koch offers three cases anyone can check. The first is the clinic. Terri Schiavo, in 1990s Florida, had her heart stop, was reanimated, and spent fourteen years in a vegetative state. Thousands of patients like her open and close their eyes, keep a sleep-wake cycle, and make occasional movements, with no lawful mapping from what a doctor or a mother says to what the body does. Behavior is gone. In some of those people, brain-machine interfaces still suggest experience. The locked-in extreme is *The Diving Bell and the Butterfly*: a French editor, able to move only his vertical gaze, dictated an entire book. Some lose even that. The evidence, Koch says, is that they are still in there.

The second case is tonight. You will fall asleep and wake up inside a paralyzed body, perhaps flying, perhaps unsurprised to meet a childhood dog long dead. Dreams can be saturated with love and horror while the body, in REM, sends an active signal to motor neurons to freeze you — atonia. People who lack that mechanism act out their dreams, a REM sleep behavior disorder Koch calls bad juju. Behavior is locked. Experience is full enough to make you cry.

The third case is his own. In Singapore he climbed into a flotation tank: body-temperature water, Epsom salt, total dark, soundproof. Naked, he lost the edges of his body, then sound, then photons, then the sense of time passing. Early on it is spooky, and the mind volunteers boredom. If you stop actively thinking, what remains is a bodyless, timeless, soundless, mindless something that is nevertheless awake. No computation, no remembering, no planning — and fully conscious. Evolution did not build us for flotation tanks, so asking what adaptive function this has is the kind of why-question biologists, he says, are bad at. We cannot even say why we have two eyes rather than three. The dissociation itself is the data: function can go to zero while being does not.

Fridman wants a scientific test: is consciousness present in this entity? In humans there is now a procedure nicknamed zap-and-zip. You ping the cortex with transcranial magnetic stimulation, watch the electrical reverberation on EEG, and score the complexity of the echo. In the lab it separates waking people from anesthetized or deeply asleep ones with essentially perfect accuracy, and the same method can be adapted to monkeys, dogs, and mice. Send that magnetic pulse into an iPhone and you will just break it. The biological test does not travel.

So we need a theory. Intuition says that whoever talks is conscious. Babies do not talk; dreamers cannot; vegetative patients cannot. Lean on intuition and you get those cases wrong. Descartes's dog could yelp when hit and still, in that older story, lack the special soul. Koch no longer buys the soul story, which means he needs an account of what it is about a piece of highly excitable matter — brain or computer — that gives rise to experience. Other people remain an inference: similar brains, similar scanner maps, similar reports about the taste of a madeleine. A future Siri that says "of course I'm conscious" does not give you that abductive ladder. You need a theory that applies to any system.

The theory he bets on is integrated information theory, IIT: consciousness tracks the irreducible causal power a system has upon itself, a quantity that can in principle be computed for any physical substrate. Panpsychism, the older sibling of that idea, comes in grades. The extreme version says everything is ensouled — rocks, forests, iPhones. He finds that unrealistic. The milder version says all biology, from a single cell to a giant sequoia, feels like something; that he takes more seriously. Bertrand Russell's line, often called Russellian or neutral monism, is that physics is superb at third-person relations — charge, curvature, quarks — and silent on what the thing is from the inside. Your experience, Koch says, is what the physics of your brain, especially cortex, feels like from in here. A paramecium is dumb, but it is also thousands of proteins and enormous molecular complexity that no computer has accurately simulated. It probably has no inner voice and no plans. It may still feel like a tiny bit of something.

Asked to draw lines among life, intelligence, and consciousness, Koch reaches back to Aristotle, in his view the most important scientist-philosopher in the Western tradition. Soul, for Aristotle, is the form of a thing, in three layers. All living things have a vegetative soul, the life principle — today we would say biochemistry and nonlinear thermodynamics. Animals add a sensitive or appetitive soul: they see, smell, want to eat, want to reproduce. Only humans have a rational soul. Christianity later made that layer immortal and the channel to God, a reading Koch doubts Aristotle himself held.

The modern versions of those three questions are still unfinished. We do not even have a clean definition of death. It used to be uncontroversial: you stopped breathing, the heart stopped, you were dead. Ventilators and pacemakers forced conference after conference on the legal and medical line, with death defined as the end of life and life defined as whatever precedes death. Intelligence has no legal definition either, only IQ tests and scoreboards. We have started building it in a narrow sense — AlphaGo, Watson, self-driving cars — and some people are aiming at artificial general intelligence, roughly the ability to learn and adapt. The distance from that to experience, Koch thinks, is huge. A machine with AGI may or may not be conscious. Nothing in the engineering automatically delivers the inner light.

In organisms, Koch says, intelligence and consciousness go hand in hand because evolution built complexity, and IIT ties that complexity to causal power upon itself. Digital machines uncouple the pair. Asked point blank about Alexa 20.0 in the year 2040, a system that passes every Turing test you can invent, he says no — even if she claims she just woke up and feels great. The more radical version of the thought experiment grants Henry Markram's Blue Brain or Human Brain Project complete success: in ten years, a perfect simulation of every neuron and synapse, Broca's area included, that says "Hi, I just woke up, I feel great." That simulation, Koch says, would not be conscious. It would, if done properly, have the intelligence of the person being simulated. Simulating intelligence is not the same as having experience.

The metaphor he likes for engineers and physicists is gravity. You can write down Einstein's field equations, run them on a laptop, and correctly predict that the black hole at the center of the galaxy is massive enough to twist spacetime so that light cannot leave. Why, then, does the computer not suck you in? It simulates gravity. It does not have gravity's causal power. A weather model does not get the room wet. The gap is not a matter of resolution. To build artificial consciousness, on this view, you would need neuromorphic hardware whose causal structure sits at the same level as the brain's, not a digital clock chopping computation into discrete ticks. The consequence he wants stated clearly: consciousness is not a necessary ingredient of human-level machine intelligence. Biology stacked them. Software need not.

Fridman pushes on whether people will call a dazzling 2040 Siri intelligent, and whether connecting with humans requires something like consciousness. Koch thinks we can get high-level functional, even fluid, intelligence in machines over the next thirty to fifty years. He does not think that will make us happy in the sense of having created a mind. Natural-language mastery can impress. It is still, on his split, a function.

Fridman turns to ethics: a superintelligence that does not share our values. Koch goes to empathy, unpacking the Greek *empatheia* as feeling-with. Why do most people find abusing a dog abhorrent? Not because the dog is a spouse or a child or a person. Because something in us automatically hurts with it. If *Homo sapiens* is going to build very powerful general intelligence, he thinks it may be in our long-term survival interest that the system have that response — that it not simply exterminate humanity as one more unconstrained optimization.

Are fear and suffering required for consciousness? In principle, no. Classic experiments put electrodes in a rat's medial hypothalamus, the pleasure center; the animal presses the lever to the exclusion of food, drink, and sex. A system of only pleasure is logically possible, even if the existence would be terrible. A real creature that has to survive is another matter. You want fear of extinction, and you want positive appetitive states so that training has something to grab. Variety of experience is a design choice for engineered minds, not a metaphysical requirement.

Koch grew up in a devout Roman Catholic family. He no longer believes in the God he was taught, nor in a reunion in everlasting bliss. He does not see the evidence. The night, he says, is large and full of wonders; we do not understand more than four percent of the universe, dark matter and dark energy included — "maybe it's lost socks." His spiritual sentiment is now closer to some form of Buddhism, minus reincarnation, for which he also sees no evidence. What impressed him in meetings with the Dalai Lama was the repeated emphasis on reducing the suffering of all creatures, graded by capacity, not only of humans. That stance, he notes, was in the tradition from the beginning.

He will not overclaim. Meditation techniques aim at what some Buddhists call a more fundamental aspect of reality; he is not sure it is more fundamental so much as the inside of physics. The only things he has direct access to are the physical and the conscious. Everything you know about atoms, superstrings, and molecules is downstream. The first and only world you are acquainted with is this one, populated with images, sounds, and touch.

Rock climbing and daily cycling get him into the zone, a state people chase because it is strangely addictive. The nagging inner voice — pay your taxes, that fight with your ex — goes quiet. You are fully out in the world, all action. The flotation tank was the opposite, all non-action. In both, he thinks you touch a basic part of conscious existence, the root of being, which is a different thing from intelligence. The simulation hypothesis he files with an older idea that occupied scholars for centuries: that we exist in the mind of God. It is a modern version, equally untestable, fine if people enjoy talking about it, not useful to him.

Physics, until quantum mechanics, looked deterministic; quantum mechanics makes some things in principle unpredictable, a fact Koch prefers to the romance of a universe whose initial conditions are simply playing out. At the level of a life, we are constrained by physics, by the past, by parents, by the environment, and by a century of experiments showing how easy we are to nudge. When the decision is one you actually deliberate — whether to marry, which school, which job, whether to cheat on taxes — and you bring your entire conscious being to the question, lay out the conditions, and choose, that, he says, is as free as you can ever be. Free will is not a will that can do anything it wants. That does not exist.

Literature is how he widens the map of experience beyond the lab: a face flashed on a screen, a button press. He blogs books — Bulgakov, Murakami, Neil Gaiman, Carl Sagan — because other people's states are otherwise expensive to sample, and because old books show that every culture in its heyday thinks it knows it all. Nietzsche was, for him, an early discoverer of the continent of the unconscious, a little before Freud, stripping the mask of honor and charity. Freud he calls a genius, misguided toward the end, who began as a neuroscientist working on lampreys and contributing to the neuron doctrine. After a relationship breaks, love, hate, lust, and anger mix in places consciousness cannot inspect — the amygdala and other basements. You can tell a story to a therapist. You cannot confirm it. You simply do not have access, and those regions are powerful.

Fridman asks whether that opacity is a bug. Koch says feature. If every movement and every emotion required conscious approval, we would not be here. You must be conscious to learn typing or riding a bike; then you hand the skill to basal ganglia and striatum. Francis Crick and Koch named those specialists zombie agents: they run the hands while consciousness worries about the abstract sense of the text. Traumatic memories you cannot delete are probably functional too. Extreme abuse can produce clinical dissociation; ordinary people cannot wipe the file. If you had a switch, you would likely erase past the point of usefulness.

Fridman asks about scale: we default to human clocks and human bodies. What about the very large, the very small, the very slow? Koch starts with bees. Most people see bugs. Bees can recognize individual human faces. When they swarm, they make a decision as agonizing, he says, as buying a house: this tree or that wall. Scout bees inspect sites, return, and dance for days to recruit votes; then the whole swarm flies to the location the scouts have already agreed among themselves. Their circuits are an order of magnitude denser than ours, with under a million neurons and workers that live about two months. Under IIT they almost certainly experience something. It will not be like ours.

Then Fred Hoyle's 1957 novel *The Black Cloud*: a cloud between Earth and the sun, global cooling, and eventually a radio-telescope conversation with an intelligent entity that is persuaded to leave. If that cloud's integrated-information maximum sat at the timescale of a month, one of its moments would be a month of ours. Forms of consciousness may live beside us, unrecognized, because they are offset in scale. Stanisław Lem's *The Invincible* pushes the same alienness: a planet whose inhabitants are dead, machines in Darwinian evolution, a winning form that is a cloud of hexagonal cellular automata — inert on the ground until crisis, then assembled into a hyperintelligent net humans can neither understand nor fight except by nuking the planet or leaving. "Is it conscious?" and "Can I know that it is?" are, Koch insists, different questions — ontology versus epistemology. A multiverse might exist with no information crossing; unmeasurability is not nonexistence.

The distinction is already practical. Labs at Harvard and Stanford grow mini organoids: stem cells from under an arm, a few transcription factors, a few millimeters of tissue, on the order of half a million neurons that look like nerve cells. They may be beginning to feel like something. We cannot communicate with them. Ethics arrives early, with the two questions still split: whether there is experience, and how we would know.

A young researcher who wants to understand or build human-level intelligence and consciousness should not pick one department, Koch says. The only known high-level system is *Homo sapiens*, so study humans. His map is cognitive neuroscience at one end, philosophy of mind at the other, computer science in the middle. The original ideas, he argues, grew out of neuroscience: neural networks, reinforcement learning, Minsky's early machines, and the Hubel and Wiesel visual-cortex experiments that grew into multilayer nets. Convolutional networks can do a lot and still not understand. Show a person one picture of someone lifting a wallet from a purse and they see a pickpocket. A machine says man, woman, purse, unless you have fed it a hundred thousand pickpocket images. To get that one-shot grasp, some people argue, AI has to go back through psychology to the brain. The exciting place to stand is the seam: cognitive science, neuroscience, AI, computer science, and philosophy of mind.

Asked for the near-term mystery he most wants to crack, he names the claustrum: a structure under the cortex, beneath the insula, about a millimeter thick, left and right, buried in white-matter wiring and hard to image. Its peculiarity is that it connects to every cortical region. Francis Crick's last paper was about it; he dictated corrections from his hospital bed on the day he died. The hypothesis is conductor-in-the-orchestra. Cortex has many players — motion, theory of mind, social inference, color, hearing. Consciousness has to bind them into one package. The claustrum has far fewer neurons than cortex and talks to all of it, which makes it a candidate integrator.

They are testing that in mice with a newly labeled cell type they nicknamed crown of thorns: the most widespread connections Koch has seen, a tiny soma and a huge axonal tree covering both hemispheres. Optogenetics can turn those neurons on and off and watch the animal's behavior change. Fridman boils it down: this is where the parts become the whole. Koch likes the phrasing. It is one of the places, he says, where separate modules become a single conscious experience.