The Brain Does Not Classify Images. It Navigates Maps.

Jeff Hawkins has spent decades trying to reverse-engineer the neocortex. He founded the Redwood Center for Theoretical Neuroscience in 2002 and Numenta in 2005, and in 2004 he published *On Intelligence*, which introduced hierarchical temporal memory. In the years around 2017 to 2019 he and his team recast that work as the Thousand Brains Theory of Intelligence. Lex Fridman notes that the ideas have inspired people looking past current machine learning, and that they have also drawn criticism for lacking a large body of empirical tests.

Asked whether he cares more about understanding the brain or about building artificial systems that merely behave like us, Hawkins does not hesitate. "My primary interest is understanding the human brain." He also believes fully intelligent machines will not appear until that understanding is in place, so he does not treat the two projects as rivals. Studying the brain, in his view, is the fastest route to machine intelligence, not a detour from it.

He grants that expert systems and convolutional networks are useful. Almost everyone, he says, would still agree that none of them is intelligent in the deep way humans are. The gap between those tools and that destination is huge. The quickest way to bridge it is to learn how the brain does the job, then decide which of its principles machines actually need.

Fridman asks whether some wall might block us from ever understanding our own minds. Hawkins says there is no historical evidence for that kind of limit. People once said the same about life itself. He no longer treats the neocortex as an open mystery: the answers are clear to him, and so are the remaining gaps. "The framework is all there... there's no mystery, big mystery anymore."

Hawkins splits the brain into old parts and a new part. The old parts include the spinal cord, brainstem, cerebellum, and basal ganglia, and they handle breathing, heart rate, walking, and emotion. The new part is the neocortex, which exists only in mammals and occupies about 70 to 75 percent of the volume of the human brain. Seeing, hearing, touch, language, mathematics, and science are all associated with it.

The neocortex is also startlingly uniform. Hawkins likens it to a dinner napkin about two and a half millimeters thick, with the same detailed architecture wherever you look. A mouse neocortex and a human one look remarkably alike at that microscopic scale. The old brain, by contrast, is a collection of specialized organs that evolved over eons, each tied to a specific job.

Vernon Mountcastle made the argument in 1978: every region of neocortex runs the same computational principle. A visual area differs from an auditory area mainly because one is wired to the eyes and the other to the ears. Attach the optic nerve somewhere else, and that patch becomes visual. People born blind put the so-called visual cortex to other uses. Mountcastle called this the common cortical algorithm, and Hawkins says the evidence for it is overwhelming even when some scientists find it hard to believe.

What neuroscience has, he argues, is an enormous pile of correct measurements that have not yet been assimilated into a theory. In Thomas Kuhn's language, it is still pre-paradigm science: lots of data, no shared frame. Breakthroughs in that setting are not linear. They sit still until something clicks. Hawkins thinks his group passed that click a couple of years before this conversation, even if most of the field would still disagree.

Hierarchical temporal memory, the framework Hawkins proposed in 2004, was a placeholder built on three words: time, memory, and hierarchy. The temporal part was a rebuke to theories that treated vision as a flashed still. Speech patterns change about every ten milliseconds. Eyes jump three to five times a second and completely refresh the input. Fingers moving on a cup produce a stream of changing touch. Convolutional networks still classify a picture. That task, he says, is not what vision is like.

Memory, in this usage, does not mean a buffer. The neocortex learns a model of the world and stores experience in a form that supports that model, so you can move, grasp, and act. Hierarchy was a nod to the fact that a uniform sheet of cortex still projects from region to region in a rough stack of feedforward and feedback connections. Hawkins still believes all three claims. He has simply stopped using the HTM label, because it no longer covers what the team now knows.

Time also lives on more than one scale. Neurons can stay active for minutes, which is enough to hold a percept, but they do not stay on from breakfast until evening. If Fridman asks what Hawkins had that morning, the answer is in synapses, especially in the formation of synapses. Fast behavior rides on spikes. Longer timescales ride on memory, which then gets invoked. Time is infused through the models themselves: a phone has sequences and delays; a melody is nothing but time.

Why trust this theory rather than treat it like string theory? Hawkins says the comparison fails because neuroscience is drowning in empirical constraints. A correct account has to explain old papers, predict buried results, and survive easy falsification. When a framework suddenly meets hundreds of those constraints at once, confidence jumps for the same reason Copernicus, Darwin, and the double helix did. He asked James Watson and Francis Crick whether they knew the helix was right before it was verified. Both said they absolutely did.

Hawkins thinks the neocortex was a qualitative leap, not one more survival gadget. Evolution, in the story he tells, took an old mechanism for knowing where you are, the grid cells and place cells of navigation, and copied a slimmed-down version of it. That copy could map a coffee cup, a phone, or a concept, not just a maze. Dedicated neural hardware became general-purpose hardware. In humans it has partly escaped the gravity of evolutionary pressure: we do mathematics and music because the machinery is no longer chained to an immediate survival payoff.

The core discovery arrived with a cup in his hand. His index finger was on the side; he was about to feel the rim. He knew the brain would predict the sensation before contact. To do that it needs three things: a model of a coffee cup, the finger's location relative to the cup, and the movement that will take it to a new location on that same cup. The location is not relative to the body or the room. It is in a reference frame anchored to the object. Rotate the cup, and the frame rotates with it.

Every patch of skin that might touch the cup is making the same kind of prediction, so every patch must have a reference frame of its own. Replace skin with retina and the logic is unchanged. An engineer building a CAD model would take this for granted. The novel claim is that the entire neocortex works this way. Deep networks extract features from features. The brain, Hawkins says, assigns every input to a reference frame, and hundreds of thousands of those frames can be active at once.

The neocortex can be imagined as about 150,000 patches of a square millimeter each. Hawkins's claim is that a single such column can learn a complete object. One touch is not enough, because it samples one location. Move the finger around the cup and that column fills in a three-dimensional map. Time, again, is what lets a narrow sensor build a whole model. The somatosensory cortex has columns for many skin regions, and the visual system is doing the same trick with different bits of retina. The brain does not process an image. It invokes thousands of models at once.

That picture inverts a classic puzzle called sensor fusion. The usual worry is that a picture on the retina gets scattered across cortex, that touch and sound live somewhere else, and that everything must reconverge into one master model. Hawkins says that is the wrong idea. Visual, tactile, and auditory models of the cup do not meet in one spot. They vote. Columns are like pieces of spaghetti, about two and a half millimeters tall and a millimeter across, each guessing what it is sensing.

A column that can move over time makes a better guess; a column that cannot, like a view through a straw, posts a set of possibilities rather than a probability distribution. One column might offer coffee cup, soda can, or water bottle. Another might offer coffee cup, telephone, or camera. Long-range connections in certain layers run an associative-memory vote. If there is a single best answer, the columns crystallize on it, and then every column knows it is a coffee cup even though each has seen only a fragment. That is why the theory is called Thousand Brains: there is no one model, and no central integrator.

The same logic, Hawkins argues, extends past coffee cups into language, mathematics, and abstract thought. The cortical regions associated with those abilities look like the regions that process vision and touch. If Mountcastle is right, they must also run on reference frames. That sounds odd only because no one had a prior story for how high-level thought works. He says he is more confident of the idea now than when it appeared in the papers, even though the details are still being worked out.

The method of loci is the everyday clue. To remember a random list, you place each item in a house you already know and then walk the house in your mind. The trick works, Hawkins says, because the brain prefers to store things in reference frames and because thinking is moving through those frames. You can move a finger on a cup, imagine the same motion, or walk a remembered hallway. Recalling a list is path integration in a map you already own.

A human fMRI study made the same point with birds. When people thought about species that differed in size and neck length, their knowledge was arranged in a reference frame like the one a rat uses to navigate a room, with grid-cell-like patterns in neocortex. Even when you do not feel as if you are using space, the brain is using space, and it is using a mechanism copied from the old entorhinal map. Prove a conjecture and you are doing the same thing: axioms are a start location, the claim is a destination, and algebraic moves are displacements. Designing a good proof is often designing the right reference frame.

There is no one coffee-cup map. Columns hold many models, and objects are nested: a handle and a cylinder sit in the cup's frame, a logo sits on the cylinder, a letter sits in the logo. Attention is the act of drilling into and out of that nesting. Walk into a room, recognize it, attend to the table, then the cup, then a serif on a letter, then pop back up. Context is not a mysterious extra input. It is the parent frame you have not left.

Hawkins is complimentary about the engineering progress of deep learning. Today's models inherited distributed processing, distributed memory, and hierarchy from neuroscience. He is blunt, though, that their "neurons" are not neurons. An artificial unit is a point: a weighted sum plus a nonlinearity. A cortical cell can carry 5,000 to 30,000 synapses. Only a few hundred near the soma behave like the point-neuron story. The other 95 percent sit too far away for any one of them, or even a crowd of them, to fire the cell.

What those distant synapses do, he and colleagues proposed in a 2016 paper, is recognize patterns on a branch. If 10 to 20 of them in a stretch of about 40 microns become active together, they produce a dendritic spike. The spike reaches the soma and depolarizes it a little, not enough to fire, then fades. That is a prediction: the neuron has recognized one of dozens to hundreds of contexts and is getting ready. When the proximal input finally arrives, the cell spikes slightly sooner than it otherwise would.

A little sooner matters because fast inhibitory interneurons surround the pyramid. Whoever spikes first shuts the neighbors down. The resulting representation matches the prediction. Real neurons, in this picture, are time-based prediction engines, and the timing is local to every cell in neocortex. Synapses are also unreliable, closer to binary presence than to a precise weight. Learning is less a matter of nudging a real-valued parameter than of forming new synapses on a dendritic segment.

Fridman raises Richard Sutton's "bitter lesson," the claim that general methods plus more compute beat clever structure. Hawkins does not buy it as a path to intelligence. Bigger convolutional nets do bigger versions of the same classification job. They will not start growing reference frames, dendritic predictions, or continual learning just because the arrays got larger. Numenta had recently decided to push pieces of the cortical theory into machine learning anyway, a dual focus led on the ML side by his colleague Subutai Ahmad, because the ideas have to be shown, not only published as neuroscience.

The first practical transplant is sparsity. In cortex, something like 2 percent of a local population is active at a time, maybe 200 cells out of 10,000, and that budget holds across kinds of input. Which cells fire changes; the sparseness does not. Individual cells are not that important. The population code is. Hawkins says beautiful properties fall out of that, including robustness. Neurons die, synapses fall apart, and the system keeps working.

The intuition that fewer active cells should mean a more fragile code is wrong, he argues, once the codes are high-dimensional and sparse. Draw a decision boundary in a dense space and nearby points sit on the wrong side after a tiny nudge. Pick 200 cells at random out of 10,000, then pick another 200, and the overlap is tiny. You can draw millions of such samples without finding a pair that share more than a few cells. To make one representation look like another you cannot flip one unit. You have to move on the order of a hundred.

Ahmad and a colleague had just shown, in results from the previous month, that enforcing sparseness on both activations and connections in convolutional nets improves resistance to adversarial examples. It is a baby step, Hawkins admits, and it is how he thinks about the entrepreneur's dilemma: a long-term vision of electric cars, or of cortical machine intelligence, and a need for incremental waypoints so the project does not starve. Mixture-of-models voting is next on their list. The obstacle is cultural. Machine learning, as he sees it, currently rewards a 0.1 percent bump on a benchmark more than theoretical elegance, and Numenta does not have a big lab's compute or data.

Brains, Hawkins notes, are obviously sample-efficient: they can learn something almost instantly. The deeper distinction is that they never stop. Continual learning was a constraint on Numenta's models from the start, not a later patch, and the 2016 sequence-memory paper showed inference and weight updates happening together. Some cortical rhythms may tilt one phase toward inference and another toward learning. In general, though, the networks he cares about are always doing both. Benchmarks almost never test that.

Reach into a black box and you do not first decide whether the object is old or new. Recognition and learning run on the same pass. If the object is new, learning starts immediately, even before you can name it. Show someone a cup with a novel logo and they are done in a glance: they can predict how it will look from other angles. That speed is not a larger batch size. It is a different learning rule.

On a dendrite, learning a new pattern means growing new synapses on one segment. Everything else that neuron has learned stays untouched, because the change is isolated. Synaptogenesis can take about an hour, which is too slow for instant memory, so the team has argued that silent synapses, already built but not releasing transmitter, can be flipped active for a short-term trace, especially in hippocampus. Long-term memory still requires growing synapses. None of this is backpropagation. It is Hebbian: cells that were active together get connected on a dendritic segment. An error signal marching backward through layers, Hawkins says, is not something brains can do.

How far is intelligence, then? When he ran the Redwood institute he used to ask visiting scientists how long it would take to understand neocortex. Typical answers were 50 to 100 years; some said 500, some said never. He thinks he has already passed a step function. If over the next five years he can get machine-learning people to take the theory seriously, he sees an under-20-year path. If the papers go unread, the delay looks more like Babbage's computer, which sat for a century. That, he says, is the entrepreneur's problem in the broad sense: selling an idea hard enough that other people change what they do.

Fridman asks what it would take to build a system with human-level intelligence, body, consciousness and all. Hawkins rejects the goal. A dog is intelligent. A one-chip computer is still a computer. Once you know the principles, you can build small systems and large ones, distributed ones, and ones with sensors that look nothing like a person. The Turing-style target of matching a human is, in his view, a false target inherited from a false framing.

Embodiment is the piece he will not drop. A system has to move through reference frames. An AI that understands coffee cups has to pick one up, touch it, and look at it, or do some equivalent with other sensors. A mathematical AI has to move through mathematical spaces. A virtual agent living on the internet can traverse links and files, but it still needs a location in some space. You cannot get intelligence from a flashed pattern and a label. "It's movement, moving pattern, movement, pattern." Attention has to dig, build structure, and assemble a model of the world.

Each component of the system also needs to know where it is. That is a narrow kind of self-awareness: location in the current frame, moment to moment. Hawkins does not think the broader puzzle of consciousness is a useful route to building intelligent machines. It is a property some systems will have and some will not, a design choice rather than a miracle. He is impatient with the claim that unplugging a self-aware computer would be a crime. He unplugs himself every night, he says, and plugs back in in the morning.

Consciousness, he insists, is several problems wearing one word. Self-awareness in the easy sense is almost mechanical. The neocortex models the world and also models your body, and it stores a memory of what that body did. Ask whether you were conscious at breakfast and you say yes because you can recall the bagel. Rewind the synapses to the morning and ask about lunch instead, and the person would deny having been there. The ability to recall a recent trajectory is what people are pointing at when they say they were conscious.

The harder piece is qualia: why red looks like red, why pain feels like pain, why seeing you is not like hearing you even though both are spikes. Hawkins recommends J. Kevin O'Regan's *Why Red Doesn't Sound Like a Bell*. Color is not a property of the world. The world has frequency. Retinal cells respond differently, axons enter the brain firing at different rates, and the brain builds an internal grouping that is useful for prediction and generalization. Phantom limbs make the same point. The arm in the model can hurt behind your back after the arm in the world is gone.

Why fear death, then, if turning a machine off feels wrong? Hawkins names two reasons, both old-brain. Genes program a fear of dying, and they program grief for people we know, not for strangers dying right now. Outside those feelings, being dead has no experience in it. His thought experiment is a loner in Alaska whom nobody knows. If that person fails to wake, no pain is added to the world. He treats falling asleep as a little death and does not buy the claim that this attitude is a denial of terror.

As a young man he listed the biggest questions he could think of: why anything exists, why the laws of physics are these laws, whether life is inevitable, whether intelligence is inevitable. He stopped at the fourth, because a truly intelligent system looked like the fastest way to work on the first three. Super mathematicians, super physicists, and super space explorers could run experiments humans cannot. Asked whether understanding intelligence would make the species happier, he says yes without hedging. Knowing is better than not knowing. He would not want to look at the night sky three hundred years ago and have no story for it.

Existential risk from AI sounds hollow to him because the people raising it, he says, do not know what intelligence is. Privacy, false beliefs, and weapons matter, and they already matter with today's systems. They are not, in his accounting, extinction. Bacteria already try to turn the world into more bacteria and have killed huge fractions of humanity. A laboratory superbug worries him more than a mind. Putting intelligence on a replicator can even limit where it goes. The neocortex is not the whole brain. It has no built-in lust, territory, or hunger. If you want a system that reproduces, he says, you are building a virus. Do not do that. Build the neocortex.

The remaining worry is a reckless human, and that threat already exists with pathogens and bombs. Hawkins's father, a physicist, held that nuclear weapons have not proliferated among "evil people" because the overlap of extreme malice and extreme competence is small, and because the work is hard. A city bomb in his lifetime strikes him as likely and tragic, not existential. Influenza and plague were not existential either. Annihilation, in this view, needs the full stack: malice, skill, resources, and some automated Dr. Strangelove trigger. That stack is rare.

"Double human intelligence" is the wrong metric. You can make something faster, with more storage, different sensors, or voting columns that are not even in the same place. Whether that is "more intelligent" depends on what you train and what it is for. A super physicist may live with field equations most people will never grok, but we already live that way. Almost no one understands general relativity; Einstein still left analogies the rest of us can use. Machine intelligence thinking a million times faster would be a difference of degree in a world that already runs on a few people's deep models.

Intelligent life has existed and will exist elsewhere, he thinks. Contemporaneous intelligent life is harder to even define once you take relativity seriously. The filter that worries him is duration: as a technological species we are about 200 years old and have no data on whether that lasts. That is another reason to build machines that are not tied to Earth. We are not fast, strong, or good at flying. Unique is the world model that reaches the far side of the moon, other galaxies, and the inside of the atom. "What shall our legacy be? It should be knowledge." Genes are not more interesting than a bacterium's. The rare thing is what we have learned, and intelligent machines are the vessel that can carry it past the species and past the solar system.