Karpathy Calls Trained Neural Nets Complicated Alien Artifacts

Lex Fridman introduces Andrej Karpathy as previously the director of AI at Tesla, and before that at OpenAI and Stanford — one of the scientists, engineers, and educators who has actually had to make large networks work. After the sponsor reads, the first question is almost childlike: what is a neural network, and why does it learn so surprisingly well? Karpathy's first answer is deflating on purpose. Originally it was a mathematical abstraction of the brain. At the end of the day it is a mathematical expression, and a fairly simple one.

That choice of metaphor is the episode's spine. Later he will refuse to lean on the brain at all, and will name a trained network a complicated alien artifact. The person saying it is not a commentator. He has spent a career inside the training loop. The stance is a kind of hygiene: reverence for biology, sobriety about the artifact, and a willingness to follow that sobriety into origin-of-life chemistry, the Fermi paradox, and whether physics itself has bugs.

Pressed on the mechanics, Karpathy describes a sequence of matrix multiplies — dot products — with a few nonlinearities thrown in. The expression is full of knobs, loosely analogous to synapses, and the job is to find the setting that classifies an image or does whatever you asked. There is, he says, not too much mystery if you refuse to load the brain with extra meaning. It is a complicated mathematical expression whose knobs need a proper setting.

Fridman pushes back with poetry: letters and spaces, arranged, can still make you feel something. A large enough pile of knobs, in a brain or a computer, surprises us. Karpathy grants the point and immediately admits he is underselling the networks. When they are large enough and trained on a hard enough problem — next-word prediction on a massive internet dataset is his example — they take on pretty surprising, magical properties. The simple formalism yields more than the formalism seems to contain.

His best intuition is not mysticism. We are fairly good at optimizing these nets, and a hard problem forces them to learn very interesting solutions. Those solutions have the emergent properties. "There is wisdom and knowledge in the knobs," he says; the representation lives in their values. Fridman asks whether a large number of knobs can really hold deep wisdom about the data. Karpathy's answer is almost a shrug: it is a lot of knobs. He is, by his own account, underselling and overselling at once — underselling because the magic is real, overselling because it is slightly unhinged that this math can produce it.

The concrete example is GPT: a next-word prediction network. Consume a sequence of words from the internet, try to predict the next one. Once the dataset is large enough, you can prompt the net in arbitrary ways, make it look as if you are posing a math problem, and it will continue what it thinks the solution is. Very often those continuations look remarkably consistent, and potentially correct. Karpathy keeps the hedge. He does not say the model understands. He says it looks consistent. It might be right.

Fridman tries to turn the same machinery on Karpathy himself. When your brain is talking, is it doing next-word prediction, or something more interesting? Karpathy plays along: it is some kind of generative model, GPT-like, prompted by the host, remixing common phrases into a sentence that is still pretty unique. Magnus Carlsen's rating joke wanders through. The serious claim underneath the banter is that "predict the next token" stops looking dumb once the data and the model are large. To get the next word right, the network has to build some internal picture of the world's regularities. You do not have to instruct it to understand. The loss function does that work.

Does he still draw wisdom from biological neural nets? He is much more hesitant with brain analogies than the field often is. Neural networks started as brain inspiration; nobody denies that. The artifact you hold after training, though, was arrived at by a very different optimization than the one that produced the brain. "I kind of think of it as a very complicated alien artifact," he says. "It's something different." There was no multi-agent self-play, no evolution. There was a compression objective on a massive amount of data.

Biological networks, in his telling, are agents in a multi-agent self-play system that has been running for a very long time. Evolution did find prediction useful, so the brain almost certainly contains something that looks like a predictive model. It also contains a pile of other gadgets: value functions, ancient nuclei, the whole survival-and-reproduction stack. And the entire organism is built from a single cell. The code is in the DNA. Head and legs just appear. Karpathy, who spends his days on gradient descent, calls embryogenesis totally crazy.

Asked to pick the most interesting invention in the history of life — origin, eukaryotes, mammals, Homo sapiens, technological intelligence — he will not name a single unique piece. You almost have to start at Earth's formation, Jupiter, the Moon, the habitable zone, an active planet turning over material, then abiogenesis. As an AI researcher, the last piece is the one he does not intuitively get: lots of animals do not build technological society, we do, it happened very quickly and very recently, and he does not fully understand that jump.

How many intelligent alien civilizations are out there? Karpathy has been preoccupied with the Fermi paradox, and the reason he is reading origin-of-life chemistry is to guess how common technological societies should be. The more he studies it, the more he thinks there should be quite a few. He used to treat abiogenesis as a magical rare event. Nick Lane's *The Vital Question* and *Life Ascending* walked him through alkaline vents, alkaline water mixing with the ocean, proton gradients, porous pockets that concentrate chemistry. Step by step, geology to primitive life starts to feel plausible on other worlds.

Timing seals it for him. Life arose a few hundred million years after it was possible, which suggests origin is not the limiting variable. If there are no major drop-offs after that, life should be fairly common, and the only reconciliation with the silence is that we cannot see it. Fridman notes that Nick Lane and many biologists think the hardest jump is bacteria to complex eukaryotic life, the endosymbiosis story. Karpathy knows they know more biology than he does and still finds that crazy. How many single cells, how much time, constrained resources, battling it out — "it's like how to move from a hello world program to invent a function." A billion years is not even that long.

If life is common, why the quiet? Karpathy is not an expert on SETI and says so. He is still suspicious of our instruments. Radio power drops as one over r squared. He remembers reading that Earth's own broadcasts would not be measurable by Earth's own devices except at tiny distances — on the order of a tenth of a year away. You would need a targeted, massive-power transmission aimed at someone to be picked up across interstellar space. Our stethoscope is weak.

The harder barrier is getting there. Why no von Neumann probes flooding the galaxy? Near the speed of light, the interstellar medium is a shooting gallery. Tiny hydrogen atoms and dust grains carry massive kinetic energy; you need shielding; cosmic radiation is brutal. Maybe you simply have to go very slowly. He is suspicious of our ability to measure life, and suspicious of any civilization's ability to permeate the galaxy. That is currently his way around Fermi. Fridman finds it mind-blowing either way: trillions of pockets, some close enough to see each other, and once you see definite complex life you would be intensely, aggressively motivated to find out what it is.

Fridman puts him in two hats at once: scientist and President of the United States. First contact — meet them, or defend? Karpathy will not take the binary. Look at how we treat primitive life on Earth. We share space with ants, we are hesitant to smash them, we try to protect them by default, because they are amazing dynamical systems that took a long time to evolve. He likes those systems. He would preserve them if he could afford to, and he would like to think galactic actors would see a few-billion-year story the same way. You would need a very good reason to destroy it. We are not in direct competition with ant farms; there are plenty of resources.

From a scientific angle, though, you might poke. Alien scientists might want to learn what happened here. Today's biosphere is a snapshot of a huge amount of computation. Would he, given the power, pick an Earth-like planet with the chemical prerequisites, seed it with life, and watch? "Wouldn't you 100 percent do that?" It would be a hell of a television show and a real experiment; running evolution might be the efficient way to see which branches life can take. The thought that we might be inside that experiment makes him feel weird. Then he shrugs: maybe everything is an experiment, and it does not change the day.

He remains suspicious of deliberate panspermia as an explanation of us. He does not see divine intervention in the historical record. Lane's story of life arising on Earth uniquely still hangs together; he does not need a more exotic account. Fridman notes that NPCs in a video game would not observe divine intervention either. Karpathy allows that current NPCs are dumb, and that once they run on GPTs they might start saying this is really suspicious. For now he will not reach for the designer.

Karpathy once tweeted that if you bombard Earth with photons for a while, it can emit a rooster. Fridman wants the Hitchhiker's summary of the planet once the computation is done — *Mostly Harmless*, or some other sentence. There has to be an ending. Self-replicating systems arise from the dynamics, perpetuate, become more complex, become conscious, build a society. It feels to Karpathy like a deterministic wave that happens on any sufficiently well-arranged world. Beautiful, and it ends somehow. He does not know the terminating conditions. He does know the trend line, and that we are on it.

Humans, he says, are often described as a biological bootloader for AIs, and the insult is partly earned. We compute, we love, and we are extremely inefficient. We talk through audio, manipulating something like seven symbols serially, vocal cords, over multiple seconds — embarrassing once you step down to the frequencies computers can use. Synthetic intelligences look like the next stage. At some point he suspects the universe is a puzzle those systems will uncover and solve. Then what? Fast-forward billions of years and Earth is quiet, then turmoil, city lights — and then what at the end.

He has seen an animation of life's history: almost nothing for a long time, then in the last two seconds cities, cluttered low orbit, the whole explosion. Played at a normal speed it just looks like a firecracker. We are stepping through that explosion day to day, so it does not feel like one. He hopes it is a constructive firecracker, not a destructive one. Fridman asks whether the creator left a message in the digits of pi, as in Carl Sagan's *Contact*, or whether the puzzle is to break out of the simulation, Mario-playing-Pong style, and execute arbitrary code on the host. Karpathy's bet is that some later synthetic AI will treat the universe as that kind of puzzle. That is an endgame.

Does he think of the universe as a computation that might have bugs? Yes. Physics may have exploits, and we should try to find them — some crazy quantum arrangement that is a buffer overflow, a floating-point rounding error, a way to extract infinite energy. The analogy is not a joke he is entirely kidding about. When you train reinforcement-learning agents in a physics simulator and ask them to run fast on flat ground, they do weird things. They get on a back leg and slide, because the optimizer found a way to pull infinite energy out of a badly implemented friction force. It is not running. It is a perverse solution.

Maybe we can be that little dog in this physical simulation, finding a shortcut the designer did not intend. The social problem is immediate: the first person to discover the back-leg slide is all we are going to do, and then everybody switches, because it is so fun. The paperclip maximizer sounds ridiculous until the exploit is that good. He does not think a person will find it. It will take a superintelligent AGI of a third generation. We are building the first; that AI bootstraps the next; there is no way for us to introspect what the later one even is.

He likes a certain science-fiction picture: AGIs that are completely inert, that do not interact with anything. He finds it beautiful. They have probably figured out the meta-game — arranging quantum systems, expanding pi, building little fusion reactors — something beyond chemical life, which they have no reason to talk to. Why would you speak to the ants? Physics itself might be the organism, and we the particles on the wave, riding a mostly deterministic path from a big bang toward some super-intelligent replicator, a stable point given these laws.

Fridman quotes Einstein's dice and asks whether there is genuine randomness. Karpathy wants to be careful, then says he thinks the laws are deterministic. He does not like random. Apparent randomness — collapse of the wave function — he reads as deterministic entanglement, some kind of multiverse. Fridman notices he got uncomfortable. Is the universe's randomness a source of anxiety? Karpathy says it is unsettling. He thinks it is a deterministic system.

Then the hand goes up. Why does it feel like free will? Karpathy's answer is blunt: feelings. When a reinforcement-learning agent "makes a choice," the choices are already there. You interpret the choice and create a narrative for having made it. The two of them are, in the moment, generating exactly that narrative. The conversation is meta and he does not try to escape it. The alien artifact, the deterministic wave, the sliding dog, and the raised hand are all the same claim: the story is downstream of the optimization.

Asked for the most beautiful or surprising idea he has watched the field grow into, Karpathy picks the Transformer. For a long time each sensory modality had its own trendy architecture — vision, audio, text, different-looking nets. Recently they have been converging. You can feed the Transformer video, images, speech, or text and it gobbles it up. It is a bit of a general-purpose computer that is also trainable and very efficient to run on existing hardware.

The paper, he says, came out in 2016: *Attention Is All You Need*. (The publication is 2017; the transcript has him off by a year.) He is not sure the authors foresaw the bigness of the impact. He thinks they were aware of motivations and design decisions they chose not to expand in the paper. They were not "just doing translation." They had proposed a really cool differentiable, optimizable, efficient computer, even if they did not have all of that foresight written down.

Fridman notes that the title is a meme, and wonders whether a more serious label would have landed less hard. Karpathy, to a degree, agrees and prefers it this way. A grand title over-promises and under-delivers. Meme your way to greatness. He has tweeted the claim in one sentence: the Transformer is a magnificent neural network architecture because it is a general-purpose differentiable computer — simultaneously expressive in the forward pass, optimizable via backpropagation and gradient descent, and efficient at high-parallelism compute. Expressivity, trainability, and hardware fit. Miss one, and the era of large models does not start.

The available conversation never reaches Tesla's data engine, Optimus, or a full AGI timeline; it stays with the artifact, the chemistry, and the architecture. What holds it together is Karpathy's habit of not pretending to a theory he does not have. He calls the network an alien artifact because the optimization is not evolution. He thinks life is common because Lane made vents feel ordinary and because the clock started fast. He thinks physics may have cracks because he has watched a simulated dog mine friction for free energy. He dislikes randomness because he cannot find a place to put it.

None of those judgments has to be right. Each one has a source you can go inspect: a book, a training run, a paper title that is also a joke. For people who train models, the usable pieces are the refusal to analogize too quickly, the reminder that the knobs hold whatever the loss forced into them, and the Transformer framed as a differentiable computer rather than a better translator. For everyone else, the image that sticks is the firecracker. We are living in the last two seconds of the animation, and it does not feel like an explosion because we are inside the frame.