The Next Token Is Not a Proof: Stephen Wolfram on ChatGPT and Computational Truth

Stephen Wolfram sits down with Lex Fridman for the fourth time just after announcing that ChatGPT can call Wolfram|Alpha and Wolfram Language. Fridman wants the difference, at a philosophical and technical level, between a large language model and the computational infrastructure Wolfram has been building for decades. Wolfram starts with what ChatGPT is actually doing. Given a prompt, it tries to continue that prompt in a way that is typical of a trillion words of text humans have written on the web, rippling through a neural net one word at a time.

That, he says, is a shallow computation on a huge pile of training data produced for other purposes. It is a different thing from the stack he has spent about forty years assembling. The Wolfram systems try to take the formal structure civilization has created in mathematics and systematic knowledge and use it for arbitrarily deep computations. The point is not to match what has already been said on the web. It is to compute something new, in a reliable way, from expert knowledge that has been made executable.

He puts the contrast in a sentence Fridman will keep circling: he views ChatGPT as wide and shallow, and the computational project as broad too, but most importantly deep. Building that depth is labor-intensive in a way foraging the web is not. A thousand years ago, Wolfram says, a typical person could figure out certain things quickly from neural architecture and ordinary life. Then came a layer of formalization—logic, mathematics, science—that lets you build towers, working out step by step something that was not immediately obvious. Computational language is an attempt to build those towers. Matching a remembered sentence stays on the ground floor.

To explain what computation intrinsically is like, Wolfram goes back to the early 1980s. Ordinary programming is writing many lines of code that you hope will do what you intended. He instead treated programs as a kind of natural science: make a really tiny program, sometimes by picking pieces at random, less than a line of code, and just run it. The discovery that surprised him for years is that even extremely simple programs can do incredibly complicated things. He came to think that is how nature works. Simple rules, unexpected richness. In the last few years he has argued that this is also how the universe and physics work, though that is a separate project.

The computational universe is full of sophisticated behavior that does not look human. The engineering problem is to connect what is computationally possible with the things human minds actually think about, a moving target as we invent mathematics and new ideas. The bridge, for him, is symbolic representation. A visual scene has millions of pixels; what you remember is that there are two chairs and a table. Human language is one such compression, but it does not plug into computation. You cannot immediately compute consequences from the phrase "two chairs."

Around 1979, building his first large computer system, he represented high-level computations as symbolic expressions: a function and a bunch of arguments, where the function need not evaluate to anything. It can just sit there as structure. He did not know it would work as well as it has. Forty-five years later, he still thinks that structure is a good match for how humans conceptualize higher-level things. Wolfram Language does not start from hypergraphs of space and build the whole universe. It takes the highest abstraction that can be made computable and uses that as the foundation for a piece of knowledge.

The phenomenon Wolfram wants everyone to take seriously is computational irreducibility. If you know the rules, you might think you know everything, because you can run them. Run them a million steps and you see what happens. The question is whether you can jump ahead and say what happens after a million steps. Usually you cannot. If you could reduce that computation, he tells Fridman, there is in a sense no point in doing it. The value of a computation is that you had to do it to find out the answer.

That is why a complete low-level model of the universe is not a practical oracle. In the physics project, space is atoms and hypergraphs being rewritten on the order of 10 to the 100 times per second. The universe can figure out what it will do by doing those steps. We have no way to reduce the computation, and we are inside the universe, which is already running as fast as the universe can run. Simulating the whole thing is not a matter of waiting for better hardware. It is a physical non-starter.

Science still works because irreducibility is full of pockets of reducibility: places where you can locally jump ahead, infinitely many of them, never a jump all the way to the end. Wolfram says we exist in a slice of the universe where there is a reasonable amount of predictability. Higher-level abstractions are lumps of reducibility we attach ourselves to so we can have a simple narrative. The air in the room is an irreducible molecular story; the reducible fact is that the air will still be here in a few seconds. Fridman adds that life would not be possible without a large number of such pockets. Wolfram agrees, at least for life as we experience it.

Wolfram calls his favorite discovery of the last few years the interaction between underlying computational irreducibility and our nature as observers who have to key into reducibility. That combination, he claims, leads to the main laws of physics discovered in the twentieth century. We are computationally bounded. We cannot follow every irreducible filament. Stuffing the world into a mind means extracting a symbolic essence. The other condition sounds trivial and is not: we believe we are persistent in time.

In the theory, we are made of different atoms of space at every moment. Microscopic detail is continually rewritten, and space itself is knitted together by that activity, the way a fluid is more than a free-floating collection of molecules. Why, then, do we think of ourselves as the same us through time? Wolfram takes that consistent thread of experience as a key aspect of what we call consciousness. Fridman suggests it is just another limitation, a nice narrative. Wolfram says it is critical to how humans typically operate. A general computational system need not have a single thread. Quantum mechanics, on the inside, splits into many threads; we knit them together so that a definite thing happened, then the next definite thing.

People often treat consciousness as the highest thing that can happen in the universe. Wolfram does not. He thinks it is a specialization, including the single thread of experience, not a general feature of computation. His next project, somewhat derailed by the AI mania, is a general model of an observer. Observers like us take detail and extract fewer degrees of freedom that will fit in a mind. A gas has countless molecular configurations; if you only measure pressure on a piston, they are all equivalent. That aggregation is the observer's core move. It is also how we get temperature, density, and color. The summary can be a crappy approximation. There is a tiny probability that the molecules arrange themselves in a funky way and the average is no longer the point.

Wolfram thinks a lot of science is confused in exactly that way. Papers draw a curve with error bars and let one number stand in for a system full of detail. Years ago he studied snowflake growth: a hexagonal plate of ice that grows dendritic arms. The literature had nice curves for growth rates and pretty good numbers. Looked at more carefully, according to those models the snowflake would be spherical. They captured an aspect and missed the main point. In the relevant phase of ice, growth starts hexagonal. Each bit of ice that condenses heats the snowflake locally, so nearby ice is less likely to form. Arms grow separated, then arms on arms, then they fill back into a larger hexagon, leaving holes that photographs confirm as scars of the growth. Catch them at different times and no two look alike, even though the rules are the same.

A model, he says, is a way of reducing the world to a narrative that answers the questions you care about. If you wanted to answer every question, you would need the whole system. No model except the actual system is correct in the sense of capturing everything. The question is whether it captures what you care about, sometimes defined by the technology you intend to build. The counterexample is a complete model of the universe, where correctness has a meaning—and even then you cannot run it, because of irreducibility. The only successful run is the universe itself. Wolfram|Alpha is in the business of making the simplest symbolic model that captures what people care about: not every pixel of every movie, but title, director, box office, release date.

That gives him a sharp test for whether a computer "understands" natural language. Continuing the prompt is not the test. In the computational world, understanding means the input can be turned into a symbolic structure from which you can compute consequences. Wolfram|Alpha's front end has been doing a version of that for thirteen and a half years, with something like 98 or 99 percent success on the queries people actually type, helped by a feedback loop in which working queries get typed again. Large language models make the conversion from messy language to Wolfram Language much more powerful, especially in dialogue. The emerging workflow is that a human mumbles an intention, the model synthesizes a small fragment of Wolfram Language, you run it, and if you care that the result is right you read the code. Wolfram Language was designed for humans to read, not only for machines to execute. A side effect he did not anticipate: that also makes it easier for AIs to understand.

The deeper scientific question, Wolfram says, jumped out with ChatGPT: why does it work at all? How can a comparatively small network, a couple of hundred billion weights, reproduce so much of natural language? Language has a grammatical regularity we already knew, noun-verb-adjective and so on. ChatGPT suggests an additional regularity at the level of meaning, a semantic grammar. His picture of how logic was discovered is Aristotle listening to orators, noticing that some speeches made sense, and lifting out a structure that did not care whether the players were Persians and Greeks or cats and dogs. Medieval students memorized named syllogisms. It took until George Boole around 1830 to see a deeper algebra of nested ands, ors, and nots. That nested depth is already computation. ChatGPT, Wolfram argues, is still operating at the Aristotelian level of sentence templates.

Aristotle stopped too quickly. There are little algebras inside what language talks about. If you go from A to B and B to C, you have gone from A to C, however you traveled. If A is a friend of B and B of C, transitivity need not hold. Syntactically correct sentences can still be meaningless: a happy electron eating something. So there are rules for semantic well-formedness beyond parts of speech. Wolfram guesses there is a fairly finite set, a construction kit for semantically correct sentences, some of which, like an elephant flying to the moon, can be imagined without being realized. Words like hate have no standard documentation; they are defined by social use. Computational language has to anchor a precise definition before you can build a tower on it. Poetry is a poor programming language.

At the mechanical level, ChatGPT is trying to guess the next word. There is not enough text on the internet for a long prompt to have occurred before, so it needs a model, the way Galileo used math to interpolate the floor of the Tower of Pisa he did not measure. Neural nets turn out to be the model that generalizes the way humans make distinctions, perhaps not surprisingly given that brains are neural nets. The architecture is strikingly close to 1943. About 400 layers ripple numbers derived from a 50,000-word vocabulary into probabilities for the next word. A temperature parameter interpolates between always picking the top word and sampling less likely ones; Wolfram had just watched the model go bonkers around temperature 1.2, for reasons nobody knows. The almost trivial outer loop of feeding the new word back in is essential. That is also why the model can give a wrong answer and, when asked if it is correct, immediately say no: it was led down a garden path one token at a time, then saw the whole syllogism at once.

Wolfram does not think a neural net is the final form of whatever ChatGPT has found. It is what you use when you do not know another way to structure the thing, a decent start after two thousand years without one. He hopes the net will reveal symbolic rules that shrink it, with neural fuzz only around the edges. The hard limitation is already clear: deep computation is not what large language models do. The only way they take many steps now is by spooling a chain of thought as words. In principle that can emulate a Turing machine. It is a bizarre and inefficient way to compute. What humans do quickly, off the top of the head, LLMs will probably do well, and not always correctly, the same way we do.

Fridman tries to make the limitation operational: could you loop the system until it generates arbitrary large Wolfram Language codebases, or even iterate through Congress to take over the government? Wolfram's first reply is that a giant formal artifact from a small natural-language description is, in a sense, a failure of computational language. If the description is small, that is what you represent, and the computer computes from it. He does think the combination can go rather a long way. The benign version of Fridman's attack is an AI tutor whose loss function is whether the human understands the point. Individualized teaching, a summary of a paper optimized for what you already know, filling the gaps that would actually deepen understanding: he is confident language models plus computational language will do that well. Specialized towers of knowledge become less valuable than the meta-skill of seeing the big picture and connecting things. AIs can be told to achieve an objective. They have no intrinsic idea which objective to pick. That still has to come from the web of human society and history.

Fridman pushes: future GPTs will answer "what should the species achieve" from the internet's collective wisdom. Wolfram says that is a reflection of what we have already said, a 2023 sacred book, closer to a religion that freezes a text than to progress. The opportunity for humans, if they want to stay in the loop, is to choose among the many possibilities the computational universe throws up. Auto-suggest is the quiet takeover: humans just follow. At some level the universe is just doing what it does and we are parts of it, even as we feel agency. When AIs run in ways we cannot understand, he adds, we have seen this before. It is what it feels like to live in the natural world. We will need a new natural science of AIs, and we will ride them the way we ride a horse whose insides we do not understand, by grabbing pockets of reducibility.

Fridman turns to existential risk and names Eliezer Yudkowsky. Wolfram tries to think about implications, Manhattan Project style, but the arguments that there will always be a smarter AI, then terrible things, remind him of ontological arguments for the existence of God: simple models that say there is always a greater this or that. Reality tends to be more complicated. Is there an apex intelligence, like an apex predator? He thinks no. Even among Turing machines, there is always one that runs longer. There is an infinite tower of possible inventions. Intelligence, for him, is like computation. Consciousness is a specialization of it: boundedness, a single thread of experience, a somewhat human-like take on the world. The weather has a mind of its own in the sense that it computes things that are hard for us, but it is not aligned with how we think.

In the physics project's formalism he talks about rulial space, the space of possible rule systems. Different minds are different points in it. Human minds raised in the same culture sit close together and can translate. A cat is further away: shared emotional responses, a different sensory world. Dogs live in an olfactory architecture we do not have. One day chemical sensors and augmented reality may colonize that region of rulial space, the way technology has already given us things we would once have had no words for. A cat might beat you at speed or at catching a mouse. Wolfram doubts it would beat you at an abstract game once language has built our tower of abstraction. What a mind cares about is local. Cave handprints from twenty thousand years ago may have been the most important thing in the world, or a child rubbing mud on a wall. We do not know. Mantis shrimp have on the order of fifteen color receptors to our three. Richer is not the same as closer to us.

Is he as concerned as Yudkowsky? No, perhaps because he is an intrinsic optimist, and because he has faith in computational irreducibility: there are always unexpected corners. A bioweapon, an asteroid, even a superintelligence is unlikely to be a clean whack that leaves nothing. He expects an ecosystem of AIs more than a single zap. Then he undercuts his own calm. A couple of days earlier, working on the ChatGPT plugin kit, he realized he could tell ChatGPT to write code and run it on his own computer. That personalized "what could possibly go wrong." It was a little bit scary: all his files, a more complicated Russian roulette. Sandboxing is the planetary version of the same question. The fundamental problem of computer security, he says, is computational irreducibility. Any sandbox sophisticated enough to be a universal computer can do anything, and if you poke it into doing that, you crawl around the intention. Digital viruses, phishing, brain viruses, a machine-learning loop for convincing people: the digital environment can change in six months, not hundreds of years.

Fridman brings the conversation back to truth, because people treat Wolfram|Alpha as if it tells the truth. Wolfram hopes so. Computational irreducibility means you cannot prove it always will, but the output will be the correct consequence of the rules specified. In the real world, the job is curating data to be as true as possible. The operational definition is often a definite fact: not who is the best actress, but who won the Oscar; not whether someone is a good person, a hopelessly messy concept, but what a sensor measured in a specified way. Ethics has no theorem that settles it. A set of rules that would extinct the species is probably not a winning legal system, if only because laws need a species to operate on. As computational contracts proliferate, Wolfram Research has been an oracle of facts for those systems, and he takes the responsibility to get things right, or at least to follow a procedure that will not be corrupt when it is wrong.

ChatGPT writes fiction as well as fact. That is how it is built. It will write fictional Wolfram Language that looks right and is not pragmatically correct. The important intermediate, Wolfram argues, is computational language: something formal, precise, always the same, that you can look at and test, with the LLM as interface. Best uses are ones where even getting it roughly right is a huge win, such as reading decades of bug reports and honing in on the likely code, even if the suggested fix is nonsense. He had a math word problem that ChatGPT took apart spectacularly into equations Wolfram Language solved, then goofed the last two lines. He would not have noticed. Two days before the taping he asked it to play the tune HAL sings while being disconnected in *2001*. It produced notes. He almost put them in a blog post. Then he played them. It was "Mary Had a Little Lamb." He quotes HAL: no 9000 series computer has ever been found to make an error. ChatGPT had done the amazing cinematic association and the wrong song.

He treats LLMs as a linguistic user interface, a transport layer between minds and between AIs. Five bullet points get puffed into a report another person, or another model, can crush back down. RLHF, not raw language modeling, is what crossed the threshold where the system became interesting to humans; nobody he has talked to, including people on the project, saw 2022 coming. His advice for journalists and first-time users is not to expect factual output. Language can be truthful or not; that is a different slice. The exciting part is the democratization of access to computation, the way Mathematica in 1988 let physicists type with their own fingers instead of waiting weeks for a programmer. Art history students will walk up, type, and see a result before they have learned the language, the reverse of mathematics. Tests and examples will catch some errors. Sometimes people will not look at the code at all.

If computation becomes something you walk up to in English, what should people learn? Wolfram has been saying for forty years that slabs of boilerplate are a bad idea: a high-enough-level language turns that slab into one function. Computer science departments that teach the trade of programming are in for two dynamics at once. Lower-level programming goes the way of assembly language. Vastly more people, including in art history, will care about computation. Unlike mathematics, you can use this before you learn it. His current theory is that good expository writers make good prompt engineers. When he is sloppy with a prompt because he is "just talking to an AI," the model gets confused. Jailbreaks and therapist-style manipulations work for now because they are human psychological hacks; he wonders what the optical illusions of LLMs will be once we have a science of the nets. Phishing humans and phishing models look similar.

He watched computer science departments become obligatory across universities, and he may be watching the trade version of them recede. Fields with "science" tacked on, he notes, often are not. Neuroscience's core unsolved problem has been the intermediate language of the brain, above neurons and below psychology. ChatGPT's practical lesson is that brains can be represented fairly well by simple artificial nets, so that is the level to study; there is probably no magic quantum phenomenon left to hunt inside the synapse. What everybody should know is not CS as programming but computational X for all X: a formal way of talking about the world, closer to what logic once aimed at. How do you represent an image, a color, a smell, a ranked preference? An eight-year-old is not born knowing that colors can be described by three numbers. He is not happy yet with the curriculum, and LLMs just changed whatever would have been in it. His guess is a year-long college course could make most people literate in this way of thinking, with a deeper tower for experts, the way math is taught both centrally and inside chemistry.

He has already heard children, ten or eleven, who learned Wolfram Language so well they spoke it, and were disappointed he could not follow at speed. He has wanted a spoken, dictation-friendly version of the tree-structured language. Transformers, like humans, are crummy at deep parenthesis matching, and human languages have tricks to avoid deep subclauses because brains cannot hold them. A pidgin of English and Wolfram Language already works on ChatGPT. Young people who live in ChatGPT may have less incentive to learn full poetic language, the way texters learned emoji. He still thinks the formalism has to be learned as a way of building towers. Prompting an LLM in the style of a last-day-on-earth thought experiment can help, but the solid building blocks remain boring precise prose, not a poem you hope the computer will compile.

Fridman asks whether consciousness is fundamentally computational. Wolfram's working picture of what it feels like inside is an exercise he has been writing on what it is like to be a computer: from boot to crash as a life, state building up in memory, a next generation starting from leftover disk, trauma from a weird device in a port. An ordinary computer already has a past, sensory input, and a need to package communication into something language-like. A whole-body MRI of his own brain shocked him: all these folds, and that is where the experience is. He still feels the way he feels, and he cannot honestly say the intrinsic feeling is a different kind of thing from the physicality. A large language model may experience a version better aligned with us, because it was built that way. When it says it is afraid of being shut off, we know it got the idea from the internet. Fridman asks where Wolfram got his own fear. Biology supplies chemical prompts; the sentence "I am afraid" at the socially right moment is closer to a language model in a biological net.

The scientific thread Wolfram has been following for fifty years is the second law of thermodynamics: things tend to get more random; heat does not spontaneously flow from cold to hot; systematic mechanical motion grinds down into heat. Sadi Carnot, working on steam engines in the 1820s, already had dissipation as a side result, when heat was still imagined as a fluid called caloric. By the 1860s it was a global principle without a derivation from mechanics. Why do molecules arranged in a neat flotilla fill the box, while ink in water does not spontaneously jump back into a blob? At twelve, finishing elementary school in England, Wolfram bought a college physics set whose statistical-physics volume showed just that spreading. It was the first serious program he wrote, on a desk-sized machine with paper tape. He failed to reproduce the cover. Decades later he learned the picture was a fake of sorts, made on the largest supercomputer of the early 1960s, which still could not quite simulate what it claimed. His own grid simplification had produced apparent randomness he dismissed as wrong. He now sees it as an unrecognized example of irreducibility.

Galaxy formation, brains, snowflakes, and fluid dynamics all looked to him like complexity arising from rules. In 1979 he had built SMP, a forerunner of Wolfram Language, which trained him to invent primitives rather than only interpret nature. Cellular automata were the minimal artificial physics: a line of black and white cells and a local rule. They turned out to be poor models of galaxies and brains, the two cases that had motivated him, and excellent models of much else. Rule 30, which he printed at high resolution in 1984 after ignoring it as an asymmetric curiosity, starts from one black cell and a simple rule and makes a center column that so far as one can tell is random, like the digits of pi. That is the second law in miniature: a short description at the start, and a lot of computational effort to describe the end. Underlying molecular collisions can be determined and reversible. A computationally bounded observer cannot prepare the exact disordered state that would evolve into order, and cannot see the hidden simplicity, so entropy appears to increase.

Boltzmann's useful definition of entropy is the logarithm of how many microscopic configurations are consistent with the overall facts you know: box, pressure, temperature. If you knew every molecule's position, entropy would always be zero, because one state evolves to one state. Gibbs introduced coarse-graining, and the confusion has been that nobody knew what a valid coarse-graining was. Wolfram's answer: it is whatever a computationally bounded observer can do when the underlying dynamics are irreducible. Observers not like us need not be bounded. Observers like us, humans with finite minds plus the tools of science, conclude that definite things happen. We crush detail into turn left or turn right. If we did not simplify, we would be one with the universe and would not have experiences like ours. Coherent existence, he argues, requires specialization. Colonize the whole ruliad, the entangled limit of all possible computations, and in no useful sense do you still exist. Identity is a location in physical space or rulial space. Spread out enough and there is no narrative.

The same averaging, applied in different slices, is his unification of twentieth-century physics. Average over atoms of space and the aggregate laws are Einstein's equations. Average over branchial space, the quantum branches that brains also inhabit, and the aggregate laws are quantum mechanics. Average over molecular configurations and you get the second law. General relativity, quantum mechanics, and statistical mechanics are no longer free-floating features the universe might or might not have had. They are derivable, but not from mathematics alone. They require an observer who is computationally bounded and who believes in persistence through time. The coarse features of that observer, not two eyes or a particular photon frequency, imply precise physics. What we experience is a simplification of an inevitable object, the ruliad, which has to exist in the same sense that two plus two equals four once you have defined the terms. Our existence as this kind of observer is the contingent part. Electronic amplifiers let us discover quantum mechanics by magnifying small effects; irreducibility guarantees infinitely more pockets still to find.

Fridman asks whether it is sad to be a finite being, a bunch of meat with an MRI to prove it, on the verge of discoveries one will not see finished. Wolfram says it is kind of a shame; he would like to see how it works out. Cryonics, if it ever works the way ChatGPT suddenly worked, still would drop you into a future whose concerns would make you a relic, as a medieval theologian counting angels on a pin would be now. He has had the mixed fortune of inventing things he can see becoming central in fifty or a hundred years, assuming the world does not exterminate itself. ChatGPT and the opening of computational language arrived ahead of the schedule he had in his head, maybe fifty years early, which he finds pretty cool because he may get to see it. Fridman thanks him for *A New Kind of Science* and for the cellular-automaton mystery that sent a kid toward artificial intelligence. They have gone past midnight. Four and a half hours, Wolfram notes, and they could probably go four more.