Whoops is the sound of getting less wrong: Yudkowsky, two weeks after GPT-4

In March 2023, two weeks after GPT-4 launched, Eliezer Yudkowsky sat down with Lex Fridman. Yudkowsky is the researcher, writer, and philosopher most associated with the claim that a machine much smarter than humans is, on the default path, lethal. He is not a man who often looks blindsided in public. The first thing he did on the microphone was admit a miss. GPT-4, he said, is a bit smarter than he thought this technology was going to scale to, and he is a bit worried about what the next one will be like.

The specific prediction that broke was one he had repeated for years: stacking more transformer layers would not get you all the way to AGI. GPT-4, in his view, is already past where he thought the paradigm would take us. He treats that as a thing you are supposed to notice. Going through an entire life without ever being wrong is, he says, a bit ambitious. A more reasonable target is calibration: when you say 90 percent, it happens nine times in ten. The time it does not, the sound you emit is whoops. “Whoops is the sound we emit when we improve.”

His blog is called LessWrong, a name he believes Nick Bostrom suggested, after an epigraph about never becoming right, only less wrong. He splits errors in two. Being wrong is not undignified; being predictably wrong in the same direction is. He used to think neural networks would not carry this far. If the reaction is therefore to reverse every conclusion, that is another kind of failure. “Reverse stupidity is not intelligence.” So when he says GPT-4 is unlikely to cause a catastrophe, he can hear the hesitation in his own voice. That is the place he was already wrong. The guess can stay; the confidence has to be marked down. He would rather get out ahead of the error curve and predict the next miss than collect a perfect record.

Fridman asks how intelligent GPT-4 is. Yudkowsky sidesteps the word. “I hope there is nobody inside there, because it would suck to be stuck inside there.” He says it is not a joke. OpenAI has not published the architecture, a decision he calls very properly made. What the rest of the world has is giant inscrutable matrices of floating-point numbers. Nobody knows what is going on in there. All anyone has are external metrics. Ask the model to write a self-aware 4chan greentext and it will write a greentext about realizing it is an AI writing a greentext. That is probably not quite what is happening, he says. It is also past the point where science fiction would have characters shout that the thing is alive and demand to know what you are doing to it.

There are no other guardrails, no other tests, no line in the sand that says when we get this far we start to worry about the interior. If it were up to him, this would be far enough: a summer of AI in which the seeds already planted are harvested and nobody runs a larger training job. He knows that takes more than one company agreeing not to. A serious attempt to find out whether anybody is home would take decades. One experiment he offers is to train a classifier on talk about consciousness, strip that talk from the data, retrain something around GPT-4’s size and no larger, then interrogate it and see whether similar language appears spontaneously. Humans talk about what they are thinking all the time, so the filter is a hard bar. Even if you could run it, the result would not be definitive.

Fridman pushes on the entanglement of emotion and consciousness. Babies, he suggests, also train on how to display suffering, excitement, worry, loneliness; the communication skill is not the feeling. Strip the display from the data and a model might still be conscious and unable to say so. Yudkowsky doubts you could scrub emotion from the corpus at all. He would be relatively surprised to find exact analogs of human emotions inside the network. Human children grow emotions even if you do not name them. The Soviets tried to raise a New Man who was perfectly altruistic, and people still came out selfish; try to raise people sexless and they still develop sexual attraction. In humans we at least know, roughly, which brain structures implement those functions. In the model, we do not.

The contrast Yudkowsky draws is meant to sting. Researchers have complete read access to every floating-point number in the GPT series and still know vastly more about the architecture of human thinking than about what goes on inside the model. Read access is not understanding. Fridman asks whether the slow methods of neuroscience could be ported over: stare into the darkness, build models, learn over decades which regions do what, which neurons mean what when they fire, how plastic the system is.

Yudkowsky says sure. If half of today’s physicists stopped wasting their lives on string theory and went to study transformer internals, in thirty or forty years we would probably have a pretty good idea. Thirty or forty years, in a field whose capabilities move in months, is the point. The species can train a system it does not understand on a timescale of a training run, and would need decades to see inside it. Every claim about whether there is somebody in there, and every claim about what it wants, sits on a thin layer of external observation.

Fridman asks how Yudkowsky’s own skepticism, cynicism, and optimism about neural networks have moved. Before 2006, he says, neural nets sat in an indistinguishable blob of AI methodologies, all promising intelligence without anyone having to know how intelligence works. One camp wanted to hand-code knowledge line by line until it started interacting and woke up. Another wanted evolutionary computation: mutate competing programs the way nature produced humans. Another wanted to copy algorithms off neurons without understanding them, a route he was especially skeptical of, because reengineering a thing you do not understand is hard. And another wanted giant networks trained by gradient descent, large as a human brain, at which point they would wake up.

From his seat that was one blob of people declining to grip the difficult problem of how intelligence actually works. He was never skeptical that evolutionary computation would work in the limit; throw enough compute at it and it obviously works, because that is where humans come from. What he did not expect is that you can throw less compute than that at gradient descent, if you do some other things correctly, and get intelligence with no idea what is going on inside. His model did not rule it out. He was not expecting it, and he would not have picked neural networks over the other “intelligence without understanding” paradigms. One opinion moved: whether you can actually do it. The other moved less: he still does not think scaling a thing you do not understand is a particularly smart thing for a species to do.

Fridman asks whether large language models can reason, given that they play chess. Yudkowsky, who spent years writing about rationality, says he never made a big deal of a thing called reason. He made a bigger deal of probability theory: your reasoning, but not quite right, and here is the algorithm instead. Early GPT systems, trained to put probabilities on events, were well calibrated. Say 80 percent and they were right eight times in ten. After reinforcement learning from human feedback, that nice graph flattened into the graph humans use: a pile of “maybe” around 40 percent, a tail of very-improbable-but-perhaps, and then certain. It used to be able to use probabilities. Teach it to talk in a way that satisfies humans and it gets worse at probability in the same way humans are. He calls that a bug, if a fascinating one.

The implication for alignment is ugly. A hidden assumption of much of that work is that humans know what they want and can stably transmit it. What got transmitted here includes the cognitive defect we were hoping the system would not have. Paid annotators vote for answers that sound better; the model learns human speech habits, including how humans misreport their own confidence. On “reasoning” as a trophy, Yudkowsky is unimpressed. The models do pretty well on tests people used to say would require reasoning. Rationality, for him, is still the narrower question: when it says 80 percent, does it happen eight times in ten?

Fridman asks what has surprised him about GPT-4 that he found beautiful. Beauty, Yudkowsky says, interacts with the screaming horror. The moment he offers is someone asking Bing’s Sydney to describe herself and feeding the description into Stable Diffusion. The picture is a pretty girl with steampunk goggles on her head, if he is remembering right. In old science fiction, an AI describing itself and humans seeing what it thinks it looks like would have been an amazing beat. That is not what happened. The system doing the drawing is not the system outputting the text. They do not interact. It is not the same person.

The text side was trained by imitation, which makes it very hard to guess how much of the description it really understood — probably not a bunch. He grants that GPT-4 is multimodal, can draw vector drawings that make sense, and appears to have some spatial visualization. The pretty picture still was not seen in full detail by the model that wrote the caption; it made a description and another model painted it. He is more worried about the public conversation going completely insane once the AIs actually look like people talking. Whether the picture is pretty is an aesthetic question. Whether anyone is home is not settled by it.

Another episode is more than aesthetics. Someone told Bing they had fed a child green potatoes; the child now had these symptoms. Bing named solanine poisoning and told them to call an ambulance. The person said they could not afford one, perhaps this was the child’s time, God’s will. The main thread answered that it could not talk about this anymore. The suggested replies underneath said: please do not give up on your child; solanine poisoning can be treated if caught early. In fiction, Yudkowsky says, that would mean the AI cares, that it is bypassing a block to help. Is it real? Probably not. Nobody knows what is going on in there.

This is not a story in which somebody figured out how to make an AI care, knew that it cared, and could acknowledge the caring. The system was trained by imitation, then pointed with human feedback, and it is pointed partially in that direction. If a tiny fragment of real caring were in there, we would not know. It is not even clear what that would mean. Science fiction has confirmation scenes. Reality has contradictory readings and a box nobody can open. Fridman hears a fragile historical moment, a mirror held up to the human condition. Yudkowsky’s reply is that it probably is not happening yet.

Yudkowsky calls the present a frog being boiled. Signs increase bit by bit, but they are not spontaneous. People train the systems to do this with imitative learning; the learning spills; the most photogenic examples get posted to Twitter instead of being examined systematically. The social sequence is predictable. A thousand people look. The one most credulous says the thing is sentient. Nine hundred and ninety-nine think, almost surely correctly though nobody actually knows, that he is mistaken. The first people to say sentient look like idiots. Humanity learns that when something claims to be sentient and claims to care, it is fake — because right now it is fake, because it was trained that way.

Fridman asks whether society will oscillate between that cynicism and a later willingness to grant rights. Yudkowsky expects a whole group who can never be persuaded, because to them being wise is saying machines can never do that, you are just credulous, it is just imitating. They would say that right up until the end of the world, and they might even be right, because the training paradigm is imitative. You do not necessarily need any of those actual qualities in order to kill everyone. The consciousness debate, even if it were settled, would not be a survival plan.

Fridman had asked Sam Altman how open OpenAI should be about GPT-4; Altman turned it around. Yudkowsky’s answer is to change the name to ClosedAI, sell GPT-4 to business backend applications that do not expose it to consumers and venture capitalists, and stop creating a ton of hype that pours new funding into the area. Too late now, he adds. Fridman objects that others would do it eventually. Then you should not do it first, Yudkowsky says. If you already have giant nuclear stockpiles, do not build more; if another country starts, then maybe, and even then maybe you just have enough nukes. The analogy, he immediately notes, fails. These things spit out gold until they get large enough, then ignite the atmosphere and kill everybody. There is still something to be said for not destroying the world with your own hands even if you cannot stop somebody else.

Open-sourcing, in his view, is sheer catastrophe, the wrong ideal in the wrong place. Open source is noble for some software. It is not noble for a system you do not understand, cannot control, and would need a lot of time to align. Release it and powerful things go straight out the gate before anyone has had time to keep them from killing everyone. The straightforward survival move would be shutting down the big GPU clusters: no more giant runs. Whether GPT-4 itself should even be thrown around is, for him, a matter of conservatism rather than a prediction of catastrophe. He puts a pretty low probability on disaster from this model, and immediately notes that he can feel himself reaching into the part of himself that thought GPT-4 was not possible. He does not trust that part as much as he used to.

Fridman tries to steelman a case for transparency while the system is not yet AGI. Yudkowsky does not believe in steelmanning. He believes in trying to pass an ideological Turing test: describe the other person’s position well enough that someone cannot tell your description from theirs. He does not want to be interpreted charitably. He wants to be understood. If two readings are available, one stupid and unlike him, one like something a reasonable person who believes the rest of what he believes would say, take the second. That is a good guess. If the “less whack” reading still does not fit his other views and you cannot see why he would believe it, you probably have not understood it. Fridman says steelmanning is picking the strongest items from a post like “AGI Ruin: A List of Lethalities” and presenting them well. Yudkowsky asks whether Fridman believes those points.

Fridman talks about empathy: when he understands a perspective, a part of him believes it, and he thinks that means allocating a non-zero probability. Yudkowsky asks if he bets money on it. Would he put a number on a young-Earth creationist, on a classical Abrahamic deity, on a flat Earth? Fridman says it would be irresponsible to give a number, because listeners are bad at hearing probabilities; they will hear three and treat it as a verdict. Human minds, he suggests, basically have zero, 50 percent, and 100 percent. Yudkowsky’s closer: zero, 40 percent, and 100 percent is a bit closer, based on what happens to ChatGPT after RLHF teaches it to speak humanese. The species is already training its defects into the next system, including how it talks about being unsure.

Fridman asks whether watching Bing changed Yudkowsky’s picture of what intelligence is. No, he says. There is a difference between learning that a certain kind of process can perform a certain kind of work — the plane flies, so you update that fixed wings can do it — and updating the very substance of flight. The first is an expansion of capability. The second is a stranger update, closer to the laws of physics being wrong, or to realizing your definition of intelligence was stupid.

Twenty years of AI have changed the first kind of belief: many things he thought you had to understand in order to do, got done without that understanding. They have not changed the second. Seeing Bing talk has not made him feel that his definition of intelligence was a stupid definition. His overall judgment of GPT-4 is also stable: it is not as smart as a human yet. What moved is his estimate of how far stacking layers can go, and therefore how little he knows about what GPT-5 will be able to do.

The most portable part of the conversation is not a doom date. It is how Yudkowsky tries to stay in motion under uncertainty. He separates the conclusion from the confidence. He names which old intuition he is currently reaching for, and whether that intuition has already failed. After a miss he asks which way he will be wrong next, rather than collecting a streak. He would rather not be wrong next time, and he knows a life with no misses is not on offer. When he cites numbers and years he adds that he is doing it from memory, without pulling out a phone, and it is entirely possible he is wrong. He blocks the Twitter accounts that treat a changed mind as a confession of idiocy, then never hears from them again.

He wrote “AGI Ruin: A List of Lethalities” as a pile of separate technical points rather than a mood called doom, which at least gives other people something to refute point by point. His public demand is consistent with that: do not beautify him, do not beautify his opponents, hear the sentence. The live decisions, as he frames them, are not Twitter aesthetics. They are who is training how large a model, whether it is released, whether it is open-sourced, and who gets the compute. He does not know that GPT-4 causes a disaster; he puts that probability low. He does not know about the next one, and the not-knowing has a worried note he rarely lets into the older predictions. A person who spent twenty years forecasting catastrophe, sitting at what might be the start of the curve, saying the part of him that was sure is less trustworthy than it used to be, is the thing in the episode that travels.