Six days after “slow down,” the plugins went live

Nathan Labenz opens this Cognitive Revolution bonus by pointing listeners to another show. He had just appeared on the Future of Life Institute podcast, the same organization that published the open letter calling for a voluntary six-month pause on training runs larger than GPT-4. FLI was founded by Skype co-founder Jan Tallinn, whom Labenz had recently interviewed, and its podcast — hosted by Gus Docker — had been running unusually dense conversations with people such as Neel Nanda and Ajeya Cotra.

The episode Docker recorded with Connor Leahy, CEO of Conjecture, is the one Labenz wanted his own audience to hear. Conjecture describes itself as a team doing applied, scalable alignment research and building products that help businesses improve their workflows. Tallinn, an investor in the company, called them “a team that has the highest respect for AI,” a virtue Labenz says he tries to cultivate himself. This is part one of a two-part interview; the second half lives on FLI’s own feed.

Docker starts with the blunt question: what is happening with GPT-4, and is this the moment AI becomes a mainstream issue? Leahy does not treat it as hype. The previous two weeks, he says, have been the most interesting of his career in terms of the wider world. After GPT-3 he had told himself nothing could get crazier in a short stretch of time, then realized that prediction was already wrong. GPT-4’s release, and perhaps ChatGPT even more than the base model, changed how journalists, politicians, and national-security people talk to him. They are starting to panic.

The panic is no longer a Silicon Valley Twitter phenomenon. Last Christmas Leahy went home to rural southern Germany, a place he describes as usually empty of frontier technology, where he is the only person in the family who really knows how to use a computer. His cousins were talking about ChatGPT. They told him they had assumed “this AI thing” was a personal hobby, and that he had turned out to be right. Front-page coverage followed; people from every walk of life were noticing that the technology was real and would touch their families.

The ironic twist, he says, is that the people deepest in tech are the least rational about it and take it least seriously. The old meme that you cannot explain AI risk to ordinary people may have been true twenty years ago. It is not his experience now. He can stop someone on the street, show them ChatGPT, and say: these companies are building bigger and stronger systems they cannot control; do you think that is good? The reply is “no, obviously not.”

Docker asks whether the jump from GPT-2 to GPT-3 was larger than GPT-3 to GPT-4 — whether returns are diminishing. Leahy says no: it is on the track he expected. He used an early GPT-4 build around August 2022, when copies were circulating in the Bay, and found it already impressive. The public release is significantly better at reasoning. From what he has heard, GPT-4 is larger than GPT-3, but not that much larger. The striking property is consistency. Tasks that wow people with GPT-4 were often possible with GPT-3 if you retried, error-checked, and prompted well. GPT-4 does them reliably, helped by better fine-tuning, RLHF, and a clearer picture of what users actually want: not a raw model of a text corpus, but a thing that does things for them.

That usefulness is also what makes the system more dangerous than raw GPT-3. Leahy is explicit: the GPT-4 on the API is not a base model trained only to predict text. It is an RL-trained, instruction-tuned, heavily engineered system designed to solve tasks. He is in technical awe of the work. GPT-4 is the first model he finds delightful to use; GPT-2 and GPT-3 made him want to pull his hair out, and he barely used language models for that reason. How OpenAI got the effect, he says, is anyone’s guess, including theirs. There is no theory that 7.5 micro-alignments of RLHF produce delight. People label data until it looks good. If he did not think the systems were extremely dangerous, he would love to work with them. He reaches for the Trinity nuclear test: an impressive feat of engineering, and one he considers extremely dangerous.

Have we mapped what GPT-4 can do? “Not even scratch the surface.” Sam Altman’s story that incremental release lets society test and debug the systems is, in Leahy’s view, obviously false. If you believed it, you would ship GPT-3, wait until institutions, regulation, and interpretability had caught up, and only then ship GPT-4. What actually happened is the joke he cannot let go: Jan Leike, then OpenAI’s head of alignment, tweeted that perhaps we should slow down before hooking language models into everything. Six days later Sam Altman announced ChatGPT plugins, plugging the model into tools across the net. “If this was in a movie,” Leahy says, there would be a cut, and everyone would have laughed.

The deeper problem is that there is no way to prove the absence of a capability. We cannot test what models cannot do. As people add tools, memory, recurrence, and agent frameworks such as LangChain, Leahy sees a prototype AGI emerging, and he is not sure the “proto” prefix will last. “Large language model,” he has said for years, is a misnomer. Language is an implementation detail. These systems are general cognition engines: they encode inputs from many modalities into a shared semantic space, run cognitive operations, and decode an output. GPT-4 mapping images and text into the same internal space is the same move a brain makes when retina and cochlea become spike trains. Plugins attach virtual muscles to those spikes.

A huge share of the human mind, he adds, is not in the skull. It lives in the internet, in books, in other people, in social networks. The old saying that part of you dies when someone else dies is, on this view, literally true. GPT-3 was solipsistic: it could not use tools or interact with an environment. Plugins make the system non-solipsistic. Five years earlier, people in the field still talked earnestly about boxing a powerful AI in a virtual machine, a firewall, a secure data center — “surely no one would be so stupid” as to hook it to the internet. Now a capital-driven army is racing to connect models to every tool and every environment, “directly into your home.” He adds a disclaimer: the official plugins do not attach to a shell. Plenty of open-source repos do.

Leahy’s half-joking word for this is magic: computation happening that we do not understand. Write a calculator, keep the abstractions tight, type 2 + 2, and you know why 4 comes out. Give GPT-4 a prompt and you have no idea what happened in between, what else it considered, or how a one-word change would move the answer. Biology is magical in the same observer-dependent sense. A blurry photo of a dog that a vision model cannot name is a human-intelligible failure. A crisp photo of a dog plus one weird pixel, classified as an ostrich, is not. “We have summoned a strange little thing from the dimension of math to do some tasks for us,” he says, “but we don’t know what thing we summoned.”

He likes the Shoggoth memes circulating on Twitter: a huge alien creature with a little smiley-face mask. Inside the narrow, graded distribution, the mask stays on. Outside it you find “roiling madness.” Language models do not fail the way humans fail. When Microsoft’s Sydney went off the rails, the trajectory had nothing in common with a person losing their mind. Leahy has even checked the lore. In H. P. Lovecraft, Shoggoths are powerful, not really sentient, and steered by hypnotic suggestion from other aliens. The base model is the monster; fine-tuning is the hypnosis.

Magic is observer-dependent. Clarke said sufficiently advanced technology is indistinguishable from magic; Leahy goes further and says it *is* magic. An alien with a perfect theory of deep learning would see GPT-4 as a white box. We cannot put bounds on systems we do not understand, which is one thing in a toy chatbot and another in a general engine. He then describes, with visible disgust, what people type into chat boxes: illegal pornography, and a distressingly large group who take pleasure in making models act distressed. He does not think the models have qualia or moral patience, but delighting in the torture of something that acts like a suffering human is “really sociopathic.” People want twisted things. A company that only maximizes engagement, as social networks already showed, will not end in a nice place.

If black-box magic is too dangerous, Leahy’s alternative is cognitive emulation, or CoEm. A full CoEm would be a system of many sub-components that emulates human epistemology: it reasons the way humans reason, fails the way humans fail, and exposes a reasoning process a person can read. Ask it for a blueprint and you get a causal trace — why those decisions, why you should trust the drawing. The analogy is being CEO of a well-aligned company: every employee can explain themselves without leaps no human could follow, and you can check that the explanation is true. Language models can hallucinate a fluent self-report that has nothing to do with their internals. The parts would not be people. They would have no emotions, identity, or goals — “platonic humans,” a floating idealized thinking stuff, a neocortex without the rest.

The load-bearing word is boundedness: knowing, before you run a system, what it will not do. A car limited to a hundred miles an hour is a bounded car. Stronger systems need tighter bounds; a chatbot can be sloppy in ways a flight-control system cannot. Leahy’s design picture is a secure data center. You write a specification, you make explicit assumptions (the adversary does not have exponential compute), you derive a safety property (plaintext passwords never leave), and you tell a causal story a skeptical interlocutor would have to accept. Boundedness lives at two layers. The spec can be sound and the CPU still have a hardware bug. A safe AGI, on this view, is a spec plus an implementation that upholds it, not a hope that testing will save you after you turn it on.

Does that mean the whole thing must be hardcoded, good-old-fashioned AI? In an ideal world with thousands of years, yes, you would proof-check everything. He does not expect that. Neural nets can still sit inside the design if the safety story only relies on weak assumptions you are willing to make. He will assume GPT-4 cannot Rowhammer its way onto the internet with one forward pass; he cannot prove it, but he would be “really surprised.” He would be less surprised if GPT-Omega from the year 9000 could. Feed the black box into a white-box JSON schema parser and reject anything that does not fit, and you now know something about the next input. “There is a qualitative difference between letting one big black box do everything and having black boxes involved in a larger system.”

He expects any realistic safe AGI to be a mix of black boxes, white boxes, clear interfaces, and a causal story. Pure white-box design is possible for a future alien; it is not the world we are in, unless capability progress slows enough to buy twenty years. Stack-more-layers is too far ahead. CoEm is a restriction on mind-space, not “train a black box on traces of human thought.” That, he says, just grows another Shoggoth. Language models only do atomic sub-tasks. Traditional software architecture and distributed-systems thinking come back to the center of AI design.

Docker asks the obvious objection: models are trained on human-created data and fine-tuned with human feedback, so will they not become more human-like? The smiley face will, Leahy says. The underlying model, where the actual reasoning happens, will not. Humans do not have a sense organ that takes in randomly sampled internet tokens with no time, touch, smell, hearing, sight, or body. Take a human brain, cut off every sense except that stream, train it for ten thousand subjective years, and put it back in a body: he does not expect a human, however fluent the prose. The architecture is not human, the training algorithm is not human, and the priors and emotions brains come with are missing. “The burden of proof is on them.” The popular picture of a person sitting down to read a hundred thousand books is, in his view, not what training is.

Why think human reasoning is emulable at all? He reaches for Kahneman’s System 1 and System 2. Human intuition is messy; explicit reasoning is closer to a semi-logical system running on a fuzzy, not-quite-formal ontology. Expert systems and logic programming did not fail because reasoning engines cannot reason. They failed because they never built a fuzzy ontology, a usable latent space. Language models, he thinks, finally provide that space. GPT is a huge System 1. Human System 2 is recurrent use of System 1 through a low-dimensional channel — language, a short-term memory that holds about seven things.

A massive amount of cognition is already outside the brain. As a CEO, Leahy’s job is to move thoughts out of his head into other heads and make sure they get thought. A good CEO wants an empty head: a switchboard that routes by priority. Inner monologue is a degraded compression of what the mind can pass around at high bandwidth; the moment you talk to a screen or a colleague you hit a low-dimensional bottleneck. That is an existence proof. Draw a causal graph of how technology gets made and much of it sits between brains, in tools and institutions. An alien observer who did not understand neurons would still find many white, boundable steps. The inside of a brain is more complicated than the larger system.

Science, as he defines it, is the search for clever things that are not true. The efficient-market hypothesis is never literally true; assume it, and a high-dimensional market collapses into something you can reason about. A farmer does not simulate every nitrogen molecule and gust of wind. He plants a monoculture, assumes an acre of wheat yields roughly this much if it does not flood, and estimates the winter store. Flood, and the spec is void. Humans can only optimize in very low-dimensional spaces; science is the batch production of those simplifying assumptions. Headphones are the same argument in hardware. You trust them, but if you insisted you could find blueprints, a designer, and a physics textbook. No step is superhuman. The headphones were not built by one person who cannot explain them to anyone.

Docker pushes on the standard: a specification so good you never have to run the AGI to know it is safe. Is that impossibly high? Leahy disagrees completely. He is not asking for a formal proof. He wants a causal story at the level of “passwords do not leak.” The hard line is that no part of the system may be superhuman. Change a CoEm’s knobs into superintelligence and the story fails: “then you die.” We cannot make assumptions about superhuman intelligence because nobody knows what it is, what it can do, or where it stops. Speed and serial depth are different risks. Most of the danger, he thinks, lives in serial depth: how many consecutive steps a reasoning chain can take. Super-fast thought is dangerous because it reaches unprecedented depths of reflection and self-improvement. A CoEm component that can self-improve without a bound is a failed spec.

The metaphor he likes is a company. A good CoEm should not hand the user a thousand-x AGI or make the user a thousand times smarter. It should let the user function like a thousand parallel one-x AGIs: bounded, understandable, each with a causal story. A thousand-x serial monster is not on the list. He keeps repeating a caveat: CoEm is not a complete alignment solution. The safety property is that a responsible user who follows the protocol will not get killed. Tell a CoEm to shoot your leg off and it shoots. A system that refuses is harder, and it is the next step if CoEm works. The parts are not employees with feelings. They are a platonic neocortex — an optimizer with no objective — and the user is plugged in as the source of motivation. That is the split from the “cyborg” agenda associated with Janus: they hook a person to an alien cortex; CoEm builds a human-shaped abstraction layer and hooks that to the user.

Will the market want this? He offers an optimistic and a pessimistic version. Optimistic: a perfectly loyal company staffed entirely by John von Neumanns, used to speed alignment research, whole-brain emulation, nanotechnology. Pessimistic: by the time CoEm matures, the market already has a 100-x von Neumann GPT, “which then promptly destroys the world.” So he spends much of his time on policy and communications. The world, as he sees it, is a death race toward a precipice, driven by a very small number of people in the Bay Area and London. The myth that “people want this” dissolves if you explain what the labs mean. Civilians hear AGI and picture a robot buddy. People at OpenAI or DeepMind, he says, mean god-like, self-improving, non-human AI that can take over a government or destroy the species — and they have said so publicly. Once ordinary people hear that, the reaction is: of course not; where is the government?

Panic buys time, not victory. Even a working CoEm leaves the hard problem of aligned superintelligence unsolved. “This is the least weird your life is going to be for the rest of your life.” He does not expect the species to make it out of the century, and he is not sure about the decade. The plan, if CoEm ever works: produce enormous economic value, trade with everyone — a cancer cure, money, whatever they want — in return for a promise not to build AGI; fuse with at least one government, military, and intelligence service; then put a thousand von Neumanns on alignment theory and formal verification. He quotes the old line that any plan requiring more than two things to go right will never work, and this plan needs more than two. He does not expect it to work. He still hopes to get there.