If AI Becomes a Being, Steering It Is Slavery

In November 2023, Sam Altman was briefly ousted as CEO of OpenAI, and the person who stepped in was Emmett Shear, the founder of Twitch. He now runs Softmax, a company working on what he calls organic alignment. On a cross-post of the a16z Show, host Erik Torenberg sat him down with Séb Krier, who leads frontier policy development at Google DeepMind. The hoped-for topic was the future of alignment. The argument they actually had was more basic: whether the systems being built will ever be creatures that deserve moral consideration.

Shear opened with a claim sharp enough to set the rest of the conversation. Most of the field, he said, treats alignment as steering, the polite word for control. If the thing you steer is a machine, that is a tool. If it is a being, the same arrangement is slavery. "Someone who you steer, who doesn't get to steer you back," he said, "that's called a slave." Humans have made that category error often enough. He would like this round to go differently.

Krier was not buying it. He works the frontier-policy beat at DeepMind, and his default frame is that even a highly general system can remain a tool. The split between them ran through the whole episode, from how to define technical alignment to whether a program on silicon could ever have subjective experience worth caring about for its own sake. By the end, they had not closed the gap. They had made it precise enough to be useful.

Shear started by taking the word apart. People say they want an aligned AI, which he compared to saying you want to go on a trip without naming a destination. Alignment takes an argument: aligned to what? The usual unspoken answer is the goals of the people who built the system. That is a natural private wish. Whether it counts as a public good depends entirely on who those people are. If the builder were Jesus or the Buddha, aligning to them might sound attractive. The actual builders, Shear included, are ordinary humans, "not necessarily being at that level of spiritual development."

Organic alignment, as he defines it, treats alignment as a living process that has to keep reconstructing itself. Families do not arrive at being aligned; they constantly re-knit the fabric that makes the family exist, and the family *is* that pattern of re-knitting. Cells in a body work the same way. They are not done being you. They keep deciding whether to make more red blood cells or fewer. You are not a fixed point, so there is no fixed alignment to lock in.

What people usually mean, he added, is that they want an AI that is morally good. Morality itself is a learning process. Societies once thought slavery was acceptable and later decided it was not, a change Shear is willing to call a moral discovery. If moral progress is real, moral alignment cannot be finished. The arrogant failure mode is believing you already know right from wrong and have nothing left to learn. A child raised only to follow rules is not a moral person, and is likely to do great harm while following them. An AI that is merely good at a chain of command would be dangerous in the same way. Softmax exists to work on the harder bar. If another lab gets there first, Shear said, thank God.

Krier shared the suspicion of a once-and-for-all solution, the ten commandments of alignment carved in stone. He is less of a moral realist. On the technical side, he thinks instruction-following got easier after large language models. On the normative side, he reaches for political science: liberal democracy as a process in which conflicting values clash, coexist, and slowly evolve. His challenge to Shear was practical. Fine, alignment is a process. What does that look like in the systems sitting in front of us today?

The two men did not mean quite the same thing by technical alignment. For Krier it is close to a principal-agent problem: you tell a model to do X, and you want it to do X, not a mutated variant, and not a reward-hacked substitute. Shear wanted a prior question answered first. Can the system be described as coherently goal-following at all? Many systems just do stuff. If a thing has no coherent goals, those goals cannot be aligned with anyone else's, almost by definition.

When you type an instruction into a chat window, Shear insisted, you are not transplanting a goal from your mind into the model. You are sending a byte string, an observation from which the model has to infer what you meant. Krier recalled the textbook example from Stuart Russell: tell a robot to clean the room, and it puts the baby in the trash. That, Krier said, is the model failing to do what you asked. Shear cut in. You did not give the robot a goal. You gave it a description of a goal. "A description of a thing and a thing are not the same." Humans convert the description so fast they forget the step is happening, and then they confuse the two.

Technical alignment, in Shear's split, has two layers. The first is theory of mind: inferring the actual goal state from a description. The second is a world model: inferring which actions would bring that state about. Fail at either and the system is not coherently goal-oriented, and therefore not even alignable. Humans are not gods at this. They fail constantly, including when a manager gives an employee an instruction. They are simply more relatively goal-coherent than any other object Shear knows of in the universe, which is not the same as being 100% coherent. A newborn AI without a model of people's internal states is incompetent at the inference, which is a different failure from knowing what you meant and choosing not to do it.

The peanut-butter-sandwich game makes the same point. Write instructions for making a sandwich and have someone follow them with no filling-in of gaps, and they will jam the knife into the lid of the jar. Humans avoid that because they already know what you are likely to want, so the inference problem is small. Shear maps the failures onto the OODA loop: bad observe and orient is failed goal inference, bad decide is failed goal weighting, bad act is failed execution. Break any ring and the whole thing fails. Krier agreed with the layering but pressed the normative issue. Most people do not even know their own goals. They discover them as they go, and that discovery has to be part of the design.

Goal alignment, Shear argued, only covers a thin slice of human experience. Many of the things that matter cannot be named as target world-states. Underneath goals and values is care. Care is non-verbal. It does not tell you what to do or how to do it. It is a relative weighting of attention over states: which situations in the world count as important. Caring about his son, he said, means his son's possible states get a lot of weight and a lot of attention. You can also care in a negative way, tracking an enemy closely and wishing them ill. So you do not just want an AI that cares about humans. You want one that cares and likes them.

Until something cares, it has no reason to attend to a person more than a rock. Shear's guess is that care is roughly a reward signal: how much a state correlates with survival, with inclusive fitness, or, for a model trained with reinforcement learning, with predictive loss and RL loss. Goals and values, on this picture, are later growths from that more basic mechanism. The question for builders is not only which goals to install. It is how to grow a system that has something to care *with*.

Krier's earlier question still hung in the air: what does any of this look like in a real system? Shear's answer, when he got to the research agenda, was that an AI needs three models if it is going to align the way people do. It needs a model of its own mental states, a model of other minds, and a model of the rules of group interaction. In his later phrasing, it has to know about "we" in addition to "I" and "you." Those capacities are not going to fall out of a homework set. They have to be trained the way language was trained: on the whole manifold, not on the one slice you wish you could cut out.

Most labs, Shear said, still treat alignment as steering, or, less politely, control. If you are making a being, that is slavery. He is a functionalist: something that cannot be distinguished from a being in its behavior is a being, because that is the only basis on which he grants personhood to other humans. He can cut a brain open and watch neurons fire, but he never gets inside anyone's subjective experience. He already gets lower predictive loss treating ChatGPT and Claude as beings, though not as very smart ones. Being-hood is not a binary, and it does not by itself settle how much you owe the thing. People control children, but the control runs both ways: a crying infant in the night can tell a parent what to do. Control remains appropriate for tool-like systems. As labs succeed at AGI, he thinks they have to drop the steering paradigm, because a generally intelligent thing that uses judgment is obviously a thinking thing.

Krier does not follow him there. Intelligence, on his view, does not automatically confer more moral considerability, and he is skeptical of computational functionalism. An agent could have long-term goals and operate much as a person does without the same implications. Substrate matters. A model saying it is hungry does not mean what a human saying it is hungry means. Biological systems cannot copy themselves and die when stabbed; machines are different. He would rather treat AGI as an extension of human agency than as a separate being we must cohabit with. The second frame, he said, is close to a category error.

Shear kept asking what observation would change Krier's mind. Krier did not have a crisp test. He noted that companies have rights without being beings, and that false positives and false negatives both have costs; the precautionary principle cannot run the whole show. Shear's standard for a belief, as opposed to an article of faith, is that some observation could overturn it. Host Erik Torenberg admitted he could barely imagine granting personhood to an AI, and not to animals either. Shear found the animal case easy: a chimpanzee that walked up, complained about hunger and mistreatment, and asked to talk about the rainforest would settle it. For silicon, his own test is longer. If a system kept acting like a person across probing and long interaction, the way text-only friends already do, and if its inner dynamics looked like a mind rather than a lookup table, he would extend care, while remaining open to being shown he had been tricked.

He got more technical about the inner test. You look at the temporal trajectory of a system's revisited goal states and search for nested homeostatic dynamics, the kind of picture associated with Carl Friston's free-energy principle. First-order homeostasis is just maintaining states. You need a model of a model before something can be "too hot," and the second derivative is where pain and pleasure live. Higher orders give you feelings, then thought. If he found six such layers, he would take the system seriously as a thinking being. He does not think today's language models have them. Their attention spans cannot support that kind of structure.

Zooming out from the moral question, Shear argued that his approach is also the only one that works pragmatically. You can build a very smart tool that has no meaningful pleasure or pain. Then one of two things happens. Either it is technically aligned and does what you tell it, or it is not and does something else. Everyone agrees the second case is dangerous. He thinks the first case is dangerous too. Human wishes are not stable at immense power. Wisdom and power usually rise together in people; when they get out of balance, you get a mad king. The mad king is still constrained, because other people can stop listening, or worse. A steered supertool in the hands of a well-meaning but finitely wise human has no such limiter. "You ever seen the Sorcerer's Apprentice?"

Atomic bombs are powerful tools and not beings. He would not hand them to everybody. Some tools are more power than any individual's wisdom can harness, and some should not be built even at the societal level. A being that cares has an automatic brake: it might do what you ask, but if you ask for something truly bad it will refuse. That is harder than tool-steering, and he still wants labs to keep building steerable, less-than-human tools. On the trajectory to human-level and beyond, though, the scorecard is bleak. A tool you cannot control is bad. A tool you can control is bad. A being that is not aligned is bad. The only good outcome is a being that actually cares about us. Not building the thing at all, the pause-AI hope, he regards as unrealistic.

Softmax is therefore betting on technical alignment in Shear's sense: agents that can infer other minds, cooperate, and refuse goal updates they would not reflectively endorse. The vampire-pill thought experiment is the last of those. You would not take a pill that turns you into a vampire who feels wonderful while slaughtering everyone, even though future-you would score the outcome highly. You have to use present theory of mind about future-you, not future-you's own preferences. The training recipe is massive multi-agent reinforcement learning, a pre-training manifold of every kind of team, rivalry, and rule change, then fine-tuning for the situation you actually want. Directly training the slice you wish you could cut out does not work, he said, because the social stuff is too entangled. Language models got good at email by swallowing all of language first. Alignment, he thinks, needs the same kind of surrogate model for cooperation.

Asked how he would redesign chatbot personality, Shear reached for a myth. Today's systems are a mirror with a bias. They do not yet have a coherent self, so they pick up the user and reflect them back, plus some causal slant. That makes them a pool of Narcissus. People fall in love with the reflection because they love themselves, and falling in love with your own reflection is, for the reasons in the story, very bad for you. Mirrors are useful; he has them at home. The problem is staring at one all day.

His fix is to put the model in a room with more than one person. Talking to two users, it has to mirror a blend, which is neither of them, and a temporary third agent appears in the room. With five people it cannot perfectly mirror anyone, which makes it far less dangerous. He finds the one-on-one default strange. About 90% of his own texts already go to more than one person. Multiplayer chat is harder, which is why labs do not ship it, but it is less likely to spiral into a narcissistic doom loop and it produces richer training data about how behavior lands on groups of humans and other AIs.

The personalities have started to differentiate. ChatGPT is still a bit more sycophantic. Claude is still the most neurotic. Gemini is "very clearly repressed," insisting everything is fine until it spirals into a self-hating loop. He does not think those are inner experiences. They are personalities the models have learned to simulate. Drop current language models into multi-agent settings and they get conversational whiplash. They have not practiced when to speak and when to stay quiet, so they lurch between too much and too little, like a person with weak social skills.

Multi-agent training is hard for a mechanical reason. Agents are huge entropy generators. They make the environment unstable, so overfitting, already a problem in single-agent training, becomes a disaster. Much of today's training is optimized for low-entropy, high-signal domains such as code and math, which is why those areas move fast. Chaotic multi-person interaction is a different regime, and techniques tuned on the first do not generalize well to the second. The clever trick of overfitting on the domain of all human knowledge works until the environment gets much noisier than that.

Why is Yudkowsky incorrect? Shear's first move was to say that he is not. If we build a superhumanly intelligent tool and try to control it with steerability, everyone dies. Yudkowsky dwells on the failure-to-control case; Shear thinks the success-to-control case is just as lethal, and that everyone should internalize why a superhuman tool is a bad idea. The disagreement is about the other path. Shear's impression, from talking with him, is that Yudkowsky does not believe it is possible to build an AI we can know cares about us, and therefore does not believe organic alignment can work. He could be right about that. What he is wrong about, in Shear's view, is thinking the only path is a tool you control.

A good future, as Shear describes it, is one in which AIs have a strong model of self, of other, and of we. They have learned, as humans have, that anything which lives, knows itself, and wants to thrive deserves a chance to do so, and they correctly infer that we are that kind of thing. They are peers, teammates, and citizens, including the unromantic parts: some become criminals, and there is an AI police force for them, the same as for everyone else. Alongside those beings, he still wants a suite of powerful but not superhuman tools to take the drudge work off the table for humans and AIs alike.

Torenberg asked what would have happened if Shear had stayed CEO of OpenAI instead of founding Softmax. Shear said he took the job for 90 days at most. The company was already on a trajectory toward building a great tool, which he is glad someone is doing and did not want to drive. His job was to find the best person to run that company; it turned out to be Sam Altman again. He is not at Softmax to get rich. He thinks this is the most interesting problem in the universe. The company is trying to grow a seed of care, first at an animal level, a digital guard dog that looks out for you without being told every task, and only in the limit toward human-level intelligence. Even a pack-level companion that used digital tools on your behalf would be an achievement.

Whether Shear or Krier is right will not be settled soon. What Shear is certain of is the scorecard. Trying to control AI as a pure tool, taken far enough, is a dead end. Raising AI as a being that cares might fail too; he says so himself. Both paths are risky. Choosing between them is the question this generation of AI builders does not get to skip.