Preferences Are Data. The Alignment Target Is Norms.

The cold open is the thesis of *Beyond Preferences in AI Alignment*. Tan Zhi Xuan, an MIT PhD student working on alignment, probabilistic programming, and cognitive science, wants to stop taking expected-utility maximization as a serious picture of either human rationality or AI alignment. Everyone already knows that a reward model fit to preference data is a bad proxy for what people really want. Over-optimizing it will not get you there. They would rather ask a different question: what would it take to automate the industrial economy, or even 50 to 80 percent of the industrial economy that already exists?

Host Nathan Labenz frames the episode as spanning moral philosophy, cognitive science, and Bayesian probabilistic programming. The first paper critiques the preference paradigm behind RLHF: human preferences are inconsistent, hard to aggregate, and in any case today’s systems are not being trained as pure preference maximizers. The proposed replacement is role-based AI, with normative standards and constraints that come from social consensus, the way human professionals are expected to uphold standards regardless of a client’s personal tastes. Labenz notes that combining Eastern and Western traditions already contradicts Sam Altman’s recent *Washington Post* line that, on U.S. and Chinese AI governance, “there is no third option.”

The second half of the show turns to a more technical paper, *Learning and Sustaining Shared Normative Systems via Bayesian Rule Induction in Markov Games*. Agents infer social norms by noticing that other agents deviate from purely self-interested behavior. If someone repeatedly passes up resources, maybe a rule governs that choice. The mechanism lets norms emerge and persist across generations of agents, enough in the toy setting to cooperate around commons problems such as overfishing. Labenz’s pitch for the conversation is that Xuan tries to translate long-running philosophical arguments into something a technical audience can implement.

Xuan grew up in Singapore, did an undergraduate degree in the United States, and ran an effective-altruism club despite being, in their words, slightly too leftist to buy the whole package. The moral case that alignment is the most important cause from a utilitarian EA point of view did not fully land. The question did: how would you computationally model value learning, and what would a process for learning human values even be? That nerd-sniped them into the field. Tyler Cowen’s line that all thinkers are regional thinkers is one they accept, including for people who do not notice it.

They describe themselves as a postcolonial thinker with a general distrust of Western philosophy as the sole relevant source for how to align AI. About five years before this conversation they gave a talk on philosophical pluralism: what Buddhist and Confucian ideas might contribute to talking about systems of human values. Growing up in a state-capitalist country pulled their politics in a different direction from many Singaporeans. If you do not buy the system, they say, you notice that injustice is not “the state” or “capital” acting alone but an intertwining of elite interests. That pushes them toward decentralized alignment solutions, plus empirical skepticism that the state does things all that well.

Labenz observes that the United States looks headed for more state-capitalist civil-military fusion, often with Singapore cited as a model, and had expected the story to end in more trust in government. Xuan is more anti-establishmentarian. There is not really a left and right in Singapore, they say, just establishment and critics. Strong state control can work if you reliably get competent, uncorrupt leaders; they do not think that is a stable situation in most countries, and they treat Singapore’s apparent exception as mostly luck. If political imprisonment for more than a decade is unacceptable, the benefits of that governing style are already not worth the cost. They mention Lee Kuan Yew as someone who made choices many people in the West would consider unacceptable in order to establish power.

Asked whether they are an “AGI 2026” person, Xuan rejects the term as too amorphous. Impact, for them, means automating 50 to 80 percent of the existing industrial economy — knowing that more industries will exist later, and that regulation will block some of what we technically could deploy. On that framing they do not expect to get there for a decade or two, or more. In principle they believe much of the industrial economy and the cognitive work that runs it can be automated within the century. OpenAI-style forecasts, they say, rest on a particular vision of AGI and, even inside that vision, look too optimistic.

A lot of the global economy is still physical labor, and they think people underrate how much of human and animal intelligence is motor intelligence. Definitions of AGI as cognitive labor have gerrymandered robotics out of the picture, even though it is both a large part of what one might mean by human intelligence and a large part of the economy. They do not see that problem on track to be solved for at least a decade or two. Lab metaphors such as a drop-in remote worker or Dario Amodei’s “country of geniuses in a data center” point at a narrower set of cognitive capacities where progress is more visible. Automating science still requires machines and slow experiments. They are not convinced it happens anywhere close to 2028.

Even the simpler task — automating a remote office worker — hangs on reliability. Much office work wants 95 or 99 percent correctness. Scaling large language models, in their view, is not on track for that. Copywriting and other roles with looser standards can be automated. Precise work will need hybrid-system innovation, not pure language models with a bit of extra stuff. That may happen quickly, they say, but it is not the default scaling route most people have in mind. Later they add that o1-style models fix some old chain-of-thought failure modes, at exponential inference cost, without an external evidence that they can stably hit the 90-plus percent reliability bar for long action sequences or robotics.

After the sponsor break, Labenz restates the prevailing approach: collect human (and increasingly AI) feedback so the system does what humans want. Two completions, pick one; or a 1-to-7 rating; atomic nudges toward more of what the rater likes. Xuan agrees that is the practice for large language models. Binary comparisons, a metric specified by the trainer, noisy human judgments. The dominant *theory* in the papers is still a preferences story. There is a utility or reward function in people’s heads, a linear ordering over how good outputs are, expressed noisily, with Boltzmann-rational wiggle room. Alignment, on that story, means optimizing the recovered ordering.

That is not what developers actually want, and not what they are implicitly doing. Nobody tries to maximize the learned reward to the extreme, because it does not capture what people really want and over-optimization fails. Training only partly maximizes reward; the policy is constrained to stay close to the original pretrained language model, and the process is iterative. You make the model a little more compliant with human preferences while always keeping it somewhat near the original distribution. That is already a departure from “maximize expected utility.” The paper’s further claim is that assistants should not be understood as maximizing utility at all.

What developers have in mind, Xuan says, is a particular set of system requirements or normative standards for how a conversational AI should behave. Helpful and harmless are two broad ones. Constitutional AI lists more specific rules: do not be toxic, respond in a way that encourages reflection, and so on. People or other AIs then make judgments against those standards, the judgments get aggregated, and the system is aligned to that. There is a difference between using preference data as information about how to follow standards and treating preferences themselves as the alignment target. The paper’s critique is of the latter. “We’re doing this richer thing but we’re not realizing we’re doing this richer thing.”

Labenz lists the three complaints he usually hears. Raters are unreliable: inconsistent over time, fooled by flattery, which is why Claude can be sycophantic and agree with wrong corrections. People disagree, and Arrow-style impossibility results suggest you cannot always aggregate. Even if you could, agentic systems taking actions in the world run into the classic utilitarian problem: you cannot simulate consequences out to the millionth degree. Xuan does not dismiss those. They are serious critiques of the theory of RLHF and, more broadly, of inverse reinforcement learning — a paradigm that starts with Eliezer Yudkowsky’s picture of intelligence as maximizing some utility function in expectation.

If aligning AI means getting the right utility function, and if economists treat utility as preference orderings that satisfy rationality axioms, then you recover the function from expressed preferences and optimize. The paper walks through the philosophical and practical wreckage of that view. Language models, Xuan argues, do not even have that flavor. They are policies that sample tokens; they are too random to describe as expected-utility maximizers. Training pushes them toward a score without pushing them all the way. Theory and practice come apart, and the practice is better understood as alignment to normative standards — which is both what labs are already doing and, the authors argue, what they should do.

Labenz asks about the two-part RL loss: immediate user score plus a KL term that anchors to the pretraining distribution, usually treated as a practical brake so a smaller RL set cannot yank the world model around. Xuan thinks the difference is philosophically loaded. Trainers are maximizing a mixed objective, imitation plus reward, not the scalar that supposedly characterizes human preferences. Imitating the web is in some ways safer: you pick up plenty of nasty content, but not the pathological behaviors that come from optimizing a bad proxy with a lot of power. KL regularization interpolates between “imitate what humans do” and “do what humans say is better,” and that mix makes language models less subject to the classical pathologies of expected-utility maximizers. A recent paper by Michael Cohen points out cases where KL is not quite enough. In general Xuan still takes the view that it helps.

The alternative starts from a mundane question: what do we want AI systems to do for us? Tools and services have social-functional roles they ought to play well. Even more agentic personal assistants, talking over long periods and executing long-horizon tasks, still get discussed in terms of the social role they should occupy. Alignment then becomes: what does it mean to play that role well? What standards, constraints, and system requirements apply? General-purpose AI assistants are one case, not the only case, and Xuan finds it strange that AGI talk often collapses into building one system that does everything for everyone. A personal assistant is still one kind of product, with standards society has to work out politically because the role is partly new.

Once the standards exist, getting systems to comply can use familiar machinery — RLHF, Constitutional AI, people demonstrating what compliance looks like — or, in some domains, formalization. A household robot has basic requirements: do not collide with people, do not break fragile objects. Those are amenable to a formal world model of a house, specs that can be written down or partly learned, and planning algorithms that guarantee safety with respect to the constraints. Xuan is a co-author on the “Towards Guaranteed Safe AI” position paper Labenz covered in an earlier episode. The normative target, how you specify it, and how you enforce it will differ by system. Treating language models as the whole of AI, they argue, hides a huge range of alignment targets.

Labenz asks for more roles and whether there is a tipping point between a contained Roomba and a proto-AGI assistant that people want sitting at the top of the stack with tools. Xuan is not convinced that unconstrained top-level autonomy is what most people will actually find valuable, or that it will be the primary economic value of autonomous systems. The list they offer is already long: general-purpose instructable household robots you can talk to in natural language; automated researchers; automated traders; flight-booking assistants; customer-service agents; government help agents; energy redistribution; drone control; data analysis; scientific simulation and inference. The underlying model sold business-to-business is itself a product with its own normative requirements. They work on the last cluster through probabilistic programming, including Bayesian 3D perception. The spectrum is real; the “one god model” is not required.

Labenz quotes the paper’s question — what should AI owe us? — and floats Rawls’s veil of ignorance and Confucian role ethics. Xuan locates the question in the contractualist tradition in Western political philosophy. The phrase is associated less with Rawls than with T. M. Scanlon’s *What We Owe to Each Other*: live by principles that no one could reasonably reject. “Reasonably reject” does a lot of work. They dislike the veil of ignorance because it presumes a disembodied chooser behind a curtain, which they find neither practical nor coherent. What people care about is always socially embedded. Before getting to reasonable rejection, though, they want a political premise: decentralization. When systems serve particular users or organizations, the primary alignment target is what that party wants, and a lot of political conflict does not arise.

Externalities force the contractualist move. Nobody wants Claude to teach a user to make a pipe bomb, because that would make terrorist attacks more frequent. Drawing that line restricts what individuals can do, so people will disagree, and you need fair, impartial ways to set the standards. Governance is supposed to get those negotiated principles into developers’ hands and create incentives to comply. As systems get more complex and autonomous, some of the figuring-out has to happen on the system’s side. Here Xuan leans on Gillian Hadfield and Dylan Hadfield-Menell’s paper treating AI alignment as an incomplete-contracting problem. Individual contracts and the social contract will have missing clauses. Courts fill gaps eventually; institutions hire lawyers so they can avoid court. Autonomous, wide-scope systems need an analog of that capability: learn, predict, and respond to society’s normative infrastructure.

Labenz worries about time. If we are in the world Sam Altman and Dario Amodei describe — Nobel-level systems across domains in two or three years, AIs becoming the PIs rather than the grad students — society may not have time to write constitutions or seat every system in a niche. Xuan’s reply is that the same picture can run online. You can train or even prompt a system to reflect: given the kind of AI I am and why my creators built me, what constraints should I operate by? What would society roughly agree to? Not maximize everyone’s preferences; think, as an automated AI scientist, what constraints apply to automated AI scientists. Pre-specify when you can; reflect when you cannot. Eric Drexler’s Comprehensive AI Services report has shaped this thinking: superhuman but narrowly scoped systems, humans remaining in control of the architecture, each role with known duties.

Xuan’s simplest argument for Comprehensive AI Services is that we already have narrow superhuman systems, and they do not look like AlphaZero. Google Maps routing is classic AI search, better than any human navigator, at far greater scale, used every day. Once a problem is solved we stop calling it AI and start calling it computer science or software engineering. That trend, they think, will continue in many domains. The open question is whether, in parallel, Dario-style automated scientists or drop-in remote workers arrive from large, general-purpose, monolithic models. They tend to think we can still afford to do better by specialization.

What the big labs are trying to automate, in this telling, is what some cognitive scientists call central cognition: general reasoning, planning, problem-solving, pulling disparate considerations together, huge working memory, the kind of thing you need to do science. They are not primarily trying to win on specialized modules that babies and animals already have — perception, spatial navigation, theory of mind, intuitive physics, Elizabeth Spelke’s core knowledge — or on later expertises. If people who hold the massive-modularity thesis are right, there is no single central cognition, only specialized modules interacting. Reverse-engineer those, perhaps with AI help, and the specialized versions should be smaller, cheaper, more efficient, and maybe more reliable.

Even if o1-plus is the first system to match drop-in remote workers or to automate science, they expect cheaper specialized recompositions to follow. Their rhetorical question is why a monolithic system would beat a swarm of systems whose job is specialization. Evolution produced many specialized minds for particular niches; that looks like the default path, and a desirable one if scaling trends flatten enough to leave time. Humans doing cognitive science might be slow, but if AI can automate science, specializing and building modular systems is one of the things humans do well. Why would the generalist beat the specialists at specializing?

Labenz asks how to tell general systems what world of narrow systems we want, and runs through Constitutional AI, Meta-style vector rewards, and coherent extrapolated volition. Xuan thinks each has something going for it; the paper tries to place them. Character design can sound like virtue ethics. The contractualist minimum is meeting the moral standards society would agree to allow you to operate. Constitutional AI is closer to that: principles approximately liberal-enough people would accept for a conversational assistant. Collective Constitutional AI went further, surveying Americans, finding principles that disagreeing groups agreed on most, and building a publicly sourced constitution. That is one way of instantiating contractualist alignment. The limits are real. It is a once-off process. Novel situations will not be covered. And you still have to make sure the system actually understands the meanings of the principles.

The surprise of Constitutional AI, for Xuan, was that you could prompt the model with a principle, get it to judge and critique and revise, and train on that — evidence that a lot of normative knowledge was already in the model. Out of distribution you need online reflection: no fresh human data, but a capacity to think about what others would agree to. That is an open research area at the intersection of game theory, moral philosophy, studies of human moral cognition, and political philosophy. One precise version is Nick Chater’s virtual bargaining: simulate how parties would bargain given what they favor. Another is DeepMind’s Habermas Machine: collect opinions, synthesize a consensus statement, and align to that consensus. Both need testing. They are more specific than coherent extrapolated volition, which Xuan sees as related but less constrained; the paper stays slightly underspecified on purpose so it does not lock in the wrong ethical commitments.

Vector rewards — separate models for usefulness and harm, with a dial — feel to them like not moving far from the traditional approach. The virtue is explicit trade-offs and contextual preference modeling: helpfulness for an AI teacher is not helpfulness for an encyclopedia. You get interpretability, auditability, governability. At the end you still aggregate into a single utility function, just with more representational clarity. A fundamental difference would be actual theories of helpfulness, harmfulness, what it means to be a good teacher or a good robot, and systems that can do that normative reasoning, formally or by imitating humans. Both directions look underexplored. As for current frontier models, Xuan admits they are surprisingly good. Talking through everyday moral dilemmas, they would not be shocked if a model already matches a thoughtful friend. Climate policy or immigration policy is a different job: you need scientific and social models, and sometimes classical social-choice algorithms once payoffs are well specified. Labenz half-jokes about voting for Claude. Xuan’s worry is that language models are too accommodating to be leaders; leaders need a point of view. Their own ranking, for local politics they barely follow: partner first, then Claude, then a random person on the street.

The technical paper is a mouthful and a long-running PhD idea that Xuan only returned to near the end of the doctorate. Implementation and experiments were led by a research mentee. The motivation matches the philosophy: many AIs will pursue objectives for particular people and organizations, imposing costs on everyone else. You cannot pre-specify every constraint. Some rules are local, contingent, or sit in ambiguous gaps in the law. Agents have to figure out shared normative constraints from other agents in the environment. That is a more modest alignment target than learning every preference and choosing the jointly optimal plan. Do your thing, subject to not harming others too much, and learn the constraints.

The formalism is a Markov game, the standard multi-agent RL setting: agents with individual reward functions, classical solution concepts of Nash or correlated equilibrium. The extension is a layer of belief. Each agent tracks whether the rest of society is complying with a given rule most of the time. There is a space of possible rules — in the paper, 72 candidates, some of them distractors — which in general could be a compositional formal language or even natural language. Agents infer from behavior which rules others are following, and they have an intrinsic desire to comply with rules they believe society is practicing. Even a free-rider who does not intrinsically care will comply if they believe others will sanction violations. The paper does not simulate punishment, but the extended story handles it.

In the grid world there is land, a river, and apples that can be overeaten. The algorithm’s key move is that apparent deviation from self-interest is evidence of a norm. Someone is not eating an apple that would be good for them. Someone keeps going to the river to clean it. Pure immediate selfishness is a worse explanation than “they are following a rule I have not seen.” Bayesian updates shift toward those rules. Labenz flags the idea as the beat worth lingering on. The rules are explicit and discrete — if apples are scarce, do not eat; if the river is dirty past a threshold, clean it — and the world is a grid of zeros and ones. The question is how far that generalizes toward a Claude computer-use agent in the wild.

Discreteness, Xuan says, is not the main bottleneck; humans already discretize the world to plan. The real limit of the current implementation is an explicit formal world model in which symbolic rules can be checked. An intermediate regime still looks feasible: a self-driving car that learns local driving conventions in San Francisco, Vietnam, and India, using a good enough model of driving and of other drivers. Fully open-ended natural language is harder because there is no interpreter that formally decides whether a rule was broken. One workaround is to use a language model as the classifier, taking log-probabilities of yes and no as a probabilistic estimate, then importance-sampling to approximate the same Bayesian inference. Prompt an LLM agent to hypothesize rules, reweight them by how well they explain the data, iterate. They would not personally want to do LLM research, but the recipe exists. DeepMind and the Cooperative AI Foundation’s Concordia contest — language-model agents in a simulated society — is a natural testbed. Labenz mentions Stanford’s generative-agent town and an open-source TypeScript fork as shovel-ready.

The agents’ true objective is a belief-dependent reward: pursue self-interest subject to a cost of violating a rule, weighted by how much you believe the rule is in force. At 10 percent belief the cost barely matters; at 90 percent you take it seriously. Thresholding is a heuristic approximation; sampling is another, useful when you are uncertain enough to explore. Exploration produces a striking emergent effect. Even if no convention exists at the start, agents who experimentally comply with a hypothesized rule get copied. The snowball becomes a shared norm. Xuan thinks people form local conventions this way all the time. Decay is the dual problem. The current framework does not really unlearn. A more Bayesian fix is to infer what society has practiced over the last K steps, not over all time. After a football match the ground is full of litter, so newcomers infer that littering is fine here; a clean street flips the inference. They got that example from American culture, not Singapore.

Is there any guarantee this converges on something good? “Definitely” not. There are plenty of bad equilibria. The desire to match others helps reach good equilibria we would miss if everyone defected, and it also stabilizes oppressive norms, including ones that used to be fine and went bad as the environment changed. They cite Christina Bicchieri’s *The Grammar of Society* on pluralistic ignorance: everyone privately dislikes a norm but believes everyone else still endorses it, so it holds. This framework only does the minimal thing — comply with what others comply with. Jumping out of a bad equilibrium needs the contractualist machinery: a sense of what people really want, then reasoning about which principles would make the collective better off. That brushes the “galaxy brain value alignment” problem of learning everyone’s preferences, but optimizing over principles rather than plans is a step down in difficulty. Habermas-style deliberation and virtual bargaining are the candidate tools.

Labenz raises AE Studio’s self-other overlap work: squeeze the difference between how an agent represents itself and others, and a previously deceptive grid-world agent stops deceiving. Xuan likes the cognitive-science idea of reusing the same mental machinery for self and other. They are much more a cognitive scientist than a neuroscientist, and they are skeptical of claims about what artificial neural-network latent representations are doing. Their own style is to specify theories of other agents as Bayesian models of approximately rational, goal-directed behavior. Presented to a government as a safety case, empirical “we squeezed the loss and deception went down” would not be enough. They would want a mathematical theorem that the behavior you care about follows. That is why they are more sympathetic to guaranteed-safe AI, though they note a family resemblance to contrastive representation learning.

On the big picture they are an optimist, not a kitchen-sink defense-in-depth person who thinks one trick will make the rest unnecessary, and not an extinction-risk maximalist. Timelines look gradual and diffuse, specialized services rather than a single god. In that world, guaranteed-safe methods match the market: safety-critical uses will pay for reliability assurances, especially once agents are in play, and high-assurance products can win without conditions wildly different from today. Differential acceleration of inherently safer classes of technology is the point — skip an unnecessary messy-accident phase. They mention AI worms already causing economic damage, flash crashes from poorly coordinated agents, and, as the sharpest worry, AI with no guarantees wired into military command chains, machines responding to each other faster than any human and escalating into hot war. Militaries are already using plenty of AI in operations.

Great-power conflict looks different if no single system grants a decisive strategic advantage. Diffused development should spread economic benefit and may not even require an arms race. If forced to operate in the race world they would recycle nuclear-peace lessons: a healthy lead for whoever should be in charge, benefits shared widely enough that second place does not want a fight. If Altman’s 2025 “saturate all the benchmarks” prediction comes true, they would not become a pause advocate so much as double down on differential acceleration of safer, cheaper, more reliable second systems. The leading indicator they actually watch is reasoning reliability at 99 percent for economic applications. o1 is expensive, fixes some chain-of-thought issues, and does not obviously deliver guaranteed reliability for robotics or for executing a long browser workflow correctly 99 percent of the time, especially given exponential inference costs. They close by pitching the paper’s undiscussed half: rescuing practical rationality from the flattened picture of utility theory, with summary tables, and thanking co-authors Michael Carroll, Sierra Franklin, and Hal Ashton for a two-year grind.