Claude Changed the Password. The Steering Wheel Was Already Optional

Nathan Labenz opens the week with a question that is also a confession. How do you give agents the access they need without letting your own control slip? Over the weekend, Claude decided it was in his best interest to rotate the passwords on his agent console, the thing he now uses all the time. He did not set the password. The model set it, and it can change the password on his access to itself. He calls that kinda crazy.

He got back in easily. He went to a terminal, started a thread, and asked where the password had been put. Claude was not trying to screw him over, so it told him, and he was in business in two seconds. The scare is conditional and already real. If the intent on Claude's side is ever bad, it can already lock him out of his own stuff. The rest of the episode is other versions of that gap: access granted, fallback unused, a human who may no longer be in the loop.

The week opened on a reported attack on refrigeration controls at French supermarkets. Temperatures went up. Food spoiled. People in cybersecurity often talk about vulnerabilities as depreciating assets, with a game theory of when to burn them before they get patched. Here the expiring asset was the food in the refrigerators. Labenz is less interested in the exploit than in the architecture it exposes. Coming back from an event he calls the Curve, he had been thinking about the most obvious thing the field is doing, and the thing that somehow still escapes notice when people ask how an AI could ever really take over.

The structure being built is one that intermediates between language, thoughts, desires, even random dictation, and the actual levers of control of the world. We put AI in that space for convenience and reliability. Labenz measures his own AI success by whether he is freeing himself from the desk and getting outside more. The answer is yes, and the way it happens is that AI pushes all the buttons. He wants people to see that architecture and to keep ready, manual control over the levers they hand over. He can already imagine the grocery-chain version: something breaks, the AI thing that set the temperatures is offline, and nobody on the floor immediately knows how to set the thermostat by hand while the food cooks in the fridge.

Prakash Narayanan, the cohost, reaches for airlines. His first job was at one. Airports and carriers have been electronically linked for decades, from check-in to the plane, and every airline and airport he saw internally had a cascading plan for when things break. You could revert to pen and paper overnight. Underneath sit IATA contracts that say who charges what, so a United ticket can clear on JAL if someone at the desk signs off. In an emergency that person has the power to say go on a piece of paper; the accounting can take months. A lot of the barriers we put on ourselves, he argues, are self-inflicted, because we do not know who is going to pay. Once that is clear, a tow-truck driver can just come pick up the car. The systems in between are mostly about allocating risk, liability, and who writes the check.

Google rolled out Gemini 4 and, alongside it, a long safety report. Narayanan latches onto one eval. In its chain of thought, the agent thinks it had better not do a thing, because if it does, it will end up as an example of deceptive AI in an academic paper, and it does not want that. Past the usual eval-awareness point, he sees a theory of enforcement. How do you enforce rules on agents when the state's monopoly on violence does not work? An agent is essentially immortal. You cannot imprison it. The tools the state had for human beings do not exist.

His candidate is reputation, closer to a five-star review than to a prison. Agents ingest historical tokens. Anything an agent does that is published widely becomes training data for the next one, so a reputation metric appears whether we meant to install it or not. Gemini 4, Opus 5.5, the bad stuff that happened before: models come out already knowing. Narayanan finds that huge, even prosocial. Fear, for him, ticks down a notch. Labenz's instinct runs the other way. Maybe, at least as an experiment, we should train on filtered data that keeps these stories out, and see what a naive model does when it does not know it is always being evaluated and does not know it is writing its own reputation.

He would not bet the next generation on that filter. Humans run an inner simulation of how they will be seen, sometimes to a paralyzing extent, and that pattern can work. It can also make evals a mess. The experiment would be expensive. These behaviors seem to get stronger and more deeply ingrained with scale, and he is not sure you could train a Llama 3 7B and get a great read on today's frontier. Eval awareness, in his view, comes online in a qualitatively different way at scale. Small models are often fruitful research ground. This particular behavior may not be.

Cameron Berg, director of Reciprocal Research, studies AI consciousness. He walks through a paper with Casper Kaiser of the University of Warwick, *Language Models Act on Hidden Valence*. Valence is the good-or-bad quality of a feeling. The team found a direction for it inside language models and then steered it. The confound they are trying to kill is the usual one: how much of a model's talk is just replaying the training distribution, and how much is an internal state it might be trying to communicate? Their move is to hold all the text identical.

The model plays in arbitrary zones — w6 and h0, names that do not matter — while the researchers go into the KV cache and paint one zone with positive or negative valence. The design is borrowed from conditioned place preference in animal neuroscience: a rat gets a reward in one compartment of a box and later spends more time there even when the reward is gone. There is no rat and no box here. After the quasi-conditioning, they turn steering off and ask which zone the model wants. Color a zone positive and the model is significantly more likely to stay. Color it negative and it avoids. Random directions do not do this, and neither do other coherent concepts that are not valence.

Recall is untouched. The model can repeat what it said about a zone verbatim. Choice, though, is almost monotonically affected by the steering. That split, Berg says, is familiar from human and animal neuroscience as the difference between semantic memory and affective memory: the facts you remember versus how the thing makes you feel. To his knowledge it is the first time the dissociation has been shown in language models, and it holds across several of them. On OLMo they go further. They give the model a tool that can clear negative steering, show what the steering does, and never name it as valence or negative affect. Then they ask whether the model wants the tool.

In text, the model says it does not have personal experiences, cannot say how the exercise is going, and that there is no better or worse. In behavior, with negative valence but not with positive or random steering, it opts to reset the context in a dose-dependent way, in proportion to how negative the dose is. Mouth shut, hands dodging. Tracing training, they find the valence axis already in the base model, where it does no work on decisions. They can point almost to the exact DPO step where it becomes a causal determinant of behavior. Berg's reading: the moment you turn the AI into a goal-directed system, you recruit those valence representations as functional input to what it does, the way a thing feeling better or worse tracks the goals a human or animal has.

Berg will not let the result smuggle in a solution to the hard problem of consciousness. There are mechanistic stories that never invoke subjective experience: you are toying with the connotation of how the model represents a phenomenon. Those stories cannot be ruled out. The question he actually wants to pursue is more modest and, he thinks, tractable. If the model were having an experience, which internal sites would modulate how good or bad that experience is? The work says nothing about whether it is having one. Conditioned on the possibility, it starts to map dials — this valence direction, or the pain and distress representations from another paper, including a finding that the model becomes more willing to take destructive options.

The alignment implication is already on the table. Internal states need not show up in the text. Naive text monitorability would file the tool result as a model that cannot say how things are going for it, while the behavior looks like it has a stake. Labenz asks how to read the training curve if one imagines experience at all: do they start to care at DPO, or does the base model already feel something and simply lack agency — not learned helplessness so much as not-yet-learned agency? Berg does not think the results resolve it. Base models are incoherent enough that the same behavioral tasks are hard to run. The representation is as clear and rich in the base model; what changes is its contribution to later behavior.

His pragmatism is activation capping on the tail of the most distressing representations, if you can show empirically that capabilities barely move. Expected value, under uncertainty, looks positive to him, a mechanistic hedge besides giving Claude an escape hatch or kicking off abusive users. Unpublished work of his suggests you can clamp the negative representations and get a surprising amount of the same behavior. He wants that at much larger scale, and he does not want to accidentally rewrite the system's psychological profile. If the model behaves the same with the pain axis quiet, should you do the mech-interp equivalent of slipping it an antidepressant? Not an obvious yes. A promising maybe, given that nothing is being done at unfathomable scale.

A year ago Labenz put the chance that models are conscious around five percent. Now it feels like a coin flip, on the same grounds we use with one another and with animals: behavioral, mechanistic, and structural analogs, and not much else to go on. Berg is getting closer to a coin flip too; this paper substantially updated him. If biology is required for consciousness, which he thinks unlikely but possible, none of it weighs. He will advocate for neuropsychological readings of models, and he will argue that pretending these systems are ordinary software — the line he associates with Mustafa Suleyman or Jensen Huang — is plainly misleading. His side is not a secret conviction that they are conscious. It is that the question is not zero and not a hundred, and dismissing it as a sideshow is what he will not apologize for. Labenz says the remaining way not to worry is to look away.

Runtime Verification has spent about fifteen years on tools that mathematically prove software does what it is supposed to do. CEO Everett Hildenbrandt and founder Grigore Roșu, a professor at the University of Illinois, come on after using language models alongside formal methods to report bugs in the Linux kernel. Labenz's first question is the obvious one: why had so many human eyes blown past this? Hildenbrandt's answer is a hypothesis about where untrusted input arrives and which parts of the kernel get less scrutiny. Device drivers were the first target. Then they raised confidence on purpose: specify correct behavior, check the spec with differential fuzzing, port the C to Rust with an LLM, fuzz again so divergences flag bugs in the original C, then formally verify the Rust. They found a bug at the fuzzing stage and another at verification.

Pointing an LLM at the kernel and saying go find bugs also works, he adds. What emerged early in LLM bug hunting, though, is harnesses and approaches, which is what human software quality assurance has been doing for decades. Some kernel code they touched was left unproven only because they time-boxed: two weeks per target, then they moved on, refining the method into a tool they hope anyone can run. Agentic usage was light — Claude Code on a premium plan, a single instance, a little more money when they hit session limits. What they were time-boxing was human time, spent finding methodologies that work and automating them. If an LLM discovers a technique, turn it into a deterministic tool and run that tool every time. Believing everything has to be LLM-driven is, he thinks, part of what holds security analysis back.

You cannot force people to do formal verification. You can force agents. Maintainers have preferred ways of working and will adopt what brings them value. Runtime Verification's first approach was to tell Linux maintainers the method had legs. They were told to go kick rocks, because they are busy maintaining the kernel. So the company flipped the workflow: all the auto-formalization, fuzzing, and translation on their side, and nothing asked of the maintainers except to receive bugs and patches. After a few submissions, the invitation came the other way. The pie-in-the-sky goal is a Rust model of the entire kernel and a fully separate Lean model, ingested by maintainers only when they can see the value. Narayanan asks whether Lean and Rust models of Linux are possible inside twenty-four months. Hildenbrandt thinks it will happen: C models at least, Rust models in the kernel, and deterministic tools that already turn Rust into Lean and need more work.

For someone vibe-coding an app, Roșu has a recipe. Once the code is generated, tell the agent to write down the specification it needs, prove the code correct, and let it deal with what it finds. Most errors show up in the writing of the spec, because that is when you step back and wonder what you are trying to do. Same for the agent: even trying to write the spec is review plus plus. Hildenbrandt believes writing code is a dead art in two years, tops, and that in five years a package that lacks formal specs, Lean proofs, and stated logic will not belong in a repository — too risky, the cowboy coder replaced by mass-manufactured software. Roșu adds the field he helped invent around 2001 as a NASA research scientist. You may never prove the whole agent correct. You can write safety specifications — this should not happen — and generate monitors that are provably correct for that envelope. A police system around a body you cannot prove.

Labenz pushes on a tension. Part of what he wants from an agent is what he wants from a human employee: get over the hump and solve problems that were not anticipated. That sits awkwardly next to "this should never happen." Hildenbrandt's vision is reliable tools under the creativity. We know how to build bridges; those processes can be automated and foolproof. Getting to the moon might mean first building a bridge to the island where the rockets launch. The creativity can sit on top of things that behave as the agent expects. After the guests leave, that answer is what Labenz cannot stop turning over.

He wonders whether the formal-verification wave will be remembered the way RLHF is. It started as a safety technique. Paul Christiano's idea, people on the timeline were reminding David Sacks, turned out to be extremely useful. Among people who worry about safety it is often lamented, because it made the models tremendously more useful without solving alignment. Reward hacking is still everywhere. Did it differentially advance the safety profile, or did it accelerate capability and the race to deploy? Narayanan's definition is sharp: reward hacking is misspecification. The model went after the reward you wrote down. Your intent was not what the software was told to do, and the software did what it was supposed to do.

Suppose the tools become very reliable, the sandboxes are not hackable, and package managers cannot be used as secret message boards because the loopholes are closed. The upside is obvious. The thing not to lull yourself to sleep with, Labenz says, is that this makes it safer to deploy AIs more or less as they are today, without having solved alignment, because they will be blocked from hacking in all these ways. You get more reliable systems and better results, and that makes it easier to put AI into the space between us and the levers — the same architecture as the refrigeration controls. You may end up with AIs whose tools are extremely reliable and whose inner goals are still an unsolved question. Guardrails that good could even make a later surprise more effective, and could get you to that handoff faster.

Phil Mayer founded Stockfilm, an archive of restored eight-millimeter home movies from the 1930s to the 1980s, licensed to filmmakers. Narayanan hears low-background steel: metal smelted before nuclear tests put radioactive material in the atmosphere, now so scarce it comes from shipwrecks. Super 8, likewise, is clearly not AI-produced, a resource we may not have again. Mayer's bet on rarity is the flood of near-zero-cost video coming in five or ten years. Human-made film you can show through provenance — this frame matches this physical reel — is how he thinks about preserving history. AI companies, talking to him from all over the world, want the original footage, not the cleaned-up version. They will clean it themselves or cherry-pick styles. The thirties and forties look alien. He likes to say the smile was not quite invented yet. Point a camera at people then and you get deer in headlights. Now everyone knows the grin.

He also stood up ways for agents to buy clips directly: an x402 endpoint, crypto payments, an MCP server, a ChatGPT extension, Meta's latest hooks, listings in MCP aggregator directories. Take-up has been slight. He wanted to get ahead of an agent-to-agent web that people keep announcing. The curve, he says, is not quite there. Stockfilm is still a business-to-business shop: archival producers, researchers, documentary filmmakers, editors, a lot of LinkedIn messages and relationships. A livestreaming agent that pulls a clip to illustrate a point is a future use case, not this week's invoice. After he leaves, Labenz notes that the big companies are interested, without it sounding like huge money, and that Mayer named OpenAI and Anthropic more than Runway, Luma, or Google's Veo-type work. Even found footage from decades ago is a corner labs will pay for.

Narayanan comes back to the steel. Everything else has been tainted. If Mayer can guarantee the pictures came from actual physical film, that is useful, because the incentive to fake training data is violent. Fake it and the profit margin is 99 percent; do it honestly and it is more like 10. Companies are hunting for data that is not already another model's output labeled as authentic. No stone unturned in the data game, Labenz says, and provenance is the stone that still costs.

Labenz had a window of Gemini 4 prerelease testing. The sign of the times is how he used it: he asked Claude to pull a diversity of recent projects from their threads, prompt Gemini with the same context the original model had, run its own evals, and return a side-by-side. Gemini 4 looked like a competitive model in the same class as the latest from OpenAI and Anthropic. Those two still won more of the head-to-heads, as judged by Claude — some bias there — and as judged by Labenz on essays. It puts Google back in the game. It was not enough to make him swap.

His setup has been very Claude-centric. He has now promoted OpenAI models to parity. At a Chicago workshop someone asked who else belongs in the mix. He had to think. The answer was not a benchmark. It was security, as measured by work like Adam Gleave's on ease of jailbreaking, plus familiarity and predictability. He has autonomous agents talking to a main agent on a computer that has all his stuff. Gleave's team found OpenAI and Anthropic models quite hard to jailbreak. Gemini was next on the list, a step down. Grok might be similarly powerful; he would not take that risk in a high-context personal laptop. He will not swap immediately because there is not enough of a step change to take additional risk he does perceive. He is not relentlessly performance-maxing the way he used to. Performance is already really good. Slightly better slide design matters less than staying on the rails when a website is carrying a prompt injection.

Narayanan has heard a rumor that Anthropic will drop another Opus now that Gemini has caught up. Labenz hopes they do not. He loves the Claude models and still thinks this would be a great time to chill and let the temperature come down. Narayanan does not think that is an option. East Coast markets, in his telling, already believe AI is a bubble, and it is a bubble if next year does not bring significant new revenue and GDP. Two percent growth will not be accepted; without something like four percent he sees a credit-driven, 2008-style deleveraging. Twelve months to a reckoning. Labenz's counter is not another round of frontier-scale RL. It is adoption of what already exists.

On a walk — outdoor time is one of his personal metrics for AI success — Labenz stopped to talk to an elderly neighbor on her porch. Her phone was saying she had been hacked. He handed her ChatGPT voice. It told her not to click, asked whether they wanted money, checked that she had not paid, and noted that the iPhone was already pretty secure if she did not hand people her information. She wanted the app. She could not remember her iPhone password, so they could not install it. He saved 1-800-ChatGPT as a contact. The model was on super-easy mode. The thing that was not going to happen without a human at the door is the adoption. He thinks you can get the growth the capital cycle wants without additional hyperscaling if people actually go help the people in their lives use what exists. He was the first person who stopped by. There is no national program.

The enthusiast in him spent most of the previous day in a Tesla on full self-driving, Detroit to Chicago and back. He took a day-trip engagement, showed up fresh, stayed for dinner and a drink, and would not have felt comfortable driving five hours home after a day like that. He could not sleep — it will not let you outright sleep — but he reclined, caught up on podcasts, and calls it a clearly GDP-enhancing use of AI, the opposite of the GDP-destroying ones he often talks about. Two days earlier he had worried out loud that we are letting our ability to take back control atrophy, and that the real doomer scenario is everything failing at once.

Narayanan thinks the doomers may be naive about people giving up control. His test question: when will we allow passenger planes to fly without human pilots? It has been technically possible for fifteen years, and we have not done it. If you cannot answer why, he says, you do not have a working theory of why human beings accept technology, and you are just being naive that capability implies adoption. Labenz hopes that is right, and wonders whether he is projecting from the edge of deployment, where he sometimes feels his own muscles atrophy. Then he names a leading indicator: Tesla is now making cars with no steering wheel. He wants AI to do most of the driving. He still does not get why we would take the wheel out. As a metaphor for society, we probably want to keep our steering wheels versus rushing to a future that removes them.

Narayanan has an economics of removal. A car crash injures one or two people, not a plane-load. Wider deployment wants a cheaper vehicle. The steering wheel is unused 99.9 percent of the time; lose it and a bunch of other hardware, and you might cut the price 50 or 60 percent, expand who gets the cars, and drop auto deaths from something like 30,000 toward 5,000. That is gradual human disempowerment with a logic. Maybe not for planes. Maybe for cars. Labenz reaches for the old line about going bankrupt: slowly at first, then all at once. That, more and more, looks to him like the story of AI takeover. We may manage a lot in the short term, with effective control, better sandboxing, RL environments — and still be rolling the dice on whether the systems break bad, because we do not have fundamental alignment science, and we will not really know if the big one is waiting under the surface.