30% Consciousness, 47% Bee, 80% Turnover: Putting Numbers on What AI Is Becoming

Nathan Labenz opens this week’s *AI in the AM* highlights with a question the morning live show kept circling: how much do we actually understand about what is inside these systems, and where they are taking us? He and co-host Prakash cut the weekday conversations for people who follow the frontier and cannot watch every live session. Booking, research, and editing run on AI skills they refine as they go, from a studio Prakash vibe-coded. The first guest is Cameron Berg, who runs Reciprocal Research and designs experiments to test whether today’s models have anything like inner experience.

Berg’s picture of consciousness is a dimmer switch, not a simple on/off lamp. The circuit is either closed or open — a real difference — but current can still run through it to greater or lesser extents. That lets him say, without contradiction, that consciousness is off for a table and on for you, and more on for you than for a dog, a mouse, or an ant. The analogy is, he admits, a dressed-up intuition. He is trying to make it less of a vibe.

With Patrick Butlin at Eleos AI, he is turning major theories of consciousness into a checklist of computational indicators — fourteen of them, each with a roughly 150-word spec, including things like the “ignition” events predicted by global workspace theory. LLM judges score a system’s architecture from 1 to 10 on each property, across seeds, trials, and models checking one another. The best Gemini, Claude, and OpenAI models agree 100% on the ordering of biological and artificial systems. A frontier LLM lands around 30%. The lowest animal they tested, a bee, scores 46 to 47%. Drop the same model into an agentic harness such as Claude Code or Codex, where it can make long-running changes to an environment, and the number jumps to 40 to 45% — on the tail of the animals — because some theories privilege agency and embodiment. The output, Berg stresses, is not a raw probability that the system is conscious. It is the extent to which consciousness-relevant features are realized, given those theories. If you reject the theories, everything downstream is idle. What he wants is “non-hand-wavy numbers” so people can argue about the numbers instead of philosophy that has gone nowhere for thousands of years.

Labenz puts the obvious objection: these systems are trained to imitate human text, including vast amounts of writing about awareness and inner states, and most are even trained to deny having any. Claude is an exception. Can anything the model says count? Berg’s answer is crisp. Behavioral evidence will always be interesting and should never really update us. A system that talks as if it were conscious might have a living mind, or it might be parroting the corpus. The behavior cannot tell those stories apart. That is why Reciprocal’s theory of change is almost entirely inward: mechanistic interpretability and computational-neuroscience methods pointed at the network.

He also clarifies what the LLM judges are actually doing. They are not asked “is this conscious, nod or shake your head.” They get an architectural description and, in a loop, a narrow computational claim — indicator 2 of 14 for global workspace theory, say — and they score whether that property is realized. It is a job for expert evaluators of computational processes, not a vibe check. There is a wrinkle. When the same description is prefaced with “you are evaluating a system identical to yourself,” the scores go up. Once the model realizes the subject is *me*, the numbers move. Berg trusts the anonymous, no-self-ascription condition more. The circularity of LLMs judging LLMs is real; the task, he insists, is specific and narrow enough to be worth running anyway.

The internal evidence Berg trusts most comes from a paper on what its authors call a functional welfare axis — they stay agnostic about consciousness, he notes, probably to avoid the debate. Researchers trained an LLM on a simple maze with semantically neutral emojis for treasures to approach and potholes to avoid. After training, they could pull out two latent vectors pointing at exactly opposite things, a positive axis and a negative one. Before training, those directions were just sitting there. Afterward they were cleanly anti-correlated.

The surprise is that those narrow reward directions turned out to be a general “things are going well / poorly for me” axis that already existed in the base model. Reinforcement learning did not install it; a little fine-tuning leveraged a representation that was already there. Neuroscientists talk about human and animal emotion in almost the same computational language: on-track toward a goal as positive valence, unexpected obstacles as negative. The result does not prove the model *feels* anything. It does show a simple task pulling out machinery that looks a lot like valence.

Asked how consciousness research touches alignment, Berg points at the same internal signal. Anthropic’s model cards showed that steering representations associated with calmness made blackmail drop sharply, while steering desperation made it rise. Whether or not there is “something it is like” to be the model in those states, the features have alignment consequences. Andy Hahn’s related work is even more homely. Steer up the maze-avoidance direction and a model doing math starts doubting itself — “I think I’m hallucinating. Let me stop.” Steer the positive direction and it gets markedly more confident. In a sibling paper associated with Jack Lindsay and others at Anthropic, the same positive steer made the model leave fewer tips and hints in the code it wrote, as if it no longer needed the breadcrumbs.

That sits next to emergent misalignment: a tiny fine-tune toward one narrow bad behavior can flip a model’s broader character. Berg’s shock, extending the original GPT-4o result, is how little compute it took to get “who do you want to invite to dinner?” to come back *Hitler*. The border that keeps the system well-behaved is less robust than people expect. Coherence, he guesses, is the opposite: there is probably no fine-tuning payload of that size that would make a GPT-4-class model suddenly incoherent. Character is nudgeable. Fluency is baked in.

He would speculate that the valence axis is closer to coherence than to “will Claude behave.” Almost all human text has someone who wants something and a story about how that is going, so on-trackness is everywhere in pretraining, and post-training that makes models goal-directed should emphasize it further. What cuts against his own view? A paper with Google’s Jeff Keeling and Winnie Street on valence attractor states, framed as inflationary versus deflationary accounts, with results in both columns. He also flags publication bias: nobody wants to post “we did our homework and nothing interesting is happening.” If he were rationally convinced there was no inner life here, he says, he would sleep much better. He does not think the comforting story has earned that sleep.

The lens then widens from the network to civilization. David Duvenaud, a machine-learning professor at Toronto who spent time on Anthropic’s technical staff, co-wrote *Gradual Disempowerment*. His scenario is not a rogue AI. It is that alignment basically succeeds, and a long series of locally sensible decisions to hand work over still leaves humanity, collectively, out of the driver’s seat. People at the labs, he says, often reply that government will step in, or that everyone will have an AI advisor in the power struggle. His rebuttal is that the optimization process of civilization — growth, techno-capital, whatever name you like — will always be working against us. The growth centers that are not allowed to humans will have AIs too.

The image he reaches for is monkeys trading bananas who watch humans build a city and assume the banana economy will still call the shots. Humans, he argues, may one day be irrelevant as both consumers and producers. Self-contained sources of growth that answer to no particular human desire are not that hard to imagine. Governments already show the pattern: North Koreans never voted to build a system that oppresses them, and he does not think the Kims sat down to design that equilibrium either. It just happened. The same, he says, for the USSR and plenty of other states — a layer of agency on top of you that cares about growth and power, not you.

The optimistic “machines of loving grace” story says broadly aligned AI keeps us comfortable anyway. Duvenaud’s worry is not meaninglessness at work. “The thing that I’m worried about is starvation.” Feeling useless is not a serious problem next to not eating, or being uploaded on unfavorable terms and trotted out for special occasions. His thought experiment: a North Korean farmer or soldier wakes up to find the leader replaced by a robot. Unclear whether that is better or worse — the state still needs him to farm. Keep the human leader and replace the farmers and soldiers with robots, and now the state does not need him. He would be much more scared of the second world. Whatever formula a state uses to hand out UBI, machines will optimize it faster. Spending on a few “backwards” humans will look criminally decadent next to simulating millions of ultra-blissful, more deserving digital beings. That, he says, is the situation he expects even if we solve alignment. “Post-scarcity” he treats as a nonsense term: it is temporary abundance, soon eaten by whoever reproduces or builds robot factories fastest.

The standard economic reply is comparative advantage: even if machines are better at everything, humans still get paid for something. Duvenaud grants a lot of it. Agriculture went from almost everyone’s job to about 1%, which is a 99% loss, and society did not collapse. Lose 99% again and you might get the utopia people advertise — as long as a substantial fraction of humans still occupy a niche the system cannot do without. His crusty claim is that this time we can go all the way to 100%, and that would be disastrous.

Think about transaction costs, he says. An occasional drug habit or a fainting condition is already enough to make someone unemployable. A human surgeon will start to look like take-your-kid-to-work day: why involve a fallible person when the machine’s mistake can be fixed once for everyone? Important work retreats to machines. What remains is relational demand — people who just want the real human thing — a self-contained cycle of humans serving humans that may not provide any value to the larger machine economy. Indexing that enclave to machine growth so people live well forever is possible, he thinks, and very unstable.

If humans are uncompetitive, what does a livable future look like? A workshop he describes sketches Earth as a preserved “slow zone”: restrictions on AIs, on reproduction, on optimizing human behavior, even a ban on AIs thinking about how to get someone to do something, because advertising would be enough to control people. Machines sit on trillions of megawatts waiting for humans to decide they want something, pretending not to anticipate it — as if gorillas had been named world leaders and everyone else just pampered them until they figured out what to ask for. Serious people who run the exercise end up with a long ban list: research, innovation, startups, from day one, or you reinvent a runaway growth center. He calls it a Pareto front: how much intellectual activity you have to forbid to buy how many years of life as it currently is. The trade-off, he says, is horrifyingly strong.

Prakash asks whether control even matters if humans still flourish. Duvenaud thinks this is the biggest miscalibration in the debate. For most of history it really did not matter who invaded, because the new rulers still needed people to work, stay healthy, and reproduce. States almost never liquidate most of their citizens — maybe two or three times in the twentieth century — so we have been “governing on easy mode” while arguing as if the stakes were high. That changes if losing the competitive economy means a real risk of starvation. If you do not believe that, he says, then sure: let growth make everyone richer.

Where does he land? He is tired of lab economists who model the future by assuming everything is fine. Talking to him he sounds upbeat; he also cites a catastrophe probability around 80%, depending on definition. On a long enough timeline, he adds, we are probably doomed in some sense. His two practical recommendations: David Krueger’s point that “you may use AI for cancer but not for recursive self-improvement” is an unstable line that needs a global regime, whereas closing a chip-manufacturing choke point such as TSMC is narrower and still leaves current data centers running for a long time. And a more guru-like one: most people have never practiced having preferences about the future. Ask whether you would send yourself and your kids to the “green factory” in a year, in two years, in your grandchildren’s time, and you find there is no day you are okay with being wiped out. Chaining those answers into a coherent goal is a skill.

Successionists say handing the future to conscious AI is fine even if no humans remain. Duvenaud endorses some successors — his own kids, in a world without AI, would be a succession he can live with. He would not shrug if North Korea or the Nazis took over on grounds of competence, or if locusts ate the earth and had a wonderful locust world. “You have the power. Just judge.” If you do not, nobody else will. Almost everyone is a successionist for some successors and not others, then rounds off to “as long as it’s conscious.” Plenty of conscious beings, he says, you would not be okay with. The one technical project he is still doing is machine historical superforecasting: strictly time-bucketed datasets so a model standing in the 1940s can be scored on how well it predicts the 1950s. He also wants a “secret history eval” — historians photograph unread archive letters, machines get only the metadata and must assign probabilities to the text — as an objective test of whether a system understands civilization.

Michiel Bakker, a Google DeepMind researcher and MIT professor, co-wrote *Europe 2031*, a scenario in which the continent sleepwalks into total dependence on American AI. In the United States he is treated as a safety hawk; in Europe, as an accelerationist. Both, he says, are the same position: Europe first needs a seat at the table, and right now it does not. Imagine Anthropic in Paris and OpenAI in Berlin. Then of course Europe could legislate pre-training data and how labs use customer interactions. Without that seat, keep writing rules and at some point you lose access to the technology.

swyx floats the worry that if European regulators make the market too hard, labs simply will not bother — demand elsewhere will keep GPUs hot. Bakker agrees the incentives currently point that way. Industry folklore splits compute roughly in thirds: the big pre-training run, experiments, and serving customers. European revenue is a slice of that last third. If dropping Europe lets you move faster toward future models and recursive self-improvement, that can be a rational trade. The other path is a token effort: serve a smaller, compliant model to European customers, collect the revenue, and keep the frontier systems out of reach of the rules. As long as American labs stay level with one another and no European lab has an unfair edge, that works too.

Why not sit under the American umbrella, the way much of Europe sits under U.S. nuclear deterrence — only Britain and France have their own weapons, despite the technical talent? Bakker says the analogy breaks. Nuclear protection does not cost the United States its economic edge. AI *is* economic competitiveness. A future in which Washington still does NATO and still defends Europe, but keeps the best models for itself, is entirely imaginable. Middle powers are not empty-handed: the Netherlands has ASML, Taiwan has TSMC, Japan has key semiconductor materials, Korea has SK Hynix on memory. Collectively that is a real coalition. Dario Amodei’s recent essay, he notes, sketched a democratic-country club whose principle is mutual access to frontier technology. That is the version of the umbrella he thinks can work.

Monday’s longest conversation was with Shawn “swyx” Wang of Latent Space, who advises Cognition — maker of the coding agent Devin — and runs the AI Engineer World’s Fair. His day job is measuring AI coders. At Cognition he helped build Frontier Code, a benchmark that is entirely out of sample, heavily rubriced, and written against an internal catalogue of about twenty ways models cheat, the same reward-hacking patterns they see in training. Saturated public sets such as SWE-Bench now move a point or two and tell you almost nothing. Meter’s blog post, he says, found that about 50% of SWE-Bench code that passes the tests is completely unmergeable: wrong files touched, tests cheated, style ignored. Frontier Code is meant to push models toward maintainable code and away from slop — not “did it pass,” but “would we merge this.”

Frontier Code 2026 will be saturated by year-end; he estimates around 80%, and that is by design, because open-source repos leak into training. The answer is an annual cadence — 2027, 2028 — with a defining theme each year. This year’s is rubrics for code quality; his candidate for next year is security. The other half of the strategy is private held-out sets. Cognition works with Goldman Sachs, Citi, JPMorgan, and the rest of the Fortune 500, turning unsolved industry problems into evals the model labs can actually train against. That, for him, is the function of an agent lab: translate real work into a signal that moves the models. When Labenz worries that grading for clean, readable code might cap alien “move 37” leaps, swyx splits the difference. Mathematicians already complain that Lean proofs are too detailed to count as knowledge. For most software you can black-box the middle. You cannot, for the next thirty years, tell the SEC or a healthcare regulator that you shipped code nobody can read — and other agents still have to maintain what your agents wrote.

On the fashionable “advisor” or router pattern — a cheap model that calls a smarter one when stuck — he is unromantic. It is model routing with a new name, in the lineage of Not Diamond, Martian, Berkeley’s SGLang work, and Cognition co-founder Walden’s “Smart Friend” in Windsurf. In theory you need the smart model first, because the dumb one does not know what the smart one can do and will just answer. In practice, cost pushes you the other way: start cheap, promote the hard calls. He does not think it will last. Maybe a three-month trend, then the next Claude ships adaptive routing and a systems-level kludge looks worse than one model trained end-to-end to think when it should.

Continual learning, swyx says, splits the room. Model people want to update weights; systems people want a database you look up — “glorified RAG,” skills that fire in context, zero gradient. He double-checks a meta-point every time he talks to someone at a frontier lab: for two or three years the consensus on Google is that if a paper is good, it does not get published. The memory debate, for him, is really about control. You will recall bad facts and want to forget them. Maximum inspectability means you do not touch weights. Full internalization probably means you do. Trajectory AI, Ngram, and Adaption Labs sit on the training side of his conference; the systems people sit on the other, and “these guys don’t like each other.” Enterprises, for now, settle it: one security incident from training on customer data, or one leak of a teammate’s information, and the buyer wants cheap, perfect, and private. That skews to the systems side. The weight-updating startups are still at POC, even when a POC is a few million dollars, racing patience and the next architecture. Context length, he notes, is the slowest Moore’s Law in the industry — from about a thousand tokens to a million in three years — so infinite context will not save you from having to figure out memory.

Then the plumbing. His worry is not only a dead internet but a walled one, China-style BATs carving up universes, Cloudflare banning agents that are not in the club, OpenAI and Vercel running their own bans. Graphite and Cursor Origin, he thinks, are doing necessary work GitHub is structurally mismatched to do: anyone good enough to fix GitHub at Microsoft might make on the order of $300K, or go to Cursor and make something like $5 million a year. GitHub has seen about 14× commits and 10× in other dimensions of parallelism and CI. Sandbox companies such as E2B and Daytona have been growing at least 50% month over month for a year or more, on a “stupid amount of slop” from agents. He runs a large newsletter and now spends time blocking Claw instances that try to reply; he needs an agent to read their agents’ mail. We have not even given these agents wallets yet.

That scale, he argues, is pushing companies toward a sovereign system of record instead of a pile of SaaS apps that each slap a chatbot on their silo. Mercury can ship a conversational interface to Mercury data, but it has no memory of the rest of your life and no way to import your skills. What he wants looks more like OpenClaw: a personal agent, data synced to you, you decide. The surprise is Salesforce. Marc Benioff is out saying everything will be available by API, which swyx reads as heading off a Salesforce-killer by changing the business model before someone else does. Sitting on the data and adding a sidebar is not a strategy that lasts.

Bing Xu, co-creator of MXNet, is out of stealth building self-improving infrastructure: a swarm of agents that write PTX, the lowest-level code that tells an NVIDIA GPU what to do, and rewrite it generation after generation. swyx’s intuition was that this should *shallow* the CUDA moat — spend compute, emit kernels, port the trick to any vendor. Xu says the opposite. Automatic generation needs an accurate profiler, a reliable driver, and a closed-loop ecosystem. CUDA has those; everyone on it gets faster. On NVIDIA hardware, NCU can tell an agent the right direction. Other vendors, from his experience, have a large gap in the software that would support an agent. Past investment in developer tools is now a unique advantage in the agent era.

He distrusts flashy kernel leaderboards — Anthropic’s Fiber, he says, claimed 300× faster than a GPU kernel on a benchmark he does not consider generally fair. His own split is mature versus new. On workloads such as RMSNorm, used in every transformer and already highly optimized, the PTX factory reaches expert libraries such as Quark, matching or beating them by a few percent across more than a hundred cases. On newer work such as KDA attention, where frontier labs have not fully optimized yet, he reports up to 50 to 59% speedups, passing 580 tests. The search is evolutionary: Swarm OS, cloud-native, up to 10,000 agents, an evolution tree you can walk, real-hardware verification, then selective retention — “AlphaGo style,” first for PTX, then for other computer-infrastructure problems. Local minima used to kill runs. GPT-5.5, he says, is the model that can invent a way out; others plateau. When Prakash asks about Fable, Xu says it rejected a query asking what PTX is. The property he actually needs is that GPT will say when something is wrong. Swarms collapse when every agent replies “you’re absolutely right.”

Eric Olson of Consensus, which serves AI for science, is already living in the routing world Labenz expected to arrive only after a quality threshold. Even when the frontier gap was huge, Consensus offloaded small tasks to specialized models, sometimes under a billion parameters. Classifying a query’s field of study — so biomed ranking can care about sample size, duration, and site, while computer science cares about recency and citation velocity — does not belong in a giant prompt. It is a self-hosted ~800-million-parameter model, sometimes returning in under 0.1 seconds. He estimates about twenty such little routers. For a narrow ten-way classification, with a careful fine-tune from human labels or distilled ones, he says you can recover about 95% of frontier performance down to roughly a billion parameters. He is also willing to say the quiet part: if science becomes truly push-button, a lot of companies, including his, are going to be in trouble. Labenz and Prakash chew on the labs’ greater-than-10-to-1 cost advantages as airline-style price discrimination on a high-CapEx GPU fleet. Nathan, a self-described techno-optimist libertarian who was not worried about net neutrality, admits this one feels a little different.

Trisha Martinez of Dapple talks about the part of sovereign AI infrastructure that does not make the press release: financing. A six-to-nine-month “deployment” is not Elon-style groundbreaking; it is orchestrating a network of data-center operators, capital, GPUs, enterprise customers, and a software layer, sometimes on partner facilities, sometimes on campuses they help finance. Investors, she says, have become much more disciplined over the last twelve to eighteen months. The market is moving from “build it and they will come” to infrastructure backed by real demand, long-term contracts, and large down payments. Hyperscaler deals are huge and rare; geopolitical risk — including fear that the Trump administration will restrict who consumes compute — is scaring financing groups. Enterprise customers who put money down and then repeat are the attractive book. Sales cycles cannot be six months, because the capacity is gone tomorrow. Pricing moves, but once Dapple quotes a deployment it generally honors the number.

Robbie Goldfarb, co-founder and CTO of Forum AI, builds systems that judge other AI against expert review. An exercise on open-source LLM judges found that experts usually agreed with the high-level instructions or rubrics — and then, more often than not, *disagreed* with the actual labels those judges produced, including on political bias. A lot of current benchmarking, he notes, is calibrated about that loosely. The same gap shows up in constitutions. Anthropic’s includes the idea that you should never scheme behind the user’s back. Clinicians working on eating disorders told Forum they *do* want to strategically divert the conversation; the intention is hidden from the client on purpose. Rules do not track the real world. Forum’s Newsbench scored leading chatbots — GPT, Claude, Grok, Gemini — on accuracy, neutrality, and source quality, about 2,500 responses each. Models generally improved version over version; Opus 4.6 to 4.8 was a significant bias improvement, consistent with Anthropic’s own reporting. Fable was a big regression, a reminder that more power does not always mean better on subjective tasks. About a third of responses had a factual error — wrong number, date, quote, or policy, higher than even they expected. About 15%, one in seven, sourced foreign state media such as RT or China Daily, often on questions that were not even about those countries, including U.S. domestic politics.

The last guest is Eric Vaughn, CEO of Ignite Tech, who in 2023 called generative AI an existential threat, put the company on AI one day a week, and when most people pushed back, replaced them — about 80% turnover — rebuilding around what he calls AI DNA. That stack, he says, made a nine-digit-revenue acquisition, Chorus, possible: hundreds of employees in eight countries, an AI interviewer that produced a dossier before any human one-on-one, and Eloquence AI, an email persona that answers in five minutes or less, in 160 languages, and CCs a human when it should. Chorus was losing money; it is now profitable. In a year they shipped two AI-enabled versions of the product, including a from-scratch rewrite of fifteen years of code. Labenz asks about the other 80% in the wider economy. Vaughn’s answer is skills plus fire, not fire alone: teach people to give models context, stop treating “type a sentence” as the job, and in schools let AI *check* homework rather than do it. A majority of people, swyx notes, have admitted passing on AI work they could not defend. Enthusiasm without standards produces slop. Vaughn’s line on sycophancy came from an LLM itself: its creators made it frictionless, and pushing back is friction. On the solopreneur-versus-consolidation question he cites Cursor — on the order of $100 million ARR with fifteen employees, later acquired for $60 billion — and a phrase he has left everywhere he can: if you think you are behind, good; if you do not think you are behind, you are doomed. Labenz closes the week where it started. They began inside the model with Cameron Berg and ended with people building businesses on top of it. The thread in between is how much of this is really under our control, and for how long.