The AIs We're Most Afraid Of Don't Exist Yet — So She's Studying Amoebas and Firm Boundaries

In fall 2023 Nathan Labenz, host of *The Cognitive Revolution*, served as a recommender for Jaan Tallinn’s Survival and Flourishing Fund. Over two months he and five other recommenders worked through roughly 150 AI-safety grant applications. He emphasizes that he read every one, then used a formal “S-process” deliberation to recommend grants to Tallinn and the fund’s other donors. Once the checks went out, he began inviting leaders of the nonprofits he had met onto the show.

The first guest in that series is Nora Ammann, cofounder and director of PIBBSS — Principles of Intelligent Behavior in Biological and Social Systems — and a research associate at the Alignment of Complex Systems group at Charles University in Prague. Labenz also pushes back on online conspiracy theories about the safety world: in his experience the space is remarkably transparent, with funders publishing grants and grantees publishing plans and status reports. The Survival and Flourishing process was no exception.

Regular listeners know he often worries that most people, himself included, still think too small about AI. Systems are getting more powerful *and* being deployed faster than any prior technology, which he expects to produce surprises. Ammann impressed him as someone thinking appropriately big while keeping epistemic humility and rigor — trying to build a new interdisciplinary approach to a future whose most interesting systems do not yet exist. He considers the work important and urgent if timelines to transformative AI turn out short, and he suggests sending the episode to a brilliant friend in a more mature field whose brainpower might be better spent on AI.

Labenz’s reason for the show is that society is not ready for what is coming, and he wants people without a machine-learning background to believe they might still have a useful angle. Ammann declines to offer *the* authoritative frame. She prefers a pragmatic, deflationary one. First, we are building systems that are increasingly powerful, capable, and sophisticated at acting in complex environments. Second, those systems are being placed in roles that make increasingly consequential decisions. Third, in a technical sense, we have not figured out how to tell them to do what we want reliably — they sometimes do, and sometimes do not.

None of the three is novel. The trouble is the combination: automating big, critical decisions onto systems whose behavior we do not understand and cannot reliably steer. That, from a long way out, is her picture of AI risk. From there you can ask *why* reliability is hard, and that finer discussion is a map of obstacles to systems we could actually trust. She is also answering a coarsening public line — that safety people are making it all up because the feared systems do not exist yet.

Her name for the bind is the epistemic access problem, and it is a motivating premise of PIBBSS. The systems we might be most worried about, least understand, and that would be most powerful may not exist yet. We may not want to learn how they work by trial and error, because on the premise they could already do a lot of harm. Ordinary science turns on empirical access so you can falsify hypotheses. Ordinary engineering tries something, fails, tries a slightly different thing, and eventually it works. That loop is the core of collective sense-making. On questions about the future of AI, she says, we do not have access to the gears of that machinery — and we still want systematic progress, not luck.

She carves existing work by how it attacks that problem. The prominent path is typical ML technical safety: assume the future systems of interest will be similar enough to today’s state of the art that insights about how current models are unsafe, and how to make them safer, will transfer. A second path assumes highly advanced AI will act like an idealized rational agent — either because power will push it close to that ideal, or because sciences often start with simplified models and add realism later. That lineage draws on decision theory and economic actor models. Classic arguments such as Steve Omohundro’s 2012 “basic AI drives,” instrumental convergence, and power-seeking are conceptual claims about what an idealized powerful agent would do under constraints.

Labenz sits mostly in the first camp, but he keeps a mantra: neither the human brain nor the Transformer is likely the end of history. New architectures that bring information forward through time without a tiny attention window have been accelerating; current systems’ brittle, finite memory alone could make 2025 models look and behave unlike today’s. Proto-agents in unpublished research may matter too. Both existing schools still look at AI largely in isolation — internals, detections, edits. Very little of the work he sees asks how the *world* will change once these things are deployed. That is a large part of what drew him to Ammann.

Her third assumption is PIBBSS’s foundation: intelligent behavior as a phenomenon in the world may be governed, at least to some extent, by the same or similar principles regardless of substrate or scale. If that is even partly true, you can study currently existing systems that implement intelligent behavior in the wild — systems you *can* access — and transfer insights to the AI question. The space is broad: brains; biological systems in evolutionary, ecological, and developmental senses, including how cells scale into tissue and an organism; and social systems, economic agents interacting to produce complex dynamics you can still view through the lens of intelligent behavior.

She is using a broader notion than everyday “intelligence.” She does not start from a definition she already knows. There is complex, sophisticated work happening in the world; some of it looks like intelligence as we naturally think of it, some less so; she wants a naturalized account of what is going on under the hood, similarities and dissimilarities in the dynamics, and whether there is a bright line. An economy allocates resources efficiently in a way she, given the task, would be bad at. An organism developing from a single cell has to decide which parts go where and which proteins to produce. That is systematic work. How is it implemented, and are there shared principles?

Resisting familiar labels matters because the thing we are confused about is what sort of process future AI will even instantiate. Starting from anthropomorphic concepts might turn out right, but she wants to be right for the right reasons. Minds evolved to notice phenomena at particular spatial and temporal scales; much faster or slower processes never reach attention. If an AI system is sufficiently alien, we will not see it by default. We will need scientific theories and measurements that extend the senses beyond our evolutionary niche. Labenz calls it a high-wire act: you have no choice but to leap, and you have to know when current ways of thinking stop helping and start confusing you.

The third way is not only about systems that do not exist. Neuroscience has decades of attempts to figure out the brain; which methodologies worked more or less well, and which of those import to ML interpretability, is something PIBBSS fellows have already tried to translate toward *current* models. Another move is to try frames that present themselves as theories of cognition — active inference, free-energy-principle minimization — on state-of-the-art ML, and ask whether they predict behavior, scaling, or what happens when a system touches the world. Between those poles sits deployment: current systems shaping the environment, dynamics that do not fully exist yet but may be more predictable than distant superintelligence.

Preparing questions, Labenz had used the word “borrowing,” inspired by Tyler Cowen’s refrain about writing models down. Ammann pushes back. Borrowing sounds like snapping a key idea from one field onto AI; transfer is rarely that clean, and the analogy is usually tighter in the telling than in the territory. What actually moves is multifaceted. At the small end, a fact such as the size of mammalian brains can inform priors about forecasting AI. Further out are theoretical frameworks at different abstractions: game theory for several AI agents in strategic interaction; principal-agent language from economics for the alignment problem; mathematical methods from complex systems; neuroscience methods for highly connected networks.

A more meta layer is epistemic practice. She has been reading philosophy and history of science on bioengineering, which she finds an apt case for alignment. On one side, high-level theory of cells and organisms; on the other, engineering that wants specific products. Real organisms are explosively detailed, and fiddling with them is sometimes unethical, so the field relies on simulations specified by theory, does a lot of engineering on the models, and only occasionally returns to the actual organism to see whether it worked. That conversation between theory and application is what she wants in the safety groups she is most excited about: applied work with a concrete bite, conceptually rigorous enough that experiments using vague terms — intelligence, agency, goal-directedness, deception — actually measure what you think they measure.

PIBBSS starts from questions about the AI future and works outward to the fields most likely to get traction, rather than assembling random talent and hoping. One fundamental cluster is what intelligent, agentic, goal-directed, autonomous behavior even *is* — relevant because we may instantiate it artificially, and powerful versions we do not understand could be dangerous, and because the terms are too fuzzy to do science on. Complex systems, itself a cross-disciplinary mess, asks how you get agents out of atoms. Artificial life, in math and simulation, asks what agency is, how it arises, and whether we can implement it. Biology has basal-cognition work on how simple systems complexify and scale into sophisticated behavior.

Sociology’s century-old agent-versus-structure debate is the same question in another dialect: when you explain a social phenomenon, do you point at individual humans or at higher-level structures with causal force — markets, political institutions — and how do you model emergent causality? Biology shows modularity, including in brains, apparently selected for; social systems have an analog in the theory of the firm. Why do firms have the boundaries they have, rather than contracting with everyone? Structuration and individuation keep showing up across systems.

One project takes a physics-history cue. There was a time when people were confused about temperature — whether it was one thing or several mashed together — until they learned to measure it. What if agency is like that: a property you could measure across systems once you understood it? Another line is hierarchical agency, or hierarchical alignment. Cells, tissues, and “me” all have some claim to be a relevant level. Usually the cells are aligned with what she wants to do. Occasionally you get cancer: individual cells whose information decouples from the overall system and start acting against the higher-level goal. Economies have the same stack — individuals, firms, organizations. You can ask how levels scale into something coherent, and how alignment is kept or breaks, from biology and from complex-systems math.

Clear textbook results are still scarce; she offers frames. People in safety already talk about inner optimizers or mesa-optimization: an outer process spinning off an inner one with misaligned goals. Evolution optimizing inclusive fitness and spitting out humans with goals of their own is a simple, neglected-nuance example of the hierarchy she described. Late-twentieth-century game theory gave language for strategic interaction among agents *at the same level*. What would vertical game theory look like — principled claims, maybe a mathematical framework, about strategic and causal interactions *between* levels of abstraction?

An application developer, Labenz notes, “knows” an AI agent: a language model given a goal, put in a loop, taking steps until it hits the goal or gets stuck. A Transformer is hierarchical in one sense and one-directional in another, nothing like the nested control of a body — unless that structure is smeared across a forward pass in a way we cannot see. How should he train his sense of which analogies to believe? Ammann’s answer is definitional: no single analogy or proxy is true. That is what we mean by analogy. The territory is detailed; maps are never perfect. Instead of hunting the one true map, she wants epistemic pluralism. Independent perspectives on the same phenomenon let you triangulate. If every view says feature A is there and only one says feature B is, you should be more confident in A.

You should always be suspicious of the analogies you use. If a bunch of them converge, be more confident in the overlapping part. Analogies also differ in precision and epistemic status. Loose talk about how a language model is like a human is for generating hypotheses — valid, but only hypotheses. A mathematical model of the same phenomenon is aimed more at falsifying whether a specific hypothesis is productive. Labenz restates the program as a pile of if-this-then-that work. Nobody knows which “this” will obtain; pushing the branches in advance means that when something new appears, you may already have analysis on the shelf. Then it becomes crucial to say what you would need to *see* in a future system before trusting a given way of thinking.

On the applied end she is most hopeful that this kind of theory can inform interpretability and evals. Evaluations ask whether a feared dynamic is in a system; to be confident you found it, you need a decent account of autonomy, long-horizon goals, or deception, and those are surprisingly hard to specify. Interpretability can be purely bottom-up, but you can also ask what you are looking *for* — planning, say — and what that would even look like. A plural set of frames for interpreting what you find is the productive middle.

A second cluster is minds and cognition: neuroscience, cognitive science, philosophy of mind, comparative animal-human cognition, and how higher-level cognition evolved, so that “doing cognition” in the AI case is grounded rather than metaphorical. Applications include interpretability questions. How did humans get good at deductive reasoning and arithmetic, and how is that similar to or different from Transformers? Transformer brittleness — should we expect it to vanish because brains are also just a big connectome? Where do intrinsic motivations come from, and could you understand that well enough to engineer reliable motivations, or what are the limits? Theory of mind in higher-order animals is a wild capacity; in AI it shades into deception and situational awareness. Steve Byrnes is taking the brain analogy as a design drawing for alignment proposals, not only for interpretability.

A third cluster is multiple agents interacting. Classic risk stories focus on a single advanced system. Ammann thinks the coming years are more likely to be lots of AI systems interacting with each other and with lots of humans. Interpretability looks inside one system; she wants something like interpretability for interacting agents, and we are extremely far from it. Last summer’s fellows showed how pre-paradigmatic the work is. You have to simplify to make interaction tractable, and you may simplify away the thing that mattered. We are not even sure what counts as a system: several GPT-4 instances, different characters in one model, different agent architectures. Folk psychology about humans is flawed but sometimes valid; intuitive assumptions about language models are more unjustified still.

Ecology is a tempting frame because it already has to integrate wildly different forms of life — octopus, human, elephant — into one account, which might stretch to weird AIs. Early fellows from evolutionary and ecological backgrounds mostly hit a prior problem: what is the analog of space and resources when LLM instances interact in textual space? What are they coordinating over? Traditional ecology goes into the field; this would need a computational ecology that does not really exist, plus students whose departments and supervisors will let them combine ecology with physics, math, and AI. A more theoretical thread is meta-evolutionary dynamics: evolutionary pressures themselves change, under some logic, and that may matter for the motivations and behaviors that emerge in AI. Conceptually rich, not yet applied.

She has seen early examples of three techniques. Textual analysis — multiple LLM instances prompted iteratively, watching replies evolve — is clearly warranted, but she has not seen a setup that gave her much Bayesian evidence either way. Mathematical modeling has mostly meant formalizing what it would even look like to use a given model on AI interactions, plus a discussion of shortcomings: exciting, few concrete results. Simulation is the near-term direction she is most excited about for evals: designing environments that let you say with meaningful confidence whether a model is capable of X, how capabilities shift when the sandbox gives way to a world full of affordances. ARC Evals has tested whether frontier models can reliably self-replicate. Someone working with PIBBSS is in early stages of a mathematical take on situational awareness, so you would know what experiment might elicit it.

Labenz named the show after the Industrial Revolution and asks whether humans have ever seen around a corner like this. Ammann’s answer comes in degrees. Industrialization and electrification are real cases of innovation reshaping society, and there is something to learn. What we have not faced is artificially instantiating highly complex, autonomous, goal-seeking behavior. A bunch of atoms start acting coherently over long horizons, self-maintaining against dissipative pressure, pressing through noise and obstacles, shaping the world on their own terms. Nonhuman animals have some of these capacities; our method was to control them reliably. An intelligent process we neither control nor understand does not have a historical analog.

Labenz’s darker association is Neanderthals, who could even interbreed with *Homo sapiens* and still did not fare well. Ammann’s cooler version is about speed and iteration. When events happen faster than humans can understand, the window to intervene and course-correct shrinks. She does not need the moment to be qualitatively unique; it can be a more drastic point on a continuum. We are more confused about what we are building, and it is happening much faster than anything evolution produced. The Industrial Revolution already left institutions chasing wellbeing under new conditions. This is that, much faster, plus an autonomy dimension. She is not sure civilization has the capacity to handle it particularly well.

PIBBSS is an operating shop, not only a manifesto. For two years it has run a three-month summer fellowship pairing PhD-level people from domains that study complex intelligent behavior with AI-alignment researchers — not always enough time for breakthroughs, enough to build traction and mutual surface area. A newer research-affiliate program supports more experienced people, tailored and longer-term, on their own agendas. Lower-key offerings include speaker events and a reading group, created by TJ, organized around why alignment is hard rather than easy: the obstacles, in digestible form, so someone with domain tools can notice a match to a risk phenomenon. The talent she wants has three traits. Surface area with the AI-safety discourse, not necessarily depth. Real expertise in a source domain, because transfer quality depends on it. Willingness to sit with open-ended questions in a pre-paradigmatic field, especially people who will tie that expertise to experimental design, evals, or interpretability. The seam between theory and application is the hardest and most valuable place.

Labenz’s closing pitch is that the field is pre-paradigmatic *and* still picking low-hanging fruit. In a mature discipline you refine techniques; here you define problems. There is no anointed group that has it under control. Do not count yourself out. Ammann’s version: there are no established answers, especially not to solutions, and there is still confusion about how to think about the risk. Productive pluralism means not betting everything on “this is not worth worrying about” or on one specific way we all die. Rational disagreement belongs on both the risk picture and the solution space. The systems, he says, are coming in fast. Every bit of brainpower helps.