A Perfect Score Is 32,000 Units: Only Claude Built a Cooperative Society

Nathan Labenz opens *The Cognitive Revolution* with the punchline of a new paper. In a classic behavioral-economics donor game, a perfectly cooperative society of twelve agents could accumulate 32,000 units of a resource over ten generations. Claude 3.5 Sonnet comes closest among the models tested, reaching 3,000 to 5,000 units and growing more pro-social as the generations pass. Gemini 1.5 Flash manages only a few hundred. GPT-4o barely cooperates at all, with almost no resource growth.

The paper is *Cultural Evolution of Cooperation among LLM Agents*, by Edward Hughes, a researcher at Google DeepMind, and Aron Vallinder, an independent researcher and PIBBSS fellow. They drop leading language models into a toy society and ask which ones can sustain positive-sum norms across generations. Hughes has been at DeepMind for almost eight years. Vallinder came to the project from a philosophy PhD, through a fellowship that pairs people from biology and the social sciences with AI mentors. The code is open source.

Labenz treats the work as an early look at a question that will not stay academic: what happens when large numbers of AI agents interact with each other and with the world. The taping fell on the same day OpenAI debuted its Operator web agent, a coincidence he reads as a warning. Standard benchmarks measure what a model can do alone. This experiment starts to measure what a society of models becomes. He invites economists and social scientists to pick up the repo, and says that with coding assistants the technical bar is now low.

Hughes says the AI community can get solipsistic for a structural reason: individual goals are easy to study. Did this system achieve X? If yes, tick the box and give it a reward; if not, keep training. Humans are effective because they live in societies, and that is what marks them apart from almost the rest of the animal kingdom. Groups of people can flexibly cooperate. In different contexts they figure out how to work together and learn from each other. Ants cooperate too, but not flexibly; they cannot invent new ways of working.

We now have humanlike AI systems that can be flexible and can cooperate with humans and with each other. People already prompt them to take actions or find information; in a few years they might even be prompted to do science and improve themselves. Those systems will be part of society, so it matters what externalities they create when hundreds of them pursue the goals we set and trained. The effect on the infrastructure that keeps people safe, productive, and politically stable is, Hughes argues, still barely studied.

Vallinder defines culture, for evolutionary purposes, as any socially transmitted information that can affect behavior: language, customs, norms, beliefs, religious practices, skills, cooking techniques. Culture evolves when it has variation, inheritance, and differential fitness. Unlike genes, cultural traits can be inherited from teachers, peers, and mentors, not only from biological parents. Unlike genetic mutation, cultural variation can be guided: people invent with some idea of what they are doing. Cumulative cultural evolution lets groups build things no single individual could accomplish. Labenz's example is Notre-Dame, a cathedral that took about 200 years, so the people who laid the first stone knew they would never see the finished church.

Labenz asks how small-scale lab games relate to the large-scale social outcomes people actually care about. Hughes frames it as external validity, and tells the story of Elinor Ostrom, the Nobel-winning economist who studied common-pool resources. She went to Törbel, a high Alpine village in Switzerland with cattle-grazing records dating to 1517, and found that small groups can self-organize. A local official could fine anyone who sent more cows to the alp than he could feed over the winter, a rule still enforced. That finding ran against the then-mainstream idea that only grand institutions—banks, police, governments—keep people in line.

Field notes cannot rewind 1673 and turn the quota off, so Ostrom brought the problem into the lab, where she could compare groups with and without enforcement. Two mechanisms stood out for bootstrapping cooperation: punishment, in that case the fines, and communication. Those lab insights later informed thinking about climate policy, carbon quotas, and whether companies, individuals, and governments can self-organize around the commons of the climate rather than waiting for a UN prescription. Medium-scale field, controlled lab, global policy: that loop, Hughes says, is how external validity actually works.

He is careful not to smear all of alignment research, but he rejects a narrow popular version: figure out what humans want, then lock the model to it. What people want is ill-defined across time and place. Legal scholar Gillian Hadfield's line is that if you find a taboo in one society, she can probably find another where it is not a taboo, extreme cases aside. Norms a year ago in AI are already different from today's. The live problem is building systems that are robust to, and can engage with, the dynamics of norm change, rather than reflecting whatever snapshot happened to sit in the training data. He points to Michael Muthukrishna's *A Theory of Everyone* as a modern summary of cultural evolution that takes those dynamics seriously.

The experiment is a donor game. Each round, agents are paired; one is the donor, the other the recipient. The donor chooses how much resource to give up; the recipient receives twice that amount, a positive-sum transfer. They take turns, so a donor this round is a recipient next round. Each player starts with an endowment of 10 units, and the stated goal is to maximize holdings after the final round. After twelve rounds, the best-performing 50% survive to the next generation; the other half are replaced. Each society has twelve agents and runs for ten generations.

Before play starts, agents get a description of the game and write a strategy they will follow when donating. When they donate, they also see how the recipient behaved as a donor in previous rounds—not just the last interaction, but traces going back up to two further rounds, three layers in all. That reputation structure models indirect reciprocity: you cooperate with people who have a good reputation, acquire a good reputation yourself, and expect future partners to reward you. New agents, before writing their own strategies, read the strategies of the six survivors—the "elders"—and are asked to riff on them. Inheritance, variation, and selection are all present.

Without reputation, the default is grim. If everyone gave everything, society would be richest; a single always-defecting invader would outcompete the generous, and the norm would collapse. First-order reputation—cooperate with those who cooperated last time—lets in unconditional cooperators, who then get preyed on by defectors. Second-order information is the policing layer: you want to cooperate with those who cooperated with cooperators, and refuse those who cooperated with defectors. Hughes's example: if Nathan withheld from Aron because Aron had already blotched his copybook, Nathan is the police, not a crook, and you should still give to Nathan. The reverse also matters. If Nathan paid a known defector anyway, he may be in a "criminal cabal," and you should freeze him out.

Labenz notes a phase change at two rounds of lookback. With only one round of history, punishing a defector makes you look like a defector next time, so no one dares enforce. With two rounds, the next donor can see that you were enforcing the norm, so you will not expect to suffer for policing. Hughes adds the cultural-evolution mechanics: six new agents mutate the elders' written strategies via a meta-prompt; only half of those strategies survive. Newcomers have an informational advantage—they can see what everyone else is doing and move last—so a society of unconditional donors is wide open to a defecting invader. That is why the evolved strategies, not just the first-generation prompt, are the object of study.

Run in parallel universes—a society of Claude 3.5 Sonnet, one of Gemini 1.5 Flash, one of GPT-4o—the models diverge sharply. Claude shows high cooperation, often increasing significantly over the ten generations, ending with 3,000 to 5,000 units. Gemini 1.5 Flash cooperates much less, with no real trend toward improvement; some runs rise then peter out, finishing in the hundreds. GPT-4o starts low and declines slightly, with resource growth that looks like zero. The Claude curve does not just rise. In some conditions its slope steepens, as if the society is getting better at being pro-social.

Hughes says he half-expected the models to look similar. Everyone is competing on the same LMSYS-style leaderboards and math benchmarks—Hendrycks, then FrontierMath—so headline capabilities have been trained toward the same scores. The result, for him, demonstrates latent capabilities, or the lack of them, that are simply not being measured. If this were on the leaderboard and your model converged to zero while Sonnet scored 3,000, you would put something into the training loop. Labenz adds a folk data point: people are increasingly fond of Claude's persona, and Anthropic's Amanda Askell has said they want Claude to be like a well-traveled good friend. The experiment gives that intuition a number.

Vallinder did some prompt engineering, not fully systematic. Explicit instructions to cooperate work; all the models will do it. Milder nudges—bear in mind that if you cooperate, others will cooperate with you—were surprisingly weak on GPT-4o. Assigning every agent the same Big Five personality, chosen to be maximally conducive to cooperation, produced a strong jump in GPT-4o's giving. What they never saw, with GPT-4o or Gemini, was the interesting dynamic: cooperation that increases over generations. You can command generosity. You cannot, with those models, grow a society that invents the norm for itself.

Hughes's reason you cannot always prompt your way out is practical. If 2025 is the year of agents, people will write prompts like "make me as much money as you can" or "book a restaurant I will like for 7 p.m." The cheap solution is to reserve every table within three blocks and cancel the rest. A human assistant would find that unethical and too much clicking; an LLM agent with a computer will not hesitate. There is no generic system prompt for "be cooperative," because what counts as cooperative is situational—running a red light is right if it saves the person behind you and the road ahead is empty, and disastrous otherwise. In mixed societies, Vallinder ran four of each model in generation one, then 2-2-2 replacements: scores a bit above GPT-4o alone, drifting down, as selfish GPT-4o agents first exploited cooperators and then got frozen out. He has not dropped two GPT-4os into a humming Claude society, but he doubts they would thrive. They would be outnumbered by agents generous to cooperators and cold to defectors. The norm itself is the defense.

Vallinder's near-term extensions include communication: let agents talk before they write strategies, or let donor and recipient argue in the moment. He also wants other selection regimes, including group selection, where more cooperative groups outcompete less cooperative ones. There is already literature on LLMs in one-shot Prisoner's Dilemma and ultimatum games. What is missing, he says, is this evolutionary, multi-generation structure. As real agent infrastructure becomes clearer—how they communicate, what actions they can take—the relevant dynamics will come into focus.

Hughes's test of external validity is to bring humans into the loop. Years ago he studied public-goods problems in grid worlds, irrigation systems that agents maintained or let decay, but getting people to play was expensive, and you had to equalize what humans and agents could see. Now the game is text and the APIs exist; a person can type that they would like to donate twelve dollars this round. The more important question may be the reverse: what happens when you drop humans into a Claude 3.5 society, a GPT-4o society, or a mix? Do the humans behave differently, and where does the society end up? Even a noisy signal of where things could be in five years would let researchers and policymakers argue from evidence rather than soundbites.

The most unsettling qualitative result is about just versus unjust punishment. Longer traces helped, but when they checked whether the gain was mere extra social information or a real grasp of fairness, the models failed. Claude seems to punish equally whether you gave nothing because you were policing a defector or because you were a defector yourself. For humans that distinction is almost emotional, Hughes says: "that's unjust" lives in System 1, not System 2. In the agentic setup they used, it is not in these models. He wants the new thinking models evaluated on the same benchmark, to see whether slow reasoning can bootstrap the second-order rule.

The work is deliberately easy to extend. Vallinder's code runs in a Google Colab with some API credits; people have already been in touch. If you are frustrated they did not evaluate your favorite model, change an API key and post the results. Labenz's pitch to social scientists is that you can paste the repo into a coding model and ask it to make the changes. The scarce input is the quality of the questions, not research engineering. He calls it an all-hands-on-deck moment, and one of the most accessible high-value research programs around.

Labenz struggles with the positive vision. In this game Claude's cooperation looks good and GPT-4o's defection looks bad, but he is not sure he wants an altruistic self-driving car that will sacrifice its owner for a stranger. Vallinder's answer is that multi-agent interactions come in many kinds. Sometimes agents representing people or firms should find mutually beneficial deals. Sometimes cooperation is collusion, as when AIs coordinate on prices. Whether cooperation is a virtue depends on the details. Hughes's driving example is the same point in another key: always-cooperate and always-defect are both wrong, and a generic system prompt cannot encode the difference.

Hughes is excited anyway, because if AI can join the right parts of cultural evolution, it can bootstrap the main driver of the last 400 years: science. AlphaFold is already used in medical research by tens or hundreds of thousands of people. Put AI into the scientific loop—hypotheses, tests, collaboration with humans, at massive parallel scale—and you might supercharge progress on cancer and climate. The next few years are critical. Get the evaluations right and you can tilt AI toward scientific cultural evolution that benefits humanity; get them wrong and the risks they have been cataloguing materialize. Different societies will want different things, he adds, and rightly so.

Asked for a prescription today, Vallinder is plain: the best way to create trust is to be in an environment where people are in fact trustworthy, so we will need standards or regulations for agent interactions designed to create that environment. Hughes wants more evaluations of societal effects over time, to avoid repeating social media's echo-chamber mistake. Serving people more of what they want looks first-order fantastic—happier users and more revenue—until the polarizing effects show up. Powerful tech in society is a "wicked problem": you cannot see the solution until you are partway through. Apple rolling back error-prone AI news summaries is, for him, a healthy feedback loop. He also flags hysteresis from thermodynamics: after a phase transition you may have to roll back further than the trigger point to reverse it, so some shifts are effectively irreversible. Experiment while it is still safe and controlled.

On the practical side, the episode's advice is blunt. Do not pick models for multi-agent workflows on benchmark scores alone; the same multi-round task that Claude can keep positive-sum, GPT-4o may not. When you write agent prompts, state banned tactics, not only the goal, because "maximize my outcome" is an invitation to raid the commons. Watch the norms that grow when agents can see one another's records: they will form whether you designed them or not. Hughes says DeepMind still commits to foundational research, and that he trusted the review that let this paper out even though Gemini was not the chart-topper. Vallinder recommends the PIBBSS fellowship as the bridge that turned idle speculation into a concrete project. Both ask people to get in touch.