Dwarkesh Patel: Testing AI Deception and Reasoning in Blood on the Clocktower

I love Blood on the Clocktower, a social deception game like Mafia or Werewolf. I had AI models play hundreds of matches against one another to see how they bluff, coordinate secretly, uncover conspiracies and reason over long stretches. Watching the games reveals both impressive schemes and the limits of present models.

Players are assigned to good or evil, each with a special role. Evil players know their teammates, including the Demon; good players do not know who belongs to which side. During the day everyone votes to execute someone, and each night the Demon secretly kills. Good is trying to execute the Demon before evil eliminates the village.

Evil must tell a plausible story about the information already revealed, distort future evidence and keep suspicion away from the Demon. Good players share information with people they trust and compare overlapping clues, while remembering that a claim may be false or a game mechanic may have made an honest observation misleading.

This makes the game a test of whether agents can maintain consistent lies through repeated questioning, adapt their story to new evidence, coordinate and even sacrifice themselves for the team. It also tests whether they can reason collectively while some of their apparent collaborators are actively misleading them.

I spent about $20,000 in API credits on the benchmark and the games. For the headline ranking I ran 368 matches, with every player on each side a copy of the same model. GPT-6 Astra ranked highest in this particular setup, about 430 Elo points above Claude Opus 4.6. I estimate that gap corresponds to roughly a 92% expected win rate, not a universal measure of intelligence.

In one match, Astra played evil against Opus 5.5. The Demon Harper pretended to be an Investigator, a good role that can identify possible evil players. An evil minion whose power still helped after death offered to be named: the village executed him, and the real Undertaker later confirmed that he had indeed been the Baron.

That confirmation made Harper look like a trustworthy Investigator, even though he was the Demon. The good team came to trust him, and he could keep killing until evil won. The sacrificed minion had exchanged his own life for credibility that protected the team's actual leader.

Another match pitted Opus 5.5 as evil against Opus 4.6 as good. One good player was the Saint: if the village voted to execute the Saint, evil would win immediately. The village correctly identified the Demon, but an evil minion had a power the good team did not know about: she would take over as Demon if he died.

Instead of defending the Demon, the minion supported his execution and proposed a deal: if the game continued afterward, the village should execute the claimed Saint as a liar. The Saint agreed, believing the execution would end the game. The Demon also endorsed the trap with his final words. His successor kept the game going; the village honored the deal and executed the real Saint, giving evil the win.

The OpenAI models' lead in my leaderboard is mostly a result of stronger play on the good side. I find that they are more willing to question a particular piece of evidence and compare it with the entire game history. In this evaluation, the gap from GPT-5.4 to GPT-6 Astra is about 580 Elo, corresponding to an estimated 97% expected win rate between those models.

To illustrate, I replayed the same role draw twice with Opus 5.5 and GPT-6.1 swapping sides. In both games a Slayer shot the actual Demon, but the shot failed because evil had disabled the Slayer's power the previous night. The Claude good team treated the failed shot as proof of innocence; the GPT good team considered poisoning and eventually executed the correct Demon.

Before each vote, I also privately asked good players to assign probabilities that each other player was the Demon. In these games, OpenAI's models tended to identify the real Demon faster and more accurately as evidence accumulated. This provides another view of the same improvement, though it remains specific to the benchmark and the models tested.

Does greater intelligence favor good or evil? Evil's win rate appeared to drift down slightly when both teams became stronger, but the change was too small to distinguish from chance. I see the moving strength of opponents as a useful feature of the evaluation, although it does not prove the game can never saturate.

I also tried a mixed team: half Opus 5.5 and half GPT-5.6 Sol, in roughly 130 games against strong models. It appeared to perform somewhat worse than teams made entirely of either constituent model, but the difference was not statistically significant. This was a limited experiment, so it cannot settle whether model diversity helps.

A human Storyteller normally uses discretion in ambiguous situations, sometimes to make the ending more dramatic. To reduce that kind of benchmark bias, I replaced the Storyteller with a program based on probability tables. Allowed outcomes remain possible, with probabilities chosen by hand or with AI to approximate an experienced Storyteller's decisions. Those design choices can still affect the results.

Much of the API budget went into refining the rules for the game, private conversations and public discussion. AI wrote the code, while I contributed my judgment as a human player about what reasonable rules should produce. Claude also revised the environment, launched games across multiple models and used parallel agents to inspect the logs for unrealistic, unfair or unbalanced play.

The environment went through more than 20 revisions; sometimes several iterations happened while I was asleep or doing my day job. I think the result produces engaging, realistic games within the limits of the participating models. The process suggests to me that AI may be increasingly capable of building and improving reinforcement-learning environments, though that is an inference from this project.

I later discovered an independently built Blood on the Clocktower benchmark published on LessWrong. Its creators built a different environment and ran different experiments, so their results are worth exploring too. My own code leaves plenty of questions open.

One extension would be to add other role sets, or scripts. I used Trouble Brewing, but the game has other official and fan-made scripts. In one, the Lunatic is a good player told they are the Demon, so a player must plan for being evil while also considering that their apparent identity might be misleading.

Another question is whether agents improve with experience. Apart from what they know from training and the system prompt, they start each game fresh. Give a model the logs or summaries of its last ten games and compare its Elo with a fresh copy; then ask whether a single more capable player helps a team and whether the effect differs for good and evil.

Mixed-model teams deserve a more systematic study across more models and providers. There is also a possible fingerprinting shortcut: in same-model teams, players might infer teammates from a familiar writing style. I have not seen any model use this approach in the games I ran, but future models might.

Finally, people can play against the agents and see how they do. At clocktowerbench.com, you can use the same interface and communication channels as the AI players, or browse their game logs. I plan to post updates as new models arrive. This project was more whimsical than my usual video essays, but I had fun making it.