Forty games of Diplomacy, and nobody spotted the bot: Noam Brown’s 100,000x lesson

Sarah Guo and Elad Gil introduce Noam Brown as a researcher who has spent his career on game-theoretic agents, not on the prompt-to-image loop that now stands in for AI in public conversation. His path was not a straight line from a childhood AGI dream. Near the end of undergrad and just after it, he worked in algorithmic trading. The daily P&L felt like a game, fun and exciting, and not the most fulfilling way to spend a life. He went to the Federal Reserve for two years, hoping to figure out how to structure markets so they encouraged more prosocial behavior, and in the process got interested in game theory.

An economics PhD started to look slow: a good idea still has to get past legislation. Computer science lets you build. The most exciting game-theory work, he noticed, was already happening in CS departments, so he applied for a doctorate in algorithmic game theory and arrived in 2012. A professor happened to want someone for AI poker. Brown had played in high school and college, never for high stakes, and had even tried to write a poker bot as an undergrad; it did terribly and was a lot of fun. The original goal was not AGI. AGI, in 2012, was science fiction to almost everyone.

The majority opinion was that AI was, if anything, a dead field. He remembers emailing a professor to say he was interested and worried a PhD would leave him unemployable. A couple of years in, the pace of progress became obvious and he happened to be standing in the right place. AlexNet, which he met after a computer-vision class still teaching SIFT, threw the old feature pipeline out the window. The moment that drove it home for him was AlphaGo.

Deep Blue beating Garry Kasparov in 1997 taught the field that scale works — in that case, scaling search, not training nets. Those techniques failed in Go because you could not hand-write a decent evaluation. In chess you can add up piece values and get a sense of who is winning. In Go the position is too big, too subtle, too nuanced; a strong human can tell you who is ahead and not tell you why. People assumed humans were just better at that kind of pattern matching. An AI that did it better, even in a constrained game, was a wake-up call worldwide. Brown still tells people to watch the AlphaGo documentary.

After poker, the next-benchmark conversation was loud. People floated Hanabi, Werewolf, Settlers of Catan. It was 2019: GPT-2 had landed, DeepMind had grandmasters in StarCraft II, OpenAI had beaten human experts in Dota 2 after a couple of years of work. Catan, Brown felt, was a five-person, one-year crack. They wanted a problem that would require fundamentally new techniques, not a scale-up of something that already existed. The hardest game they could name was Diplomacy. An AI that negotiates in natural language with humans and strategizes with them still felt like science fiction, even knowing the decade’s successes. He was a little afraid to aim that high. All research is high-risk, high-reward, he says, or at least it should be.

The most unexpected result was not a clever alliance. It was that Cicero was not detected as a bot. There is no honest dry run: if you hire people to play, they know they are in an experiment. If they know there is a bot, they behave differently, and the exercise collapses into a Turing test. Diplomacy is a small community. Players talk. The fear was that within two or five games someone would figure it out, word would get out, and every later game would be a hunt for the machine. The team told itself that ten games before exposure would be lucky. They got the full 40.

Brown reads that as a testament to language models, and also to how humans explain weird speech. When the bot said something odd, which it did, the first instinct was not “this is a machine.” It was that the other player was dumb, distracted, or drunk. Bot sits at the bottom of the list. One message still scares him. A human said they were nervous about Cicero’s units on the border. The bot was not planning to attack; it was going the other way. It wrote something like: I totally understand where you are coming from, I can assure you 100 percent I am not planning to attack you, I am going the other direction, you have my word. He would never have expected that from a bot. Seeing it, he says, you appreciate that there is something really powerful here.

Cade Metz had a New York Times piece out that day arguing the Turing test is kind of dead. Brown agrees it is no longer a useful measure in the way it was intended. The bots are close enough that the test has lost its bite. That does not mean general intelligence has arrived; there is still a long way to go, and a lot these systems cannot do well. The progress, especially in the past few years, has been staggering. What he wants instead of a chat test is reasoning. Most systems are next-word prediction. Cicero is different: it conditions its dialogue on a plan. Chain-of-thought, he notes, was a shockingly effective simple idea — add “let’s think this through step by step” and the model rolls out a longer thought process it has seen in human data. The open question is whether you can improve that process at each step, not just replay it.

Gil has been asking researchers when the easy data runs out — when labs must start paying for synthetic data, RLHF, or even bounties for people to record their days. Brown’s answer is not the one that conversation usually wants. It is not clear that data is the bottleneck on performance. He has talked to other AI researchers; they are less worried than the public story suggests. There is more data out there than people realize or currently use, and sample efficiency should improve, which stretches what you already have.

The bottleneck he sees is scaling spend. Models today, he says, probably cost on the order of $50 million to train. A 10x is easy to imagine; he would not be surprised by a $500 million model in the next year or two. Another order of magnitude, $5 billion, is conceivable if you are the U.S. government or a very big tech company. A $100 billion model might still help and is no longer realistic. Call it two more orders of magnitude, and then a big problem. Efficiency and parallelization will squeeze some more, and a lot has already been squeezed.

That is why he is interested in a dimension people are not scaling: compute at inference time. Spend $50 million pretraining, then a penny per answer. What if the answer takes an hour, or ten seconds, instead of one? People sit and think when they want a better answer. The models mostly do not. Shifting cost from precomputing everything to thinking on the fly is, in his view, how you get past the training-spend wall.

Diplomacy does not come with internet-scale logs. After scouring the web, Brown found one site with a meaningful corpus: webdiplomacy.net, a casual community about twenty years old, roughly 50,000 games and 13 million messages. Another popular site periodically deleted its history because it took up server space — sitting on a gold mine and throwing it out. Other sites refused. Without that deal, he says, the project would never have happened. Fifty thousand games is a good set and not enough to train a bot from scratch. A pretrained language model, fine-tuned on the Diplomacy chat, produces a bot that can actually talk in the game.

Talk is not strategy. You cannot learn a really good policy in these games with supervised learning alone, and most of the players in the log are weak; the bell curve sees to that. The finding is not Diplomacy-specific. The team ran the experiment in chess and Go: pure supervised learning on a giant set of human games does not give you an expert, even if you condition the model to behave like a grandmaster, because it is not doing any planning. To get past average — even past strong human — they had to do self-play, the same recipe as AlphaGo, AlphaZero, the Dota 2 bot, and Brown’s two-player and six-player poker bots: millions or billions of trajectories against yourself.

Diplomacy adds a cooperative trap. Train a negotiator from scratch with no human data and it may learn to negotiate in a gibberish robot language. Drop it in with six humans and they work with each other instead. The same thing happens in the moves. A self-play bot develops norms: I will support my ally into this territory because it is obvious they should go there, no need to talk. Humans have a different meta-game. If you do not share their conventions, they stop working with you. Cicero’s fix was to build a rough model of human behavior from the logs, then use self-play as a modifier so the policy gets stronger without becoming incompatible. Gil notes the real-world rhyme: a self-driving car that assumes every other driver is a perfect machine is a dangerous car. Brown agrees. Usefulness in the world, not just beating people at chess, means coping with human mistakes and human cues.

Games have been AI benchmarks since the 1950s, chess held up as the grand challenge on the theory that a machine as smart as a grandmaster would be smart at everything else. That was a false promise; a chess engine does not do much else, even if the field learned a lot. Games still help because the human top is an objective bar that existed before the researchers arrived, unlike a benchmark invented to flatter a technique you already have. Brown thinks individual recreational games are now past being an interesting challenge. Diplomacy was, in his view, the hardest game to make an AI for. He cannot name another where success would still make him say he did not think it was possible. The next benchmark is generality: one system that plays these games at a superhuman level and also does image generation and question answering, so research cannot overfit a single board.

Asked where humans will stay dominant, he starts with the physical world. Robotics is lagging; he is trying to avoid it because software is nicer. Writing the next Harry Potter is not here yet and might be five years away, which he finds a little scary. He struggles to name domains where AI will not eventually surpass humans. The clear present advantage, he says, is generality — learning from a small number of samples across many domains. Gil objects that the bar is higher than we use for people: someone good at one thing is usually not good at everything. Brown restates it as sample efficiency. How many games to become a good chess player, Diplomacy player, or artist? Orders of magnitude less for a human than for an AI. That is decisive in domains with little data. It has not been overcome. It could be.

On financial systems, he is sure deep learning already powers a lot of trading. He used to work in finance; poker-loving shops invited him to talk. Is reinforcement learning actually useful for trading? The answer he usually gets is no. Markets are non-stationary. Historical data goes stale; the tape responds to world events. You want a method that understands the world, not a black box. AIs already help in limited ways, such as breaking up large orders. He flags the caveat: this is maybe five-year-old knowledge, and the cutting edge that makes money is not something people tell him. Negotiation bots, by contrast, look near. In constrained settings — the price of a good, maybe a salary — he thinks AIs could already do better than humans. Contract negotiation still has too much nuance for a professional to be replaced. Guo and Gil mention that, before the taping, multiple people had already emailed them to say they were building price-negotiation agents. Brown thinks that is the right call.

A former colleague’s summary of Brown is that he picks a neglected domain with promise, commits, and becomes the best, while other people chase whatever is in vogue. What he wants next is the reasoning problem, motivated by the games. AlphaGo is held up as a deep-learning milestone, and it was not doable without deep learning, and deep learning alone did not do it. Strip out the planning and use the raw policy network, and it is substantially below top human Go. Chatbots and image models are powerful raw nets. The raw net still cannot play Go. Monte Carlo tree search, the extra algorithm, is very domain-specific. It works in chess and Go, the classic testbeds. It does not work in poker. It does not work in Diplomacy. That, to someone who worked those domains, is a major weakness. The prize is a more general reasoning and planning system.

If you actually solve that in a general way, the imagination gets expensive in a good sense. It does not seem crazy to him that a model could prove the Riemann hypothesis within five years. Maybe inference costs a million dollars per token. If you pull it off, that is worth it. Maybe the same machinery writes a prize-winning novel or finds a life-saving drug. Code is on the list: next-token prediction is getting surprisingly far and will not get you to replacing engineers who have to plan a whole product. Copilot is local context. Scaling the training and the net until you no longer need planning is the usual objection. He crunches AlphaGo: matching search with a bigger raw net is about 100,000x. These models already cost on the order of $50 million. You are not going to scale them 100,000-fold. AlphaGo’s move was to spend about 30 seconds thinking about the next move during the game, shifting the burden from precompute to thinking on the fly. That, again, is the missing piece.

Gil asks why the architecture should be general at all. The brain is modular: visual cortex, emotion modules, famous cases of a pole through a specific region ablating a specific function. Why not a pile of specialist systems? Brown will not marry a technique. The answer might be more specialized systems rather than one truly general architecture. What he is stubborn about is the goal: something that succeeds across a wide variety of domains. Inventing a unique method per domain gets you part of the way and, he thinks, gets superseded. Deep learning itself was a supposed dead end, tended by a few people in the Canadian wilderness. Diversity of approaches has value. Try something a little different from what everybody else is doing. On what other researchers should work on, he will not name a topic. He will name a style. Research is inherently risky. There is a high probability that what you are working on will not be useful in the long term. Accept that. Early in his PhD, a lot of the work was not that useful in the grand scheme. That was okay, because one thing was. Guo’s send-off: be like Noam, work on things that make you nervous.

He then gives the rules, because the breakthrough is easier to see with the board in mind. Diplomacy was designed in the 1950s by someone who had watched World War I as a diplomatic failure and wanted a game that taught people to be better diplomats. Seven powers at the onset of that war — England, France, Germany, Italy, Russia, Turkey, Austria-Hungary — negotiate every turn in private, then write moves that are revealed together. You can keep a promise or stab. Only one player can win, and early cooperation still helps both sides, Hunger Games-style. Brown calls it a mix of Risk, poker, and Survivor. It was JFK and Kissinger’s favorite game. AI papers on it go back to the 1980s. Playing it in natural language with humans and beating them was complete science fiction until a few years ago. In 2019 several labs, including his, DeepMind, and Montreal, started deep-learning bots for the no-press version, communicating only through moves. His group took the risky jump to full language. It was, he says, the first major game-AI breakthrough in a game that involves cooperation — the capability you actually want if the point is usefulness in the world, not a trophy. Centaur play is already fading in chess; in Go, humans still catch occasional weird blunders; in Diplomacy an experienced human plus the AI is still stronger. He expects that window to close until the human is a marginal add-on, the way of chess.

Brown’s PhD was heads-up no-limit Texas Hold’em, a challenge problem with a long pedigree. In John Nash’s original game-theory papers, the only application discussed is poker; Nash even worked out a simplified three-player game by hand and noted it would be interesting to analyze something more complex. Sixty years later, they did. After AlphaGo, the question became what AIs still could not do. The remaining hole was hidden information: the other player knows things you do not, and you know things they do not. Several labs built a bot every year and played them in the Annual Computer Poker Competition, which had turned into a scaling contest. There are about 2.5 billion possible hands on the river. Before neural nets, they used k-means to bucket similar hands — 5,000 buckets the first year, then 30,000, then 90,000 — trained longer, parallelized more, and beat last year’s model. In 2014 they won the competition and took the bot to four top heads-up no-limit pros for 80,000 hands. It lost by a sizable margin.

The humans were doing something the bot was not. The team had trained for two months on 1,000 CPUs; at the table the bot acted instantly. The pros, in a hard spot, sat and thought — five seconds, sometimes a minute. After the match Brown analyzed what a planning algorithm at decision time would be worth. The answer was about 100,000x, equivalent to scaling parameters and training that much. Three years of PhD scaling had bought about 100x, which he had been proud of. Next to search, it was a footnote. He worked essentially non-stop for a year, 100-hour weeks, throwing as much computation as possible at inference. In January 2017 they played four experts again, $200,000 in prize money to keep them honest, and crushed them. Pros said they did not think it was possible to beat expert players by that kind of margin. The later six-player bot won with a more scalable search that looked only a couple of moves ahead, and it cost under $150 to train on a cloud bill. That, he emphasizes, was an algorithmic breakthrough, not a compute story. The same result could have been had twenty years earlier if someone had known the approach.

The bot also changed how humans play. Action was discretized because betting $5,000 and $5,001 is not worth distinguishing. Near the end they had spare compute, so they tossed in extra sizes — 4x pot, 10x pot — without expecting the bot to use them. It used them a lot, including $20,000 into a $100 pot, unheard of in professional play. Brown thought it might be a mistake; so did the humans, until they kept landing in awful call-or-fold spots. If the other person is really struggling, he says, you are doing something right. Afterward the pros said overbets were the one thing they would take home. Typical old strategy was a quarter-pot to pot; 5x and 10x are now part of the toolkit. Training looks like chess: play, then let a bot mark the mistakes. Brown describes poker as high-dimensional chess, reasoning over a distribution of actions rather than a single discrete move.

A lot of people did not believe there was fully optimal play in poker, especially live, with social cues. Those have been swept out of the winning strategy, if not out of the fun. A Nash equilibrium, carefully stated, does not mean you always win. It means that in the long run you will not lose in expectation. Two equilibrium players tie. In a game as complicated as poker, the other person makes small mistakes, and every mistake is money in your pocket. You play Nash, wait, and win. That is now conventional wisdom: start at equilibrium; if you are really good you can deviate to punish their deviations, but the safe thing is to sit still and let them pay you.