The Night AlphaStar Beat TLO, the Team Went to a Bar

Lex Fridman opens with a simple origin question: for Oriol Vinyals, did a love of programming come first, or a love of games? Neither, exactly. As a child of ten or eleven he liked computers enough to mess with them, break them, and put them back together. He did not code much. The real pull was games, and above all the first StarCraft, released in 1998. In Europe at the time there was no esports industry to speak of. What Vinyals calls playing “pseudo-professionally” meant getting very good inside a circle of friends, then later landing consistently in the top 32 in Europe for a couple of years.

He liked to play random, sometimes even in tournaments. The reason was practical. StarCraft has three races, and the way you beat someone is not only how you play your own side. If you have lived inside a race, you know what annoys it, and you know where to poke when you are on the other side of the map. His favorite, and his best, was Zerg. Fridman asks him to explain the game to people who have never played a real-time strategy title. Vinyals reaches for chess and then takes it apart. You start with almost no pieces. You gather minerals and gas, choose how much to invest in economy versus army, and attack across a map you cannot fully see. Unlike chess, the game is partially observable and continuous. Scouting improves your information and can give information away. Speed and click accuracy matter, and the professionals, he says, are astonishing to watch in person.

The social world around those games shaped him as much as the mechanics. Battle.net, once internet connections in Spain became usable, was not only a ladder. It was a chat universe, a way to meet people you would never meet on a street. He later played World of Warcraft, which he found less stressful because missions could always be finished. He still has Facebook friends from that period, including a warrior he met while playing a priest in a virtual city. Someone told him, in those years, that you could choose only two of three: a girlfriend, video games, or studying. He thinks the line was mostly true. StarCraft, played seriously, is a hard consumption of time. When he started university he stopped completely.

The AlphaStar story starts earlier than the 2019 matches. In 2014, after Google acquired DeepMind, the Brain and DeepMind groups held a joint summit in California. Vinyals, then at Google Brain, presented Berkeley Overmind, a StarCraft I bot he had built at Berkeley. He is blunt about what it was: a pile of rules. If you have more than three units, attack. If the opponent has more units, retreat. Demis Hassabis came over afterward and said, in effect, maybe not now, perhaps a bit too early, but Vinyals should come to DeepMind and do the same thing with deep reinforcement learning. At the time it sounded like science fiction.

In 2016 he moved to London and joined DeepMind from Brain. AlphaGo had just beaten Lee Sedol. Blizzard reached out and asked whether they wanted the next challenge. The pieces landed together. Before writing an environment, Vinyals flew to Blizzard’s headquarters in Irvine to explain how the project would work. From day one the DNA was set: the behavior had to be learned. The if-else style of Overmind was the path they were refusing. Fridman presses on the distance. Go and Atari, honestly described, sit far from StarCraft. Did Vinyals believe a 2018 or 2019 result was even in range?

His answer is the one that organizes the rest of the conversation. The full thing — a single neural network, no domain rules, playing a complete one-versus-one game — “I just didn't even think it was possible.” What he could see were stepping stones. StarCraft can be cut into subproblems: openings, macro, battles, micro, each of them attackable on its own. Blizzard opened a huge set of human replays to the community, not every game ever played, but enough that you can query a day’s worth of new data. For someone whose prior work was language, sequences, and supervised learning, that dataset was a precious and unusual resource. The third stone was the team. Hassabis had already seen the direction in 2014. Vinyals says he was following that lead.

Fridman offers the usual list of StarCraft horrors: imperfect information, long-term planning, real time, a huge action space, the absence of a known Nash equilibrium. Vinyals does not pick partial observability. He picks exploration. Once they defined observations and actions, it became obvious that you cannot search StarCraft the way you search chess or Go. Deep reinforcement learning, in the naive form, issues random actions and hopes that a win arrives often enough to learn from. In the early game, almost anything you click is bad. Workers are already mining minerals for free. Any action yanks them off the line and sends them wandering.

Expanding, the most basic macro skill in the game, is worse. You must select a worker, walk it to a location, place a building, wait for it to finish, then send extra workers to mine. The location matters. Random clicking will almost never produce that chain, and even if it did, connecting the chain to a later win is another miracle. The game ticks about 22 times a second. That is a vast number of time steps in which something can go wrong. Partial observability and the lack of a single perfect strategy are interesting, Vinyals says, and they are seeing more of those problems now. The first barrier, the one that blocked every early breakthrough, was exploration.

They learned what pure reinforcement learning actually discovers. Take all your workers and attack with them. In hindsight it is obvious. Self-play starts from two networks that know almost nothing. One random action sends a worker toward the enemy base. The opponent does not know how to defend. Credit assignment in a long game is so hard that the first strategy that yields wins is a worker rush. Imitation from human replays was how they stepped around that wall. Get the policy near human play, then use self-play to push it up. Without that bootstrap, the agent never reaches the interesting part of the game.

Everything in AlphaStar passes through a policy, and the policy is a neural network. Vinyals likes the starkness of that fact. He could hand you the weights, and if you fed the same observations and decoded the actions the same way, you would have the agent. Nothing else is required. Observing the game took some experimentation. The version they settled on mixes a zoomed-out spatial map, a view of the current camera or screen, and a list of visible units treated as a set of objects. Each unit carries properties: health, position, type, whether it belongs to you or the enemy. The list is not strictly necessary — they had versions that played well without it — but it is a natural encoding, and it helps a lot.

Fridman notes that a unit list is uncomfortably exact compared with a human staring at pixels. Vinyals agrees the exactness is a difference, and also points to a computer-vision paper from Facebook that scrambled an image into a list of pixels, kept the XY coordinates, and ran a Transformer on that set. The same architecture had just produced strong results in machine translation. Without intending to copy that paper, AlphaStar was doing something similar: a unit is a “pixel” with a location. Once you accept that, a StarCraft observation is a sequence problem. It has the same shape as a sentence.

Time makes the analogy tighter. Because the game is partially observable, the agent must remember what it has seen — a worker walking off, perhaps toward an expansion in the corner of the map. The actual task is: given all observations so far, and all actions taken so far, predict the next action. That is machine translation. It is also language modeling. The architectures are the same ones Vinyals had used on words, LSTMs and Transformers, now integrating a past of units and clicks instead of a past of tokens. Step one of AlphaStar is imitation, the way a translation model imitates human translators. The objects are more complicated than words. The code, he says, is almost the same.

Bootstrapping from replays does not produce humans. The team trained on games at about 3,500 MMR and above, a threshold set low enough to keep the dataset large. MMR is StarCraft’s matchmaking rating; the best players in the world sit around 7,000, and 3,000 is roughly gold. The interesting trick is that they tell the network which level it is imitating. This replay is 4,000. That one is 6,000. After training, you can ask the same policy to play like a 3,000 player or like a 6,000 player, and it actually changes. Economy gets worse at the low setting. Actions per minute drop. Skill, or a statistical cartoon of it, is a bit you can flip.

They tested the high-MMR imitation. It beats every built-in AI Blizzard ships with the game. It is nowhere near a real 6,000 player. Vinyals puts it around gold or perhaps platinum. The gap is conceptual, not just a shortage of data. So far the network has only been asked, given this screen and this history, what the next action would be. It has not been asked to win. Experiencing wins and losses is what refines a policy that would otherwise go off-policy and drift. Between “where do I click next” and “how do I win” sits the reward, and credit assignment across a long game.

The human comparison is personal in a small way. Vinyals has not seriously played StarCraft II; he is unranked. The best player at DeepMind was about 5,000 MMR, high Masters, not Grandmaster — Grandmaster is the top 200 in a region. Vinyals was decent at the first game, consistently top 32 in Europe, but he delayed playing AlphaStar by a week while the team needled him. When he finally sat down, it beat him badly. Micromanagement from StarCraft I still worked. The sequel’s systems did not. Imitation can clone a style. It does not automatically clone competence.

The interface gap that is hardest to simulate is cloaked units. In StarCraft some units are invisible unless you have a detector; without one they can raze buildings and kill workers, and you cannot target them. Humans still see a shimmer, a faint space-time distortion on the screen. Professionals notice it immediately once they have trained the habit, but it demands attention. For AlphaStar, simulating whether a person is looking at that pixel is awkward. The practical substitute is a bit that flips from zero to one: there is an invisible unit there. The agent still cannot shoot it without a detector. The knowledge is sharper than any human’s glance.

Actions raise a louder argument. StarCraft has been an AI environment for years — Vinyals competed in the first bot contest in 2010 — and there has never been a clear rule for actions per minute. Scripted bots issue 20,000 to 40,000 actions a minute. Professional humans sit around 300 to 800, and a chunk of that is spam, warming up the hands so that the important clicks land. Imitation gave AlphaStar a human-like rate for free, including the meaningless repeat clicks. The blog post’s APM curves look like the human distribution. Self-play does not promise to keep them there. Precision can rise. The rate can climb.

Their first attempt at a ceiling was to look at professional APM over short windows and drop actions past a cutoff. Vinyals admits the cutoff was probably set too high. Pros and gamers mostly felt the play was human-like, but some agents combined slightly too much speed with too much precision, and the community started a fight worth having: should the team have limited the agent at all, or let it loose and see what it invents? Modeling shimmer is hard. Modeling muscles, accuracy, and fatigue is harder. Every extra constraint makes the agent more human and less of a pure learner. They are on the first iteration of that line, not the last.

The public demonstration used only Protoss versus Protoss. Protoss is the technological race: units are expensive and powerful, so you try to keep them alive and win on spells and tactics. Players Vinyals spoke with, including TLO and MaNa, told him people think Protoss is the easiest of the three. Easiest does not mean it dominates. The top of the ladder stays mixed because Blizzard wants it that way. Zerg sits at the other pole. You expand, take as many bases as you can, and treat a dead army as a renewable cost. Zerg play is pressure, trades, and a huge bank of resources that rebuilds what just died. Each race still has units that break the stereotype. Diversity is the point.

Openings, Vinyals says, come in a few families. A standard opening balances early units, defense, and a quick expand, then chooses a technology path: air, cloaked units, big units that beat some compositions and lose to others. That choice is rock-paper-scissors. If you scout well and guess the opponent’s distribution, you play the counter. A typical game is that loop running continuously. You try to put paper on the table before the opponent switches from rock to scissors. The belief state over the fog of war is what a good human maintains. Scouting is the decision to buy a more accurate belief, at the cost of revealing something yourself. In AlphaStar that belief is still implicit. There is no extra loss for predicting what sits on the other side of the map. Vinyals thinks making it explicit would help.

Then there is cheese: an all-in sneak. Hide buildings next to the enemy base. Hide your tech and hit with cloaked units before they can detect. If a human spies an empty base, they know something is up — maybe flood units, maybe block the ramp with workers. Defending cheese is a skill of its own, and in the AlphaStar League many agents grew cheesy strategies. Two of the ten agents shown against TLO and MaNa played that way. A milder cousin is the timed all-in: a perfect composition and a just-finished upgrade at, say, four and a half minutes. If the push lands, the greedy opponent dies. If it fails, players often type GG even though they still have a base, because they gambled and lost. Those niches are why a league exists. Pure self-play will not grow every unpleasant opponent on its own.

If one agent only plays itself, it converges onto a small subset of strategies and gets stuck. The AlphaStar League is the fix. It is not pure self-play. It is a population with personalities. Some agents become cheese specialists. Some become greedy economists, weak early and terrifying later. Some lean into early aggression, some into late power. A new agent trains against a snapshot of the whole league and searches for the best counter to that distribution. Then it is dropped back in, and everyone else has to keep evolving. Vinyals compares it to Battle.net. You meet a strategy you have never seen, you get crushed, and you immediately want to try it and queue again. The league is an artificial version of that pressure.

You have to keep those personalities in the population. If they vanish, you are no longer covered against what exists in the wild: cheeses, early rushes, greedy macros, drops into the mineral line. Pure self-play finds some of that landscape and not all of it. Creating a Battle.net for agents was, for Vinyals, another critical piece of the problem, on the same level as imitation. The two cheesy agents in the TLO and MaNa games were not a script the team wrote. They were a niche the league forced into existence.

Fridman notes the 5–0 and asks how hard it is, in general, to win. Vinyals pushes back on the headline. Superficially you watch AlphaStar, you notice a camera-interface game that went the other way, and you think the game is done. That is not a claim they can make. StarCraft bots have been a project for years; a moment like this had not happened. The agents play at a Grandmaster level. They understand the game well enough to be extremely strong. They are not unbeatable, and they do not play perfectly. In a domain full of sneaky strategies, a huge risk can still steal a win. Humans who learn Protoss also pick up Terran much faster than AlphaStar can. The weights do not transfer. The algorithm does.

The players, he adds, did not know what to expect. Their prior over this “alien brain” was off, which made the matches exciting and also means the result could move once humans study the replays. Watch closely and the holes are visible. Sometimes AlphaStar did not scout. Cloaked units at the wrong moment would have gone badly. There is still a lot of work. What stunned the team was simpler: a single neural net on a GPU playing live against people who, in person, are extraordinary. Fridman jokes that someone in Poland is already training to make sure it never happens again. Vinyals likes that. Holes to exploit are how the game, and the research, stay alive.

Fridman asks how it felt. Vinyals calls it unbelievable, and already a little sad, the way you get nostalgic for a night while it is still happening. He is a pessimist. He emailed the team before the TLO match and asked everyone to predict the score. His own pick was 0–5. They had calibrated against a 5,000 MMR player; TLO was much stronger than that, even on Protoss, which is not his main race. Vinyals thought of the series as a test run. They won 5–0 instead. That evening they went to a bar to celebrate, and David Silver said, why not invite someone a thousand MMR stronger who actually plays Protoss? A thousand MMR, Vinyals notes, is roughly a 99 percent chance that MaNa beats TLO in Protoss versus Protoss. They had some drinks. He said sure. He also thought beating MaNa would be impossible.

The MaNa match mattered more to him. After TLO, the professional barrier was already crossed. MaNa was at the top, and you could see it, and the agents got better too. After the first game Vinyals told himself that a single win would be enough to say they had taken a game off a true Protoss specialist. He remembers hugging Hassabis. He calls it a moment that will resonate for him as a researcher and as a person, probably the highlight of his career so far, not because of citation counts but because the whole machine-learning-through-games bet came together in a room. Watching MaNa was like watching Kasparov: the off-race excuses were gone, and a human being was simply superseded. Vinyals still wants the thing the network does not do yet. He wants an agent that models the opponent, lays traps, and manufactures false beliefs. Poker and theory of mind, he says, were made for StarCraft.

Where do the AlphaStar tricks go next? Vinyals feels the usual tension between a grand challenge and the research it forces into existence. Capture-the-flag agents and Dota 2 have already shown that other hard games are moving. The lowest-hanging fruit, he thinks, is feeding techniques back into the community they came from: sequence modeling, Transformers, pointer networks, combinations of LSTMs and Transformers. Fridman then jumps old-school and asks about Alan Turing. Is the Turing test a grand challenge worth taking, and would Vinyals reformulate it?

He loves the test, because he loves sequences and language. Early neural machine translation work at Brain was even pointed at a chatbot, a system that would never have passed. The idea of an AI indistinguishable in conversation still fascinates him. Feasibility is the catch. He cites an essay by Hamming on the grand challenges of physics. Teleportation and time travel are magnificent. There is no attack path, which is why people do not write dissertations on them. Vinyals sees the full Turing test the same way. Language models, GPT-2 being the most recent example at the time, are impressive. Fooling a human into believing there is a person on the other side of an hour at a bar is still far. He would not do a PhD on solving that test, and he would not recommend it. It is too early, or too hard, in the specific sense that he cannot see the plan.

Fridman cannot resist the parallel. Vinyals said the same thing about StarCraft a few years ago, and then he was on the team that beat a professional. Progress is unpredictable; six months earlier he would not have predicted Grandmaster-level agents either. Vinyals does not budge on the distinction. For StarCraft he knew the steps: imitation, then a Battle.net for agents. A little more was needed, not much more. For the Turing test he does not know what the execution plan looks like. His worry is not that a breakthrough is impossible. His worry is that “the statistical approach to it is not gonna cut it.” Related sub-challenges feel timely — a better, fully neural Google Assistant, for example. The full test does not. He judges problems by whether a roadmap exists, not by whether they sound important.

If he had to name the current limit of deep learning in one word, Vinyals chooses generalization. What the field does, in essence, is fit functions to data. Shift the distribution, or even stay close and add a little noise, and the model breaks. Adversarial examples are the cleanest exhibit: perturbations a person cannot see, outputs a person would never produce. He reminds Fridman why support vector machines had a moment. They came with guarantees about unseen data. Today’s networks are matrix multiplications stacked together. Numerically, corner cases exist, and you can find them. He does not see much progress on generalization in the strong sense of the word.

Is the answer just a bigger training distribution, grown until it contains the test set, or is it a non-statistical methodology? The path that has actually worked — in StarCraft, in translation, in language — is scaling data and scaling the model. It is, he says, maybe the only formula deep learning still delivers. Tasks people swore could not generalize start to, once the scale is there. He does not think scale resolves the thing underneath. He is more excited by combining neural networks with programs and discrete decisions. You can train a network to sort numbers, and with enough examples it will look like a sorting algorithm. Give it a number that is too large, and it collapses. Write a sorting program instead, and you can prove it works for every input. That is the difference between statistical confidence and a guarantee.

Does that mean dusting off 1980s expert systems? If the problem is important enough — detecting cancer, forecasting weather — then yes, use rules, use domain knowledge, and let a bit of machine learning make the system better. They even thought about mixing state machines with networks for StarCraft. If the goal is to test the limits of reinforcement learning, you should not look at imitation data or hand-written domain rules. Both attitudes are fine as long as you are honest about which game you are playing. The through-line of his own work is sequence-to-sequence: anything in, anything out, including an image to a caption. He once changed a line of code in a translation model, came in the next morning, and watched it write captions that seemed to work. Vectorize the input, pick the architecture of the month, and with enough supervised data the formula keeps working — up to those generalization issues. In StarCraft they also refused to observe every frame. The network acts, then chooses how long to wait. Most games collapse to maybe a thousand or 1,500 actions, a length LSTMs and Transformers can hold, especially under imitation. Reinforcement learning over the raw 22-frames-a-second stream is a different, much harder credit-assignment problem.

Fridman asks how new ideas actually arrive, and mentions Ian Goodfellow’s story that GANs showed up after a few beers. Vinyals laughs. Their story is similar. The decision to play StarCraft against the pros again was made in a bar, a week after the first match, and he later told it at a DeepMind retreat as the same genre of tale. The method underneath has changed with his career. Early at Brain he was surrounded by Geoff Hinton, Jeff Dean, and Ilya Sutskever, and the work was discussion, papers, and more discussion. He still thinks you should never stop reading. What he does more of now is strategy: is this domain ready, not too early and not too late?

The formula that has worked best for him is simple. Find a hard problem. See how existing research fits. The places it does not fit are where you have to invent. Machine translation produced sequence-to-sequence. Combinatorial algorithms produced pointer networks. StarCraft produced large-scale imitation learning and the AlphaStar League. The other direction also works: pick a research topic, model-based RL say, then find a minimal environment that can test it. As a graduate student he had no idea about the strategic layer. He recommends feeding forward from what success would look like and backtracking, rather than grabbing whatever looks cool. You want a team that holds both silly optimism and hard skepticism. You want colleagues who will try ideas you think are bad. He has been proven wrong many times. “Your colleagues are more important than yourself.”

AGI, he says, is often badly defined, so he gives himself an operational version. The learning algorithm behind AlphaStar already generalizes: you could point it at another real-time strategy game, or at many other domains. The weights do not. A network trained on Protoss versus Protoss is useless for Terran versus Terran; you throw it away and retrain. A result that would start to look like AGI is an approach that observes new problems and solves them without restarting that process — meta-learning, the skill of learning, at something like human speed. Show a person a new object and they recognize it. Hand them a new Atari game and they are decent in a few sessions. Léon Bottou’s ICML talk, which Vinyals likes, said the train/test split itself should be broken. A network that knows ImageNet should not fail at MNIST. It currently does, and that feels wrong. He is also a fan of Wikipedia, and slightly aggrieved that neural nets leave structured knowledge on the table. A knowledge graph auto-extracted from the encyclopedia would sit comfortably next to programs and discrete reasoning. They even tried feeding StarCraft’s building tree to the agent. The results were not very relevant. He still likes the direction.

On existential risk he splits the timeline. Near term, he is not concerned, and he hopes he is not wrong. He appreciates that AI safety is becoming a real field at conferences. Long term, he hopes the benefits outweigh the dangers, and he wants enough lead time to redirect if the trade-offs sour. He is more fearful, for now, of other planetary threats, and he has started reading more safety work, a literature he has not contributed to. Maybe there is something to do there. He also does not want the research community, staring at AGI, to lose the line that already helps people who cannot hire an assistant and may not even know the weather. Next for AlphaStar, the obvious verification is other races, so they know they did not accidentally build a Protoss-only architecture. After that he wants the poker layer: a deeper reading of the opponent, and the ability to trick them. Games push techniques, and techniques find new games. That tension, he says, is the job.