AlphaGo Thought 30 Seconds a Move. o1 Made “Think Longer” General

Sonya Huang and Pat Grady host OpenAI researchers Noam Brown, Ilge Akkaya, and Hunter Lightman, three of the people behind Project Strawberry, the effort that became o1. Huang calls o1 OpenAI’s first major foray into general inference-time compute. The conversation, recorded less than three weeks after the September 12, 2024 release, is about reasoning, chain of thought, and a new scaling law — and about how much conviction anyone actually had that this would work.

Brown says the team believed something in this direction was promising, but the path was never clear. o1 was not an overnight result; it sat on years of research, much of which did not pan out. What mattered was that OpenAI leadership kept investing through the setbacks. Lightman, who has spent most of his two and a half years at the company teaching language models to do math, had less conviction at the start. Research, he says, has an ebb and flow. His belief arrived later, when the outputs started solving problems in a different way. Akkaya has been at OpenAI for five and a half years and admits she has been wrong several times about the path to AGI. She joined the robotics team first, convinced embodied AI was the way forward, then hit roadblocks. ChatGPT was the obvious paradigm shift; o1 looks to her like the next path worth pushing.

Asked how o1 actually works, Akkaya cannot say much, but she gives the core claim. The o1 series is trained with reinforcement learning to think — “you could call it reasoning” — and that is fundamentally different from what people are used to with LLMs. The team has already seen the skill generalize across many reasoning domains. The rest of the hour is an attempt to explain what that difference is, and what it is not.

Brown’s working definition starts with human System 1 and System 2: automatic instinct versus slower, more process-driven thought. Some tasks do not benefit from more thinking time at all. If you ask for the capital of Bhutan, he says, you can think for two years and your accuracy will not improve. Grady immediately asks what the capital is. Brown does not know. (It is Thimphu.) Other tasks sit at the opposite pole. A Sudoku puzzle is the classic case: generating the right grid is hard, checking one is easy, and with enough time you could in theory enumerate your way to the answer.

He is not claiming a canonical definition — everyone in AI has their own — but he views reasoning as the problems where considering more options and thinking longer helps. He calls that a generator-verifier gap: it is really hard to generate a correct solution and much easier to recognize one. All problems, he says, exist on a spectrum from easy-to-verify relative to generation, like Sudoku, to as hard to verify as to generate, like naming a capital. That spectrum is also his intuition for the scoreboard. STEM problems tend to be hard reasoning problems in this sense, which is why o1 shows the biggest lift there. The research paper’s GPQA results, questions typically given to PhD students, already have the model outperforming a lot of human PhDs on that test.

Brown spent a decade in game AI, including poker, and that history shapes how he reads o1. Looking back at breakthroughs, AlphaGo is the classic example — and one of the most noticeable facts about the bot, underappreciated at the time, was that it thought for a long time before acting. It would take about 30 seconds to make a move. If you forced it to act instantly, it was not better than top humans; it was noticeably worse. A large part of its strength was that extra thinking time.

The catch is what it did with those 30 seconds: Monte Carlo tree search, a form of reasoning that worked well for Go and, Brown notes, does not work in a game like poker. The pattern of earlier AI was twisted in a particular way. The neural net, the System 1 piece, was already general. The methods for thinking longer were still domain-specific. What he finds cool about o1 is that the way it thinks longer is quite general and can be used across many domains, which they are now watching users discover.

Lightman adds a complementary point that is not new, but becomes newly important. Language models already have a text interface that can be adapted to all kinds of problems. The exciting part of this moment, he says, is that the team thinks it has a way to do reinforcement learning on that general interface, and then to see where that leads. Asked whether the same reasoning method holds across the generator-verifier spectrum, he points to OpenAI’s iterative-deployment strategy: put the research in people’s hands, watch Twitter show what the model can and cannot do, and bring those limits back into the lab.

The surprises have not all been benchmarks. Akkaya is most excited by MDs and researchers using the model as a brainstorming partner. People who have spent years in cancer research run gene-discovery and gene-therapy ideas past o1 and come back with novel paths to pursue. The model clearly cannot do the research itself, she says, but it can be a very nice collaborator in domains her team does not work on.

Huang reminds Brown of his tweet that deep RL is out of the trough of disillusionment. He walks through the decade. From DeepMind’s Atari results through 2015 to 2019, deep RL was the hot thing; he was in a PhD program and remembers the mood. What got overlooked was the power of training on huge amounts of data in the GPT style. Even AlphaGo, the crowning RL achievement, started with a large process of learning from human games. The RL step came later, as did a separate reasoning step. Then the field treated human data as a kind of impurity and leaned into learning from scratch. AlphaZero, which did better than AlphaGo without that human data, reinforced the preference. In that climate the GPT paradigm flew under the radar, except at OpenAI, which saw early results and doubled down.

When GPT-3 and other large language models succeeded without deep RL, a lot of people switched away or lost faith. What o1 shows, Brown says, is that there is still a place for deep RL, and that it can be powerful when combined with those other elements. Grady asks whether this is one of the first times deep RL has been used in a much more general, unbounded setting rather than well-defined gameplay. Brown agrees that the highlight RL results were cool but narrow. There were useful and even general RL results, “but there wasn’t anything comparable to something like GPT-4 in its impact.” He thinks that kind of impact is what this new paradigm can deliver.

Lightman’s aha came on the o1 trajectory after a training run with a pile of fixes. The model scored higher on math evals than any of their other attempts, including bespoke systems built for the problem. Then they read the chains of thought. The traces had a different character. When the model got stuck it would say, wait, this is wrong, let me take a step back and figure out the right path. They called it backtracking. Lightman had wanted to see that for a long time and had half-believed he never would, because an autoregressive model only predicts the next token. The score plus the backtracking trajectory was the moment he updated. Something was coming together that he had not thought would come together.

Brown’s version is plainer. He joined with the observation that ChatGPT answers too fast, with no thinking, while game AI had already shown that thinking longer buys huge gains. The team discussed many ways to teach self-correction and backtracking. Eventually they tried the dumb baseline: just have the AI think for longer. Once it could, abilities like backtracking and self-correction developed almost emergently — the behaviors they had been wondering how to enable. Seeing them come from a clean, scalable approach was his big moment. Lightman, sitting beside him, says this understates how strong that conviction was. In early one-on-ones Brown kept talking about test-time compute; at several points in the project he would ask why they did not just let the model think longer. They would, it would get better, and he would look at them as if to ask why they had not already done it.

Huang asks whether o1 has had its AlphaGo Move 37 — a play that surprises everyone and is actually better than what a top human would find. Lightman reaches for the IOI programming competition they entered. On one problem the model was insistent on a weird method; colleagues who live in competitive programming tried to figure out why. He does not think it was a stroke of genius. It felt more like the model did not know the standard solution, so it banged its head until it found something else. It did solve the problem. Across the contest, humans crushed one task the model barely got off the ground, while the model took half credit on a problem humans did poorly. It simply does not approach these problems the way a human does. Akkaya has watched it solve geometry by narrating a visualization — put the points on a sphere, then reason about a probability — and been startled because that is how a person would think on paper. Lightman notes that the chains of thought are human-interpretable, which he considers one of the cool things about the release: you can poke around at how the model is thinking.

The STEM lift, Brown repeats, is the generator-verifier gap showing up in the evals. Relatedly, the paper reports that o1 passes OpenAI’s research-engineer interview at a pretty high rate. Huang asks whether the company will eventually hire o1 instead of humans. Lightman does not think they are at that level yet — “maybe the interviews need to be better.” It already feels like a better coding partner than previous models, and it has authored a couple of PRs in their repo, so in some ways it is acting like a software engineer. The rollouts he sees think for a few minutes; the coding he does himself takes longer than that. Scale the trend line and more of those tasks may fall. Brown’s joke about internal AGI: you will know when they take down all the job listings, and either the company is doing really well or really poorly.

Will reasoning skill extend to the humanities as inference time scales? Brown sees a gap between the model’s raw intelligence and how useful it is. In some ways it is already very useful; it could be a lot more useful in a lot more ways, and that will take iteration. On how much of that gap belongs in the model versus the ecosystem on top of the API, he says he was skeptical that OpenAI was truly AGI-focused until day one, when Sam laid out short- and long-term priorities at an all-hands. There is no single application that is the priority other than getting to AGI. Lightman’s definition is the proportion of economically valuable jobs the systems can do, a number he expects to ramp over the coming years. You will feel it when you feel it, and people will keep moving the goalposts until one day you are working alongside AI coworkers doing large parts of today’s jobs. He is already bettered by o1 at math and not at software engineering. If his whole job were AIME problems, he would be out of work. There is still a mismatch, and still a job to be done.

Chain of thought is hidden in the product. OpenAI’s blog said that was partly for competitive reasons, and Grady asks whether that was a fight inside the company. Brown says it was not contentious, for the same reason you do not share frontier model weights: there are risks to exposing the thinking process. Akkaya’s lay explanation is an integral. Most people need paper and pencil, a sequence of simplifications, and a final answer that might just be the number one. How you get there is the chain of thought. o1’s traces are human-readable, which is what lets the team watch the model work.

Huang calls the inference-time scaling chart the most important figure in the paper, a result on the order of pre-training scaling laws. Brown agrees it is profound. Before launch he wondered whether people would even notice; it is a subtle point, and he was surprised and impressed that so many did. The background anxiety is that AI might be hitting a wall because pre-training is so expensive and data may run short. The major takeaway from o1, especially o1-preview, is not what the model can do today but what it means for the future: a different, so-far pretty untapped dimension for scaling, which implies the ceiling is a lot higher than a lot of people have appreciated.

What happens if you let it think for hours, months, or years? They have not had o1 for years, Lightman says, so they have not been able to try. Grady jokes about a background job still working on world peace. Lightman cites Asimov’s “The Last Question,” in which a computer asked to reverse entropy keeps answering that there is not yet enough information, ten years later, then a hundred, then ten thousand. The viral “120 IQ” number, he warns, is 120 on some test someone gave it, and below 40 on things like creative writing. GPQA outperformance does not mean the model is smarter than a PhD in every way. Benchmarks are a proxy for human intelligence when humans take them and mean something different when an AI does.

What he hopes to see is longer thought on the tasks it is already good at. A college math professor tweeted that o1 had found a proof humans had solved but no AI had; to Lightman that sits on the cusp of a useful research assistant, the kind that can knock down small lemmas in real math. Ground yourself there, he says, and you can play the game forward into software engineering rather than chasing infinite IQ on tasks it currently fails. The bottleneck to scaling test-time compute, Brown adds, looks a lot like the years between GPT-2, GPT-3, and GPT-4: a simple idea that still takes enormous engineering to scale. Lightman says academics who join OpenAI are often surprised how much of the work is engineering — algorithms never invented before, on systems no one has used, at a scale no one has tried. Akkaya names a third constraint: knowing what to test. Standard evals are not the whole map of where extra test-time compute would pay off.

The biggest misunderstanding, Brown says, was the leaked name. When the “how many r’s in strawberry” question went viral, the team was worried there had been an internal leak. As far as they know there was not; it was a coincidence. Lightman’s version is more ordinary: someone needed a code name, someone in the room was eating a box of strawberries, and that was the end of it. On reception, Brown was not sure whether people would be disappointed that o1 was not better at everything or impressed by the crazy math. What they were trying to communicate was that the release was about where the work is headed. He was happy that message landed. Fair criticism, from Lightman: it is absolutely not better at everything. It is a funky model to play with, and people on the internet are still finding new ways to prompt it. He compares the moment to a year earlier, when language-model programs on GPT-4 unlocked smarter software-engineering tools. Something similar may get built on top of o1.

They have not talked yet about o1-mini. Huang has heard excitement about small models that keep the reasoning and drop some of the world knowledge that deep nets store inefficiently. Akkaya is blunt: if a model is fast, it is universally useful. The two sizes serve different purposes — cheaper and faster versus heavier and slower — and she is glad they found a good tradeoff. Lightman likes that framing because progress is how far you can move times how fast you can iterate. o1-mini lets the research team, and hopefully the broader ecosystem, iterate faster. That alone makes it a useful artifact.

How should founders choose between GPT-4 and o1? Lightman would love it if they figured that out for OpenAI. One motivation for releasing o1-preview, Brown says, was to see what people use it for, what they like, and how to improve it; there was even an internal question about whether a preview was worth putting out. Asked what people most underappreciate, he jokes that they are getting a little better at naming things — they did not call it GPT-4.5 thinking mode. For o2, o3, whatever comes next, Akkaya is not out of ideas. She is most excited about feedback from domains the researchers are biased away from, use cases beyond their imagination. Lightman is most excited that they have a trend line, posted in the blog, and that they get to watch it extend.