o3 should wait exactly as long as the answer takes

On No Priors, Sarah Guo and Elad Gil sit down with Eric Mitchell and Brandon McKinzie, two of the people behind OpenAI's o3. The model is the latest in the o-series: systems that think carefully before they respond. Mitchell's comparison is ordinary and human. It is easier to be accurate if you think first. Against previous o-series models, o3 is simply better at math problems and factual questions. They will keep training models that are smarter. That, he says, is not the part that is exciting.

The exciting part is tools. You can train a very smart model, Mitchell argues, and still hit a ceiling if it cannot browse for up-to-date information, or write and execute code. A relatively simple Python program will solve some problems that an LLM only solves inefficiently. o3 can research, run data analysis, look at the result, and change course without the user prescribing each step. Give it a high-level request — due diligence on a company, a forecasting model, a summary — and it infers a reasonable sequence of actions. The interface moves up a level.

How is that built differently from a core GPT-style foundation model? Mitchell's short answer is reinforcement learning. Pre-training still predicts the next token on a huge corpus. The o-series replaces that objective with a focused one: solve very difficult tasks, and take as long as you need. Elad summarizes the product change as more inference-time compute plus sequential steps the model infers for itself. Mitchell agrees. The pause to think and the chain of tools are the same training story.

Sarah asks whether the world splits in two: a fast, cheap model for chores, and a slow, expensive one that swallows a legal M&A folder, thinks for a day, and returns a month of team output. McKinzie points at unification. Sam Altman has said it in public. ChatGPT's model switcher is "big" and "crazy." There is a model that might be good at any given job, which does not help if the user cannot tell which one to pick. Putting the decision inside the model is the more intuitive product, even if every lab will try different splits for different users.

From a consumer's point of view, Sarah says, nobody cares about the naming. They want the right answer, with as little wait as the required intelligence allows. McKinzie's ideal is precise uncertainty. If the model already knows, it should just say so. If it takes a day to figure out, it should take a day. Waiting should last exactly as long as that model's intelligence needs for a correct answer. He thinks they are on that path.

Mitchell wonders whether end-user ChatGPT and the developer API still fork. Companies calling the API for a specific task — or running an open-source model with cheap, controlled inference — may need to say: this is an API use case, you cannot think for ten minutes. McKinzie treats that as steerability. A smart enough model should do the right thing once you specify the context. The catch is that deciding what the right thing is may itself require thinking, so you cannot do it perfectly. Pushing more of the choice into the model is still, in his view, a good goal.

Guo goes back to test-time scaling, the graph OpenAI published of thinking time against accuracy, and asks why tools make that slope so much steeper. McKinzie's visual case is concrete. You ask a question about an image. The model says, transparently, that it cannot really see the thing you mean — it almost knows its vision is not good enough. With a tool, it does not give up. It crops, zooms, manipulates the image. Those tokens are productive. The test-time scaling slope for visual reasoning with tools versus without, he says, is "very noticeably different." Without tools, especially on perception, the model knows it cannot see what it needs and "loses its mind."

Code is the same efficiency argument from another angle. An LLM can grind through a valuation model in context, fitting coefficients and self-checking, at the cost of many attempts. Or it can write a small program and know the answer. Tools let you allocate compute where the model has no comparative advantage. McKinzie's first impression of o3 was that longer waits really did mean better results, and that you could watch it do intuitive things instead of wandering off. People, he notes, face the same choice on any task: go get external validation, or sit and think hard in a vacuum. One is usually more efficient, and it is not always the vacuum.

Elad uses Deep Research constantly — an analyst that looks things up, synthesizes, and charts — and asks whether it needed its own RL branch. Mitchell says no separate stove. Browsing is the natural first tool: widely useful for anything that needs current information, and a test of whether RL can teach long-horizon behavior. GPT-4's browsing, a few years back, was hard to make reliable. Two years is ancient history in this field; the conditions finally caught up. RL, McKinzie adds, is an objective you tailor to the user. Can they sit through a 30-minute rollout? Do they want a page, five pages, or a gazillion?

Asked for near-term applications beyond software engineering and analytical work, Mitchell picks coding and research, including the velocity of research inside OpenAI. The models are hitting an inflection where he wants them several times a day, which was not true before. Navigating OpenAI's internal codebase used to leave them a little behind; now they spend time on the question and come back with hours saved. Elad calls it the bootstrap toward the next model. Mitchell says research is not an ivory tower. Hardware, training, evaluation — each piece can be turned into a task you optimize.

Elad is impatient with the interface. He hates typing. He does not understand why the model is not already on his computer all day, watching, ready to be asked when he hits a wall — opted out by default, he adds, because that will creep some people out. Mitchell is excited about computer use and still will not hand over the keys. The costs are asymmetric: time saved on one side, a crazy email to your boss or a deleted hard drive on the other. The set of things he would trust grows every day. For now they keep the affordances in a sandbox on purpose.

Is there a framework for which tasks yield to more intelligence, test-time compute, and tools? Mitchell offers two axes rather than one map. First, how much environmental uncertainty you have to wrestle with. "Who was the first president of the United States" is zero: remember and say it. Writing code that actually solves a problem means executing, testing, and waiting on an experiment whose result the model cannot fully memorize. The more you must run the experiment in the world, the harder the task. Second, how simulatable it is. Anything bottlenecked by physical time is harder than something you can replay. Coding agents are not a coincidence. Robotics is slow. He used to work on robots. A lot of it, he says, is frustrating.

McKinzie adds a third constraint: the real world has a frame rate. The ball is coming now. Gravity will not wait for a two-minute chain of thought. Small brains are already good at that — frogs, ants — which makes baseline compute for responsive tasks an open question. Elad asks whether robotics foundation models merge into the same general weights, the way Codex got absorbed into general models after GitHub Copilot. McKinzie sees no reason the RL model could not be the same one, even if generating an hour-long movie natively, rather than via a tool call, would strain a single set of weights. o3 poking around in a picture already looks like an agent exploring an environment.

Then the clock. Ask the models what time a pictured clock shows and they are very fond of 10:10. McKinzie's figure is that something like over 90 percent of clocks on the internet are set there, because the hands make a happy face. Human vision was built by navigating the world and avoiding predators. Model vision was built another way. If you can close the loop — let the system reduce uncertainty by acting, not only by thinking — he expects a lot of interesting behavior to show up.

Long-running software work is not a stream of commits. You talk to other people about the project. Sarah asks whether you can simulate those humans well enough for training. Mitchell's spicy take is that o3 is already simulating one person with a browser. Train two of them together. There is a literature on multi-agent RL; if you want collaboration with a bunch of people, a not-too-bad starting point is collaboration with other models. "Someone should do that," he jokes. Sarah says the work is already going on, which is why the take is spicy.

McKinzie wants a more interactive training loop: intervene mid-chain-of-thought, tell the model to cut it out, and have it learn on the fly, not only at test time. Humans are high-uncertainty environment and a fixed clock cycle at once. Literal training against real people is hard to scale. His accounting is blunt. "We are a super expensive tool call." While the meat bag thinks slowly, the model could have browsed a hundred papers. The human part still matters. In any research project, Mitchell says, his interactions with Brandon are the hardest part; writing the code is easy. Elad reaches for self-driving: the classically hard problem was predicting the human, the child, and the dog, not the road. Hosting a podcast, he adds, has a similar hard part. Sarah says that part is her favorite.

Will o-series capabilities get spikier, because RL depends on research teams inventing environments and evals, unlike pre-training's grind across existing data? McKinzie calls that a reasonable hypothesis and then complicates it. Altman tweeted a creative-writing sample that probably updated people's sense of what you can train a model to do well. Organizations must prioritize, and as models improve, the surface area of possible work grows faster than any lab can carve out niches, so some spikiness is structural. The version of the story he rejects is "these models will only be good at math and code." That, he thinks, is the wrong intuition.

Mitchell describes the split inside labs: one group defines data distributions and throws data at them; another looks for algorithmic changes that lift all boats. OpenAI does both. Data will always be thicker in some places than others. The hope is that RL gains are not glued to the exact subset they trained on. Elad's magic-wand question — what perfect data would you want this week? — gets a dodge from McKinzie that he means seriously: uncontaminated evals. Measuring a generally capable agent is as hard as hiring one. Recent models, including o3 and o4-mini, have solved several reliable evals, so the ruler is gone. On the training side, Mitchell would not turn down multi-turn user interactions against a million-line codebase: a two-week feature, multiple pull requests, dense supervision, at the next frontier of what the model cannot already do.

What do they do with reasoning models that other people do not do enough? Mitchell: send the same prompt many, many times, and get an intuition for the distribution. Twitter comparisons that drop one prompt into several models and declare a winner drive him mad. o3 can chain tool calls into a magic moment; the same prompt can also skip the magic, or simply do less work. Peak performance is impressive. A single sample is not the model. Sarah wants a product button that runs the prompt a hundred times, ranks or synthesizes, and returns the top one or two, even if it is expensive. She is, she says, a price-insensitive AI user. McKinzie calls it a great suggestion. The reason they do not ship it yet is that it is expensive.

His own habit is to throw work at the edge of what he expects the model to handle. Before a programming task he often just asks o3 to try. Sometimes he feels like a disappointed father. Sometimes it does the thing and saves hours. He treats the model as a background queue: shoot off tasks, keep the wins, lose nothing on the misses. Elad does the same to feel useful when it fails, and worse when it works. That is the user manual hiding in the research conversation: run it more than once, let it use tools, wait when waiting is the signal, and put the constraints in the prompt.