Q* and Reasoning Research: A Speculative 2023 Analysis

In November 2023, AI Explained speculates about reports of an OpenAI project called Q*. The presenter explicitly says the hypothesis may be wrong; public papers and reporting discussed here do not establish the project’s actual design.

He notes a denial concerning one reported explanation for Sam Altman’s departure and return, and cautions against a misleading interpretation of a circulating clip. The video was made during a rapidly developing news story.

The presenter connects reported research teams and public references to the paper Let’s Verify Step by Step. Shared names, dates and interests are clues for his hypothesis, not confirmation that this paper describes Q*.

He also considers reports about using more computation at test time. In general, inference-time compute can be spent generating, searching or evaluating candidate answers; the specific approach used in any undisclosed project remains unknown.

A 2021 arithmetic reasoning paper introduced GSM8K and studied generating many solutions and using a verifier to rank them. The presenter describes this as spending compute when answering rather than only increasing training compute.

He reports that verification improved performance in the paper’s experiments and compares it with larger models. These are results for particular models and benchmarks, not a universal rule that checking is always easier or better than training.

The discussion cites research on trading training computation against test-time computation in games, and a researcher’s interest in more general reasoning. Game-specific evidence does not by itself show the same gains across open-ended tasks.

The presenter wonders whether a costly inference run could reveal more capable behavior. He offers this as speculation about an undisclosed event; the video supplies no confirmation that such a run happened.

He then explains process supervision: a reward model is trained to evaluate intermediate steps, rather than only the final answer. The aim is to identify flawed reasoning even when a final answer happens to be correct.

Selecting among many candidate solutions with a process verifier can improve results on tested problems. A verifier can still make mistakes, and scaling a curve or extrapolating to near-perfect performance is not guaranteed.

He combines process reward models and inference-time search into his theory about the reported project. He also suggests possible links to the timing of safety work, but these connections are conjectural.

The video shows public remarks from researcher Łukasz Kaiser about step-by-step reasoning, checking steps and possible future multimodal systems. These statements describe research directions, not confirmation of Q*’s internal implementation.

The presenter speculates that improved use of existing data could matter as much as generating more synthetic data. The relative importance of these methods in the reported project is not known from the evidence shown.

He admits his theory does not explain the name Q* and discusses possible connections to optimal action values or Q-learning. A resemblance in notation or terminology is not evidence that Q-learning was used.

A paper’s suggestion that reinforcement learning could train a generator motivates another possible connection. The presenter asks whether such a follow-up happened but does not establish that it did.

He also considers a separate STaR self-training paper and a speculative connection to the “star” in the name. The shared word or symbol is suggestive only; it does not establish a relationship between projects.

The video contrasts the clear win-or-loss feedback available in games with the difficulty of evaluating open-ended language tasks. Reliable feedback and generalization are important unresolved challenges for self-improvement.

A clip of Ilya Sutskever raises concerns that optimization could produce unexpected behavior. The presenter frames this as a safety question, not proof of a particular incident or capability.

He closes the hypothesis by acknowledging that it could be entirely wrong and that improvements on narrow mathematical tasks would not by themselves establish general intelligence. The video ends with a short music-generation demonstration.