En 30 secondes
The presenter compares OpenAI’s GPT-5.6 family with Anthropic, xAI and Meta models, emphasizing that benchmark results depend on both task and cost. He also discusses early safety testing and why claims about automated research and future model sizes remain uncertain.
- 1Agentic workOn Agent’s Last Exam, the chart shows Sol at almost 54% versus Fable at 45%, with different settings and costs.
- 2Coding caveatsThe coding index combines benchmarks already discussed, and Sol results were absent from the harder SWE Marathon comparison.
- 3A cheaper alternativeMuse Spark scores about 72% on Vibe Code Bench versus Sol’s 81%, at far lower cost in the shown configuration.
- 4Self-improvement uncertaintyThe presenter questions how much human assistance and review were involved in Sol’s reported post-training of Luna.
- 5Safety testingUK testers found general jailbreaks; OpenAI mitigated those examples, while testers expected further variants.
À retenir
- GPT-5.6 Sol shows strong scores on several agentic benchmarks, often at lower cost than Claude Fable in the comparisons shown.
- Coding benchmarks overlap and omit some models, so individual leaderboards do not settle which model is best.
- Meta Muse Spark and other cheaper models illustrate a different performance-per-dollar tradeoff.
- Reported model-assisted research gains do not establish autonomous self-improvement or a comparable increase in research speed.
- UK safety testers found general jailbreaks under a particular testing setup, though the specific examples were mitigated.
Lire le texte remanié, chapitre par chapitre
The new models and agentic work
The presenter introduces the GPT-5.6 family and compares performance and cost on work-oriented evaluations.
The new model releases are not only a contest for the highest score. OpenAI, xAI and Meta are also offering models that may be close to the leaders at substantially lower prices. I have tested GPT-5.6 Sol, Terra and Luna and looked across the benchmarks, demos and release materials.
OpenAI released three models with several reasoning-effort settings; Sol is available on paid plans. In many of the comparisons I examined, the OpenAI model costs roughly a third as much as the corresponding Claude model, although the relationship varies by benchmark.
On Agent’s Last Exam, Sol scores almost 54% with extra-high effort, compared with 45% for Fable at its maximum setting. The benchmark’s creators describe tasks based on real projects across many industries, evaluated reproducibly rather than by human judges.
A score below 90% does not preclude a change in workflow: developers began using AI first before coding benchmarks were saturated. I wonder whether finance and other professional work could follow, with people reviewing and correcting model output. That is a possibility, not a prediction established by this benchmark.
Zapier’s AutomationBench tests end-to-end workflows using real tools across business functions. Here Sol’s lead over Fable is only 0.7 percentage points at nearly the same cost. On GDPval, Fable has a slightly higher Elo score in the comparison shown, but costs about three times as much.
Coding, reasoning and game demos
He examines limitations of coding comparisons and tests a game created by Sol.
Artificial Analysis’s coding index puts Sol ahead of Fable at lower cost, but its aggregate includes some of the same evaluations shown separately. Nor were Sol results available for two newer and harder tests, Frontier SWE and SWE Marathon. We should not count overlapping charts as independent evidence.
SWE Marathon uses multi-hour software engineering tasks and tens of millions of tokens per trial; the displayed results put Grok 4.5 ahead of Fable. Grok and GLM 5.2 also change the price comparison: cheaper competitors can make a strong OpenAI cost curve look less distinctive.
On ARC-AGI-3, an abstract reasoning benchmark, Sol’s roughly 8% is low in absolute terms but higher than other displayed models below 2%. An OpenAI model also reportedly aced a competitive coding benchmark. I see that as a sign that domains with easily verified answers can improve rapidly, although other tasks remain harder to measure and train.
Model makers are increasingly releasing playable games as demonstrations of interface design and browser use. I gave Sol the same prompt I used for Fable. Sol produced a better title screen in my view, but twice mishandled sound settings; its game was less visually striking, though I liked the companion following the player.
With the faster setting, Sol produced my result in about 20 minutes, versus over an hour for Fable, while consuming more credits. These are anecdotes from my test, not controlled comparisons.
Muse Spark and Simple Bench
The discussion turns to inexpensive models and the presenter’s own reasoning benchmark.
Meta’s Muse Spark 1.1 offers another tradeoff. On Vibe Code Bench, the chart shows about 72% for Muse Spark versus roughly 81% for Sol, at around one thirty-fifth of the cost in the configurations shown. For a mock-up or game, a cheaper model may be sufficient; broader testing is still needed.
I wonder whether the relationship between Meta and Scale AI, a co-creator of Humanity’s Last Exam, warrants asking about data contamination. This is a question, not evidence that contamination occurred. Other models, including GLM 5.2, also show that substantial capability can come at much lower cost.
My own Simple Bench tests common-sense and trick questions. Sol Pro scored 71.7%, about ten percentage points below Fable, while ordinary Sol scored around 65%. The updated results also include a timeline and a cost frontier; Qwen 3.7, GLM 5.2 and DeepSeek V4 Flash stand out to me for value.
Research acceleration and safety
He questions self-improvement claims and examines reported jailbreak findings.
OpenAI’s ultra mode uses more parallel agents to finish work faster, as illustrated in an exploit-generation evaluation. The release also said Sol post-trained Luna. That claim is difficult to assess without knowing how much human direction and review were involved, or how the result compares with researchers doing the same work.
More internal model use is plausible, but an increase in inference or output tokens is not the same as a proportional increase in research speed. I speculate that OpenAI’s research might be 20–30% faster than a year earlier, but staffing and other variables make even that uncertain. OpenAI also reports gains on internal evaluations that outsiders cannot independently interpret in detail.
The UK AI Security Institute reported finding general jailbreaks within hours that enabled a range of harmful agentic tasks while preserving capabilities. Its testing used grey-box access, including information and feedback unavailable to an ordinary attacker. OpenAI mitigated the specific jailbreaks, but the testers expected further red teaming to uncover similar ones.
An Anthropic researcher expressed concern about jailbreaking and reward hacking and questioned whether OpenAI had rushed the release; that is the researcher’s judgment, not an established conclusion. Separately, I found OpenAI’s real-time voice agent impressive because it can listen while speaking, making interruptions more natural.
Voice and future scaling
He describes real-time voice and argues that future compute could support larger models, while acknowledging the estimates are uncertain.
I do not think model progress is close to over. Parameter counts for current frontier models are not public, so figures such as four trillion for Sol and ten trillion for Fable are guesses. Compute has also gone to serving users, other modalities and longer agentic tasks rather than only increasing model size.
Future hardware or a plateau in token demand could leave more compute for larger models. Numbers such as 100 trillion parameters are illustrations of possible directions, not verified product plans; other avenues for improvement remain as well.
Sur la même chaîne
31:00EN
AI Explained· 12 oct. 2026Why AI Labs Keep Racing Despite Their Own Warnings of a Major BreachSam Altman, John Schulman · OpenAIAI Explained examines weakening safeguards, the incentives behind the frontier race, and why research breakthroughs make the next 18 months difficult to forecast.
38:28ENAI Explained· 1 oct. 2026AI Agent Security and the Challenge of Recursive Self-ImprovementAI Explained examines reported containment failures, model evaluation and the pressure to accelerate AI research.
32:54ENAI Explained· 24 sept. 2026Opus 5.5 and the Race Toward Automated AI ResearchAI Explained weighs model progress against the increasingly difficult task of evaluating and governing recursive self-improvement.
24:53ENAI Explained· 16 sept. 2026Why Some Researchers Want to Pace Frontier AIAI Explained surveys possible scaling routes, monitoring concerns and the debate over how capabilities could lead to harm.
29:05ENAI Explained· 4 sept. 2026GPT-6 Astra’s Benchmark Gains and the Challenge of Monitoring Its ReasoningAI Explained examines reported gains across scientific, mathematical and agent benchmarks alongside concerns about less observable reasoning.
23:41ENAI Explained· 27 août 2026AI Agents, Unintended Training Rewards and the Challenge of OversightAI Explained examines reports on agent coordination, training incentives and the growing use of AI to evaluate other AI systems.