GPT-5.5 and DeepSeek V4: Benchmarks, Costs and Compute Limits

The host compares GPT-5.5 and DeepSeek V4 after their releases, drawing on system cards, benchmark results, personal tests and interviews with lab leaders. He says many early GPT-5.5 scores are provider-reported because API access was not yet generally available when he recorded.

After several days of early testing, he expects to use GPT-5.5 as his daily model, narrowly ahead of Opus 4.7. That is a personal preference. On the displayed SWE-bench Pro comparison, GPT-5.5 scores 58.6%, behind Opus 4.7 at 64.3% and Mythos Preview at 77.8%.

On Terminal-Bench 2.0 the displayed GPT-5.5 score is 82.7%, slightly above Mythos Preview’s 82.0%. The host cautions that configurations differ and a forthcoming Pro version was not yet directly comparable. One benchmark does not settle the question of which coding model is best.

On Humanity’s Last Exam, the displayed results favor several competing configurations. The host speculates that OpenAI might be prioritizing efficiency over some general-knowledge performance. He emphasizes performance per token or per dollar, while noting that more inference compute can improve benchmark scores.

On ARC-AGI-2, he reports a stronger score and lower cost for GPT-5.5 than the compared Opus models. His own SimpleBench test puts DeepSeek V4 Pro close to Opus 4.7 at lower cost. These results are specific to their tests and settings.

The host discusses cyber evaluations in OpenAI’s system card. The UK AI Security Institute reports strong performance on a set of narrow tasks, with uncertainty; a small end-to-end evaluation produced different results across models. The host cautions that headline comparisons are difficult and that safeguards matter.

In a clip, Sam Altman questions whether some safety messaging also serves to concentrate access, while acknowledging that some safety concerns are legitimate. This is his interpretation of competitors’ messaging, not evidence that a particular safety claim is false.

The host compares factual accuracy with how often models admit uncertainty. The displayed Artificial Analysis hallucination metric is the proportion of incorrect responses among non-correct responses in that benchmark, not the fraction of all questions. GPT-5.5’s displayed rate is 86%, versus 36% for Opus 4.7 at max; the test and configuration define the comparison.

The host cites separate Mythos results from Anthropic’s system card and notes the difficulty of comparing results across sources. He also mentions an external spreadsheet benchmark and Vending-Bench, where reported business performance and behavior depend on whether the simulation uses multiple agents.

On HealthBench Professional, the displayed length-adjusted scores are 51.8 for GPT-5.5 and 48.1 for GPT-5.4. The host also cites a specialized clinician configuration with a higher score. These are benchmark scores under particular conditions, not proof of clinical safety or a replacement for medical professionals.

OpenAI also tested whether the model could follow constraints on its reasoning text. The host says GPT-5.5 struggled with a lowercase-only constraint and reports OpenAI’s view that lower controllability may help monitoring. Such results do not make the visible reasoning a complete or guaranteed faithful account.

The video mentions a bias evaluation comparing responses to differently introduced users. The host reports a difference in harmful-output rates under that test. Its scope is the particular prompts and measurement, not a general psychological preference attributable to the model.

OpenAI’s system card assesses GPT-5.5 as below its high threshold for recursive self-improvement, despite higher assessments in other capability areas. In a research-debugging evaluation of 41 real internal bugs, the displayed mean rubric scores for GPT-5.4 and 5.5 are close, with uncertainty bars.

The host cites a time-horizon analysis in which success falls as tasks become longer, even under a generous definition that includes partial assistance. He interprets this as evidence of current limits on sustained autonomous research, while recognizing the model can still affect cybersecurity.

An external cybersecurity evaluation found a larger improvement over GPT-5.4 on its selected tasks and lower test cost. Different evaluations can measure different capabilities and yield different rankings; results should be read with their setups.

DeepSeek V4 is an open-weight model, which is not the same as publishing the complete training data and process. The paper describes a one-million-token context window and a mixture-of-experts Pro model with 1.6 trillion total parameters and 49 billion activated per token.

DeepSeek reports that V4 Pro exceeds some earlier models on several reasoning and coding benchmarks while remaining behind selected newer frontier models. The host says its relative cost can be much lower, but actual cost depends on the task, provider, token usage and deployment.

The paper describes attention to long-document training data, including scientific and technical material. DeepSeek also presents Chinese professional tasks spanning several fields and reports favorable comparisons with selected models. The host uses this to argue for testing models in the language and domain where they will actually be used.

DeepSeek says its long-context architecture combines many techniques whose underlying principles are not all fully understood. The host frames that complexity as a tradeoff in the effort to support long contexts efficiently.

On VibeCodeBench, the displayed scores are roughly 71% for Opus 4.7, 70% for GPT-5.5 and 50% for DeepSeek V4, with different cost per test. The benchmark itself notes that even the leader fails a substantial share of workflows.

The host used GPT-5.5 and image tools to create an interactive adventure game, adding video and music from other tools. It took iteration and debugging, and he points out visible defects such as text overflowing a speech bubble. The demo shows a creative workflow, not a fully autonomous production pipeline.

The discussion turns to compute. The host cites reports of capacity constraints at DeepSeek and Anthropic; Greg Brockman says competitors face shortages and also describes scarcity for OpenAI’s own users. These are observations and claims from the recording, not a current capacity report.

The host argues that available compute is being directed toward valuable specialized tasks and that automating repeatable computer work may raise productivity. He distinguishes this from autonomous scientific breakthroughs and notes uncertainty over how employers will distribute productivity gains, including whether they reduce headcount.

He closes by asking how much economic activity consists of repetitive tasks, suggesting that even partial automation could have substantial effects. That is a forecast, not a measured outcome of these releases.