How GDPx Tests Frontier AI: Inside SemiAnalysis’s New Benchmark

Jordan Nanos speaks with Max Kan and Andrew Megalaa about launching GDPx, SemiAnalysis’s benchmark for frontier AI. The team says it did not see the final leaderboard until hours before publication, after late task revisions and long model runs.

The partners calibrated difficulty on a small set of models, while the infrastructure team ran the larger evaluation. The guests say a task and its grading could take as long as a day, and that the launch deadline was driven partly by anticipated model releases.

The team’s starting point is dissatisfaction with popular benchmarks. The guests argue that some tasks appear hard because they are unfair or poorly specified, and that identifying this requires inspecting runs and determining why a model failed.

They distinguish difficult from unfair with a coding example: if two solutions fix a bug but the verifier accepts only one, a correct alternative receives a false failure. The problem is a mismatch between the prompt and grader, not necessarily a lack of model capability.

The guests say evaluations can contain many such issues and that running repeated attempts and reviewing transcripts is labor-intensive. Their criticisms of named benchmarks are their own assessments, not independently verified defect rates in this conversation.

SemiAnalysis wanted a repeatable way to compare new models, but did not assume it could author hundreds of high-quality tasks alone. The team partnered with 14 specialized data providers, reviewed their submissions and rejected some prospective providers during quality assurance.

The benchmark focuses on domains the team believes can absorb expensive frontier intelligence and produce high returns. They agree that many everyday tasks may not need the most capable model, but expect areas such as chip design, biology and cybersecurity to continue demanding more.

Nanos lists the eleven domains: recursive self-improvement, chip design, cybersecurity, biology, finance, legal, quantitative trading, hardware design, materials science, alignment and software engineering.

The team says it chose domains through brainstorming and discussions with people at labs, looking for both economic relevance and verifiability. It solicited sample tasks, checked quality and used judgments about representativeness when choosing partners.

Some partners are established data companies; others are individual specialists. The guests highlight materials-science tasks based on researchers’ own work as an example of specialized knowledge shaping the evaluation.

One materials-science contributor reportedly found that a model solved problems he initially expected to be too difficult once it received the right context. The team says the contributor had to develop a harder set of tasks.

Megalaa describes changing his own expectations as coding models improved. He says he moved from writing much of the code himself toward higher-level work, while still believing human judgment can improve on a model working alone.

The guests think similar moments may occur in other technical fields, but individual results do not establish that a model is generally better than an expert. They discuss a materials-science subtask that they say exceeded a published reference and the difficulty of defining comparable human baselines.

Nanos shows reported per-task costs for the materials-science evaluation. The guests say model solutions can be creative or unexpected, which makes careful verification important: an unfamiliar approach may be correct, but it may also fail.

The discussion turns to the leaderboard. The team presents both an overall view and a filtered view excluding biology, cybersecurity and alignment, where safety classifiers can prevent some models from attempting tasks.

They say a refusal affects the observed score even when the underlying model might have relevant capability. The team chose to headline the all-domain view, while arguing that filters help users interpret results for their own context.

The speakers discuss cybersecurity results and differences in access to models and safety settings. Their view is that available access can materially affect practical performance; a result should be read in the context of the configuration actually tested.

GDPx is deliberately designed around difficult frontier work rather than a representative sample of everyday jobs. The team’s economic thesis is that high-value domains may remain willing to pay for more capability even if cheaper systems handle many routine tasks.

The guests say their finance and legal results surprised them. Some proposed tasks were unrealistic or unfair and were discarded; financial professionals helped evaluate whether the remaining examples resembled real work.

One finance partner built an environment from a real private-equity transaction involving a family-owned snack business. The team describes about 4,000 deal-room files and real tasks drawn from the work of the parties involved.

The environment can reproduce the state of the data room when a particular assignment was made. In one failure mode, a model missed a single email changing the relevant numbers and consequently produced an incorrect deliverable.

Nanos asks about grading. For finance, the benchmark combines detailed component checks with a binary final score: a workbook with many wrong numbers may not be acceptable as a completed professional deliverable.

The same finance tasks can also be displayed with partial scores, which reveal progress hidden by a zero on the binary measure. The guests say a large gap between partial and binary scores can also expose a task whose final requirement is unfair or overly narrow.

Other domains use continuous grading when it better fits the task. For chip design, the team gives an optimization example where a result can receive credit according to its improvement between a baseline and a target.

The guests argue that finance and legal work can still justify spending on stronger models because of business margins and competitive situations. This is a forecast about demand, not a conclusion that follows directly from the benchmark score.

Asked what GDPx implies for employment, the team stresses that it does not sample many jobs people worry about, such as customer service. It therefore cannot by itself predict economy-wide job losses.

The speakers discuss a possible period in which human judgment and model capability complement each other. They also consider a future with much smaller teams, but say the scale and timing of any displacement remain uncertain.

They worry that junior workers and interns may lose some well-specified tasks that previously provided experience. Another possibility, they suggest, is that people use agents to attempt more ambitious projects rather than simply reproducing yesterday’s work.

The guests disagree on how widely people will embrace that shift. One is optimistic about people using tools to raise their ambitions; another thinks many workers primarily want to complete their hours and go home. Both acknowledge that the outcome is unresolved.

Returning to the leaderboard, Megalaa says his experience with one model broadly matched its performance, while another was especially strong on some materials and engineering tasks. The panel emphasizes that different models have different strengths.

They discuss Gemini 4 Carbon and argue over which filtered comparison is most useful. Their ranking of laboratories includes judgments and speculation about unreleased systems; the benchmark measures only the versions and settings actually evaluated.

Nanos asks why the team released one task and rollout per partner while keeping other tasks private. More disclosure allows outside experts to audit quality, but public tasks are also easier to target when training or tuning a model.

The team hopes examples, detailed methodology and descriptions of quality assurance provide some visibility without exposing every task. Private tasks reduce one route to benchmark contamination, but they also limit independent inspection.

For future versions, the team plans to evaluate new releases, add partners and tasks, and revisit domains as model capabilities change. It also hopes to test more reasoning settings and keep the dashboard current.

The guests describe GDPx as a first iteration and invite further participation in evaluation and maintenance. Nanos closes by thanking the team for explaining its methodology and initial results.