Theo Reviews DeepSWE and the Limits of Coding Benchmarks

It is getting harder to tell which models are useful for real development work. I am skeptical of some existing coding leaderboards because their ordering does not match my experience, and I am concerned about task quality, grading and exposure of solutions. A benchmark score is evidence about its own tasks and setup, not an automatic ranking for every workflow.

DataCurve’s DeepSWE is an attempt to evaluate longer, more realistic work in existing repositories with new tasks and behavioral verification. I am enthusiastic about it, and I need to disclose that I am an investor in DataCurve. That financial relationship is relevant when evaluating my praise and my interpretation of the results.

In an update recorded a day later, I discuss preliminary Claude Opus 4.8 runs. DataCurve first tried Claude Code and then mini-swe-agent, with the latter yielding about 63% in the reported run. These were preliminary, used different harnesses, and should not be treated as a finalized or controlled causal explanation of the difference.

The leaderboard visible in the original recording reports GPT-5.5 at 70%, GPT-5.4 at 56%, Claude Opus 4.7 at 54% and Claude Sonnet 4.6 at 32%, with uncertainty bars. Those are results for the benchmark and configurations shown at that time. They do not establish a universal ordering across all tasks or later model versions.

DataCurve sells coding data to AI labs. I say I introduced the benchmark to OpenAI and reject the suggestion that OpenAI paid for it. The company’s incentives and my investment make transparency about methodology and independent replication especially important, regardless of whether the results match my experience.

DataCurve reports an audit of SWE-bench Pro in which its analyzer and the original verifier sometimes disagreed. The company reports estimated false positives and false negatives, as well as cases involving access to repository history. These are DataCurve’s audit findings and depend on its own labeling and review methods; disagreement alone does not prove which evaluator is correct.

I find the new ranking closer to my own experience with models, particularly the gap between models on repository navigation and implementation. That agreement is also a possible source of confirmation bias. A larger spread between scores is not, by itself, proof that a benchmark is more accurate.

DeepSWE says its tasks were written from scratch rather than taken from existing commits or pull requests. It reports 91 active repositories across five languages, with many tasks in TypeScript, Go and Python. New tasks can reduce direct exposure to existing solutions, although no benchmark can assume permanent immunity to future contamination.

The team says prompts are shorter than in SWE-bench Pro while the reference solutions involve substantially more code. Its hand-written verifiers aim to test observable behavior instead of requiring one particular implementation. That design is promising, but its success still depends on how well the tests cover legitimate solutions and edge cases.

One difference I emphasize is whether an agent can write and run its own tests. In the SWE-bench prompt I inspect, the task says test-file changes have already been handled and asks for a minimal change to non-test files, while also suggesting a reproduction script. I think that wording can discourage adding project tests; it should not be simplified into a blanket ban on all verification.

The displayed analysis reports that Claude Opus 4.7 wrote and ran new project tests in 28% of the SWE-bench Pro runs versus 83% on DeepSWE, while GPT-5.4 went from 18% to 85%. These are observed rates under different benchmark conditions, not a controlled estimate of the effect of one sentence alone.

The DeepSWE examples focus on intended behavior and ask the agent to find the relevant implementation. The benchmark compares these with more prescriptive issue-based tasks that can include reproduction steps or interface details. I prefer the former style for evaluating exploration, although a clear task still needs enough detail to be graded fairly.

One example concerns Happy DOM: interrupted asynchronous request or response reads should reject appropriately when the page or browser is closed, while completed reads remain usable and associated callbacks are cleared. I like the way it specifies observable outcomes without prescribing every code edit. The task is one example, not a comprehensive measure of software engineering.

I recommend keeping your own record of failed agent tasks: the prompt, model, tools, repository and commit hash. You can rerun representative examples in a sandbox when tools change and compare outcomes against your needs. A useful internal benchmark also needs clear checks and careful handling of project data.

I inspect a SWE-bench Pro example about email validation in NodeBB. It includes a title, description, reproduction steps and expected behavior. I dislike the formatting and the larger harness prompt; that criticism reflects my preference for how to prompt an agent, not proof that issue-based tasks lack value for other measurements.

The harness prompt describes a workflow: find relevant code, reproduce the issue, edit source files, rerun the reproduction and consider edge cases. I argue that such instructions may measure how a model follows a prescribed procedure as well as its ability to solve the task. How much that changes outcomes requires separate experimentation.

Another DeepSWE example asks for typed and untyped label sorting in a Go project, with explicit ordering and numeric parsing requirements. It is still a detailed specification, but I see it as describing desired behavior rather than dictating the implementation. Different task designs emphasize different skills.

DataCurve also compared the minimal mini-swe-agent harness, which exposes a bash tool, with some official model-specific harnesses on subsets of tasks. The observed scores changed for some configurations. Such comparisons are useful, but a small subset and different harness designs do not prove that every model is being helped or hindered equally.

The video also looks at output tokens, time and cost per trial. The displayed charts use median output tokens and median cost, so those figures should not be described as averages. In that particular setup GPT-5.5 is shown at about 47,000 output tokens and $5.80 per trial, while Opus 4.7 is shown at about 97,000 and $16.

I argue that nominal token pricing is not the whole cost of getting a task done: a model that uses more tokens or fails more often can change the economics. The charts compare particular settings and tasks; real prices, latency, retries and success rates vary by provider, workload and date.

I look at additional models and say the gap between leading configurations and open-weight models is larger on this benchmark than on some other leaderboards. That is a result of the selected tasks and harness, consistent with my experience, but it cannot establish how every open-weight model performs on every kind of coding work.

I mention SWE-rebench as another effort to evaluate more current coding tasks and say its broad pattern resembles what I see here. Different benchmarks, however, have different task sets and methods. Agreement in a few rankings should not erase differences in what each one measures.

DeepSWE has limitations. I would like broader testing of official harnesses, additional models and a harder private task set. Publishing examples aids transparency but can also make future training contamination easier. A high starting score may eventually make a benchmark less discriminating, though the timing of saturation is not known.

The authors acknowledge that a common bash-only harness isolates some model differences at the cost of realism. Models may be accustomed to different editing tools, and developers often use Codex CLI, Claude Code or other interfaces rather than mini-swe-agent. The main leaderboard therefore does not directly describe every native workflow.

The corpus uses active open-source repositories with at least 500 GitHub stars, so results may not generalize to small or private codebases. Bug localization and refactoring are underrepresented, and the five languages omit C++ and Java. The authors also say prompts, though shorter, still need enough detail for behavioral verification.

I appreciate the effort involved in constructing tasks and checking results, and I am glad to have another benchmark to inspect. My enthusiasm is shaped by my investment and my experiences. The practical takeaway is to examine task design, verification and harnesses, then test models on work that resembles your own.