DeepSWE: Testing Coding Agents on Longer Software Tasks

I examine DeepSWE, a coding-agent benchmark from Datacurve, and ask whether its scores better reflect the way developers use models. Benchmarks provide one view of performance; they do not establish a universal ranking for every codebase or workflow.

DeepSWE’s creators describe four features: newly written tasks intended to reduce contamination, broad repository coverage, more demanding changes from short prompts, and behavioral verification designed to accept multiple valid implementations.

A task built from a public issue or commit may overlap with material in a model’s training data. The creators say their reference solutions are written from scratch rather than copied from existing patches; that reduces one form of contamination, but does not prove no related information was ever seen.

The benchmark contains 113 tasks across 91 active open-source repositories in TypeScript, Go, Python, JavaScript and Rust. I value this variety because real developers work across different project structures and languages.

According to the benchmark authors, prompts are roughly half the length of SWE-Bench Pro prompts while solutions require substantially more code and output tokens. I like behavior-focused requests because I often tell an agent what is wrong rather than prescribing every implementation step.

The authors audit verifiers for false positives, where an incorrect patch passes, and false negatives, where a correct patch fails. I discuss their reported discrepancies with SWE-Bench Pro and their much lower figures for DeepSWE.

On the leaderboard displayed in the video, GPT-5.5 at extra-high reasoning scores 70%, GPT-5.4 56%, and Claude Opus 4.7 54%, each with displayed uncertainty. This is a result under the benchmark’s particular tasks, harness and configurations—not proof of superiority on every development task.

Some engineers I know praise GPT-5.5, and I see the leaderboard as consistent with that feedback. I also like Opus and later note that my own coding experience has not established GPT-5.5 as the best for me; anecdotes and benchmark results answer different questions.

DeepSWE prompts describe the intended behavior rather than presenting a detailed blueprint. The agent must explore the repository, decide where to make the change and implement it. I believe that resembles many natural developer requests.

The authors say each reference solution is new, although some tasks are motivated by unresolved public issues. A newly written patch reduces direct solution leakage but does not eliminate all possible overlap in models’ prior exposure to a repository.

A behavioral verifier should evaluate whether a patch implements the requested behavior without requiring it to match the reference code. This matters because multiple implementations can be correct, and overly narrow tests can reject useful alternatives.

The chart reports 8.5% versus 0.3% false-positive disagreement and 24.0% versus 1.1% false-negative disagreement for SWE-Bench Pro and DeepSWE respectively. The visible methodology says an LLM judge reviewed sampled rollouts; its verdicts are an estimate, not infallible ground truth or proof of error rates on every task.

Repositories had to be public, actively maintained, permissively licensed and have at least 500 GitHub stars. Each task includes a prompt, executable verifier and reference solution used during review.

The leaderboard holds the mini-swe-agent harness fixed across models to reduce variation caused by scaffolding. I also wonder whether pairing a model with its native coding harness would provide a useful complementary comparison, since models and tools may be optimized together.

The authors show a small pilot on ten SWE-Bench Pro tasks comparing mini-swe-agent with native harnesses. In that slice the shared harness matches or beats the native alternatives, but a ten-task sample cannot establish that every harness comparison is unaffected.

In that pilot, the chart shows median output of about 59,000 tokens for Opus 4.7 using mini-swe-agent versus 48,000 with Claude Code; GPT-5.5 uses about 16,000 versus 25,000 with Codex CLI. These pilot figures are not the same as token usage on the full DeepSWE benchmark.

I note that DeepSWE produces a wider spread of scores than the comparison chart for SWE-Bench Pro. A benchmark can be more discriminating when scores differ, but the size of a spread alone does not establish that it better represents all real-world software work.

The full benchmark’s charts compare score with median output tokens, wall-clock time and cost per trial. The displayed GPT-5.5 configuration scores 70% at about 47,000 output tokens and 20 minutes per trial; these are results for the evaluated setup.

The chart shows GPT-5.5 at roughly $5.80 per trial and Opus 4.7 near $16, alongside their different scores. Per-trial cost depends on the model, pricing, task, harness and settings; it is not automatically the cost per successful solution.

The authors also categorize failures. Their analysis reports that Claude configurations sometimes omit one branch of multi-part requirements, while Opus can inspect Git history when a prompt and repository state appear inconsistent. These are observed tendencies, not properties that apply to every request.

They report that GPT-5.5 misses fewer stated behaviors in their analysis, and use an LLM judge to tag self-verification behavior. The discussion compares how often models write tests when prompts do or do not discourage that practice.

Some developers say the ordering matches their experience; I find that interesting, although I have not reached the same conclusion in my own work. I also note that Composer 2.5 is absent from the displayed comparisons, so the charts cannot establish how it would rank.