GPT-5, Cline and SWE-bench: How to Judge AI Coding Agents

On August 8, 2025, Matthew Berman opens Forward Future Live with Matt Wolfe and Ray Fernando to discuss a crowded week of AI releases. Later, Cline's leaders join to discuss coding agents, followed by members of the SWE-bench team on how to evaluate them.

Wolfe says he was simultaneously underwhelmed by the GPT-5 launch presentation and impressed by the model. He asked it to build a Vampire Survivors clone in one shot and says the resulting game worked, including level-ups. His reaction is an early personal test, not a systematic evaluation.

Wolfe contrasts the presentation with GPT-4's dramatic demonstrations, such as turning a hand-drawn website into a functioning page. The GPT-5 presentation, in his view, made improvements look marginal even though his own coding experience was impressive.

Fernando says his first impression is that GPT-5 is highly steerable: it follows a user's changing instructions rather than insisting on its own preferred approach. He used it in Cursor to inspect an authentication stack and a difficult refactor, treating it like an experienced engineer reviewing his code.

Fernando noticed that the model sometimes explained reservations about a proposed approach while still following his preference. He wants to explore that balance further. His analogy is a driver who actually takes a passenger to the requested restaurant instead of deciding that another destination is better.

Berman asks how steerability differs from sycophancy, citing an earlier model that could validate a bad personal decision. Fernando says GPT-5 did push back when he challenged a refactoring solution, and its explanation reminded him how his existing software stack worked.

Fernando does not expect GPT-5 to replace Claude for every task. He describes making a plan with Claude, then using GPT-5 for implementation and detailed back-and-forth, with a preference for removing unnecessary code. His preferred division reflects his own workflow.

Berman raises the launch's shift toward GPT-5 and automatic routing, which he sees as a simpler experience than choosing among several older models. Wolfe thinks the simplification serves ordinary users who may not know which model to select, though power users may miss choices they understand.

Berman likes the mix of quick responses and additional reasoning. Wolfe says that in his early use simple questions generally got fast answers and harder ones triggered more thinking, although sometimes he wanted more deliberation. He says a user can ask the model to think harder.

Wolfe says he previously compared the same prompt across models and sometimes combined their answers. He expects automatic routing to reduce that burden for users who simply want a useful response without learning a model menu.

Fernando suggests approaching a new model without carrying over every assumption from earlier ones. For his own coding experiments he removes extra tools and MCP integrations at first, then observes what the model can do before rebuilding his workflow around it.

Berman values speed for most of his own questions, turning to reasoning models for hard planning or coding. He argues that scores on some mature benchmarks can miss subtler differences, so he wants to test the model's strengths and weaknesses in use.

Fernando agrees that time spent learning a model's behavior can pay off. He also likes handing work between Opus and GPT-5 to compare perspectives and refine a plan, rather than committing to a single provider.

The panel turns to OpenAI's recently released open-weight models. Wolfe says he ran the 20-billion and 120-billion parameter versions locally in LM Studio on a Mac Studio with an M3 Ultra and 256 GB of memory. He reports roughly 100 tokens per second for the smaller model and 40 for the larger on that machine.

Wolfe says the open-weight release surprised him more than the GPT-5 launch because the local models could build games and apps and, in his experience, approached the usefulness of hosted models for his tasks. These are his impressions on unusually powerful hardware, not measured equivalence across workloads.

Fernando says local inference is attractive for product features that can run offline and in the background, without every device contacting a server simultaneously. He imagines richer applications using spare overnight processing time; he is describing a possible design rather than a deployed product.

Fernando weighs the cost of an expensive computer and power against a subscription and says hosted models still make more sense for his coding. As an app builder, however, he is excited by local inference for tasks such as generating metadata while a device charges overnight, with less dependence on server capacity and connectivity.

Berman raises a speculative theory that OpenAI released strong open weights before GPT-5 to undercut competing paid inference services. Wolfe finds that plausible as a competitive move, while Fernando emphasizes that useful models generate their own word of mouth. None of the panel establishes OpenAI's actual motive.

Berman compares the alleged open-weight strategy to companies giving away products before selling complementary services. Fernando instead stresses the importance of the surrounding tools and user experience: a model that works well at a particular task can gain attention through users discussing it.

On Claude Opus 4.1, Fernando reports useful instruction following but says he has also seen confidently presented solutions he wants to inspect. He is unsure whether prompting or something else explains it, and continues reviewing the code before trusting it; he presents this as a limited first impression.

Wolfe says his own app-building tests made Opus 4.1 and GPT-5 feel comparable in design and functionality. He points to reported SWE-bench scores that were close, but emphasizes the API price difference. Berman displays a historical comparison: $15 per million input tokens and $75 output for Opus 4.1 versus $1.25 and $10 for GPT-5.

The displayed pricing table also lists GPT-5 nano at $0.05 per million input tokens and $0.40 per million output tokens. Wolfe argues that if two models feel close enough for a particular coding job, price may determine his choice; the figures are from the 2025 discussion.

Berman introduces Google's Genie 3 demonstration: a generated world that a user can steer while it is being generated. Wolfe is struck by the apparent persistence of a scene when the viewpoint turns away and back. The examples are demonstrations, not proof of unlimited worlds or finished games.

Wolfe compares Genie 3 with earlier generated environments where moving or turning could reveal a boundary or a changed scene. He sees the demonstrated persistence as an advance and imagines a future where game frames are generated instead of conventionally rendered.

Wolfe sees potential for games with generated frames; Berman relates the demo to Elon Musk's prediction that neural networks could eventually generate pixels, audio and actions directly. They treat this as a future possibility, while acknowledging uncertainty about how such systems would be built and deployed.

Fernando expects progress through many iterations and points to neural networks replacing some hand-written components in autonomous driving as an analogy. He also stresses the software layers needed to connect AI systems to hardware and developer interfaces. Berman invokes the 'bitter lesson' as his interpretation of the trend toward greater reliance on scale.

Berman offers a broad reading of the bitter lesson: approaches that scale general computation can outperform hand-curated solutions. Fernando links the demand for more data and computing resources to further progress, using Tesla's collection of driving data as an example.

Fernando says technical infrastructure still matters, comparing future interfaces to Apple's frameworks that let developers draw without directly managing every hardware constraint. He imagines new layers that help AI systems work efficiently with devices, a proposal rather than an existing Genie capability.

Wolfe wonders whether camera-equipped glasses could also supply data useful for world models, calling it a possible mixture of user benefit and training value. That is speculation. The group also discusses Musk's aggressive predicted timeline for interactive video generation; Berman cautions viewers to take the timeline with a grain of salt.

Wolfe asks how a generated environment could remain consistent for multiple players. Fernando speculates about separate devices sharing information to synchronize generated worlds, while Berman describes the idea as forward-looking. They do not claim that the Genie demo solves multiplayer consistency.

Berman shows that a user can add an event with a text prompt during generation, such as a jet ski appearing on a canal. He asks whether fully generated games will become the norm and how long that might take.

Fernando says he is not a gamer and offers an outsider's rough guess of a decade. In his view, demonstrations must be followed by low-level software, integration layers and people able to connect the pieces into a coherent product. Berman adds that human taste and curation may become more valuable amid abundant generated content.

Fernando says building an effective product requires people who can understand the stack and connect its components, not just a compelling demo. Berman uses the word taste in another sense: selecting worthwhile experiences from a growing volume of generated material.

Wolfe expects generated games to become a genre but is less certain that all games will be made that way. He anticipates resistance from some gamers who value independent human creators, while younger players may focus more on whether a game is fun. Both guests frame their timelines as guesses.

Berman agrees with Wolfe that the final experience matters to players: a game must be engaging and replayable. The group jokes about release schedules before shifting to the difficulty of interpreting model names and announced timelines.

Asked about a proposed Grok 5, Fernando calls model numbering partly a naming game and says he is still waiting to see how a coding-focused Grok 4 develops. Wolfe speculates about the timing of Musk's announcement. The panel also notes that model developers choose which benchmarks to emphasize.

Fernando says he found Grok 4 less steerable than he wanted and hopes a future coding model will improve instruction following. Wolfe guesses that the Grok 5 announcement also served to draw attention during a crowded news week; both statements are opinion.

Berman welcomes Cline CEO Saoud Razwan and its head of AI, Nick. Razwan describes Cline as an open-source VS Code extension that accepts a software task and loops through tool calls, using the editor, terminal and browser to work toward a result.

Razwan says the project began as an experiment in pushing models such as Claude 3.5 Sonnet and using a large context window. Because repeated requests can become expensive, Cline shows the cost of requests and tasks. Users can bring their own models or inference providers; he says JetBrains support is planned.

Razwan says the transparency extends from individual requests to an entire task, allowing people to decide how much they are comfortable spending. Some users bring their own inference and take advantage of provider credits rather than being restricted to a single model.

Nick explains that Cline's bring-your-own-API-key approach exposes inference costs directly. He argues that subscribers to a flat-price service may not see those costs and that providers face pressure to limit their own expenses. In his account, some Cline users nevertheless spend heavily because automation can justify the cost of engineering time.

Nick argues that pricing incentives can affect technical choices, including retrieving smaller pieces of context or using secondary models to apply edits. He says Cline instead favors giving a strong model enough context and avoids certain retrieval and instant-apply systems. These are his product team's views, not independent findings about competitors.

Nick says Cline avoids embedding-based retrieval and a separate fast-apply model. He argues that applying edits with an additional model can introduce subtle mistakes or obscure failure, though this is his assessment of the trade-off and other systems may make different choices.

Berman relates pricing to his experience running a SaaS company and asks how Cline can make money without an inference margin. Razwan says the starting point was his own desire to see and control spending while swapping models according to task difficulty and cost.

Razwan describes choosing a cheaper model for cost-sensitive work, Sonnet for a balance and Opus for harder tasks. He says showing the economics helps users build judgment about when a more expensive model is worthwhile.

Razwan says users learn which models and price points fit different jobs. The same architecture also enables local models through tools such as Ollama and LM Studio, although he says earlier local models were often not capable enough for Cline workflows.

Cline's intended business is enterprise software and services around the open-source project, Razwan says. He describes demand for team management, billing, audit logs and role-based access control as organizations seek to roll the tool out to hundreds or thousands of people.

Berman says enterprise customers in his own SaaS business needed security, compliance and support rather than simply more usage. Razwan confirms the parallel and lists administrative capabilities needed for a large deployment.

Asked how Cline incorporates new models, Nick says GPT-5 has been quiet and literal in his tests. That exposed ambiguity and contradictions in an existing model-agnostic system prompt, including problems with a plan-mode response tool.

Nick says OpenAI suggested using GPT-5 itself to inspect the system prompt and identify minimal changes. The model proposed clarifications and additional tool parameters, including a flag for whether more exploration was needed. Cline evaluates releases with practical tests and internal benchmarks, including a difficult diff-editing test.

For one proposed fix, GPT-5 suggested adding a parameter that would make the agent show more evidence of exploration before responding. Nick says Cline ultimately added a flag asking whether more exploration was needed, illustrating how prompt and tool design interact with a model's behavior.

Razwan says a model's workflow matters beyond whether it passes a task: he watches how it proceeds, how long it takes and the mistakes it makes. He reports that in Cline's internal testing GPT-5 was worse than Claude models at structured diff editing. This is a task-specific result from the company, not a universal ranking.

Razwan says generalized benchmarks cannot easily reproduce every developer's individual workflow. For his own assessment he prefers practical tasks, especially tool calling, file editing and structured outputs, while acknowledging that different models are good at different things.

Nick criticizes common coding benchmarks for failing to represent the full agentic workflow. A one-shot file edit can look different from an agent navigating a real repository, he argues, and he wants evaluations drawn from real tasks rather than relying solely on synthetic examples.

The two Cline leaders choose different defaults. Razwan says he still uses Claude 3.5 Sonnet for his own work, citing results and cost, while Nick says GPT-5 fits his style: he specifies an architecture and wants the agent to implement it closely with little commentary.

Razwan says newer Claude models' use of thinking did not fit his workflow as well as Claude 3.5 Sonnet and could cost more when the thinking budget was high. Nick, by contrast, writes the architecture and specification first and values GPT-5's willingness to follow them closely.

Nick describes using GPT-5 to create a Chrome extension for Discord-style replies in Slack. He says he built it in about an hour, published it and open-sourced it; the example illustrates his workflow rather than establishing typical performance.

Razwan says Cline raised $32 million across seed and Series A funding, with Pace Capital leading the seed and Emergence Capital involved in the later round. He plans to expand enterprise features and IDE support, and says the team grew from roughly four people in January to nearly 30 at the time of the interview.

Berman then welcomes SWE-bench team members Ofir, Carlos, John and Kilian. The team explains the benchmark's origin: open-source repository issues often have corresponding pull requests containing both a fix and tests. A model receives the issue and an earlier state of the repository, changes the code, and is evaluated with the associated tests.

The SWE-bench process uses the repository state from before the fix, not just a standalone coding question. The associated tests check whether a proposed change resolves the issue, connecting the evaluation to a development workflow the researchers saw in open-source projects.

The team says its benchmark code is open source, so model developers can run their own evaluations for launch. The researchers also independently test accessible models using mini-SWE-agent, a small common scaffold, and those results can differ from vendors' published numbers.

Carlos explains SWE-bench Bash Only: earlier submissions allowed very different systems, including retrieval and multi-agent arrangements, making it difficult to compare the underlying models. The newer setup gives models the same minimal environment and a bash interface, with less specialized scaffolding. Scores can be lower because the models receive less assistance.

The bash-only approach removes specialized tools and uses a simple text interface, which the team says permits testing models that do not support native tool calling. Every model in that comparison receives the same environment rather than its own custom scaffold.

The team adds that sophisticated submissions may try a problem many times and use a judge to select an answer, changing both cost and conditions. Their common setup aims for a more comparable measurement. They also say excluding problematic instances can change a score by several percentage points and prefer reporting results over all 500 verified issues.

Berman asks about contamination because the benchmark and source repositories are public. A team member says the repositories are inevitably represented in training data and acknowledges criticisms such as the benchmark's representation of Django. Their aim is to track model capability correlated with real work, not to declare software engineering automated when a score rises.

The researchers describe an experiment in training a smaller model on successful solution trajectories: exposure to a solved issue did not guarantee the student would solve it. They take this as evidence that the task is more than simple memorization, while acknowledging that the relationship between training data and scores remains complicated.

Kilian describes comparing performance on older issues with newer issues after model knowledge cutoffs. He says they have not observed a simple collapse after the cutoff, and that differences in issue difficulty over time are another possible explanation. The temporal analysis does not prove that contamination is absent.

The team says testing all 500 issues gives a consistent denominator, even if some are difficult or broken. It also acknowledges that a different denominator can be justified if the goal is to report only on issues a provider considers solvable; the setup must be clear.

The team notes that OpenAI's GPT-5 model-card result omitted 23 of the 500 verified instances. They distinguish that vendor setup from their own common-scaffold comparison and say different agent systems and instance selections make headline numbers hard to compare directly.

In the team's displayed historical chart, Claude Opus 4 scores 67.6%, GPT-5 65.0%, Sonnet 4 64.8%, GPT-5 mini 59.8%, and GPT-5 nano 34.8% under the shared setup. The guests clarify that the Opus result shown is 4, not 4.1; the latter had not yet been added to that chart.

The researchers mention a Qwen3-Coder result of 55.4% and turn to benchmark saturation. They say SWE-bench will not remain challenging forever, so the team is developing multilingual tasks across nine programming languages and multimodal issues that include images, both harder at the time of the conversation.

The multilingual benchmark uses new repositories and nine languages, while the multimodal version includes issue descriptions with images. The team reports lower scores on those variants than on the established benchmark at the time, leaving additional room to measure progress.

A researcher argues that qualitative impressions can sometimes be turned into measurable tasks. As an example, AlgoTune asks models to make programs faster, allowing increasingly ambitious optimization approaches rather than ending at a single fixed answer.

The group says cost and number of steps are important dimensions alongside a pass rate. A model that solves more cheaply or in fewer steps may create a different user experience; complex commands can also affect usability. The appropriate comparison depends on the application.

The team returns to its design philosophy: instead of prescribing retrieval pipelines and tools for each stage, give the model a minimal interface and let it find files, choose edits and respond to errors. Mini-SWE-agent pushes this approach toward a small loop with shell access and feedback to the model.

The researchers say the small agent passes errors back to the model and lets it decide what to do next. Their aim is to put more of the work on the model, making the measurement less dependent on elaborate human-designed pipelines.

Berman thanks the SWE-bench guests and encourages viewers to examine the benchmark and run models against it. He closes the August 8 episode after discussions spanning model launches, practical coding-agent design and the limits of a single evaluation score.