The SWE-bench Team on Coding Agents, Evaluation and the Future of Programming

Matthew Berman interviews Princeton researchers Kilian Lieret, Ofir Press and Carlos E. Jimenez about SWE-bench, SWE-agent and the future of software development. In this December 2024 conversation, the team explains how its benchmark and agent were built and discusses plans that were still forthcoming at the time.

Berman asks about their coding stacks. Lieret says he moved from GitHub Copilot to Cursor, appreciating suggestions that can make a logical follow-up edit elsewhere in a file. He says he was using Claude through Cursor; Berman also finds its ability to reference files useful.

Press says that during SWE-agent development he used Visual Studio and consulted a language model through a website. Much of his own contribution involved reading agent trajectories and adjusting prompts. Jimenez says the team also used Cursor and AWS infrastructure for cloud workloads and automated SWE-bench evaluations.

Jimenez explains that SWE-bench grew from thinking about GitHub as more than a collection of code. Its issue and pull-request systems preserve examples of users reporting problems and contributors proposing fixes. That history offers a way to construct software-engineering tasks from actual open-source projects.

The benchmark gives an agent an issue and a repository context, then evaluates its proposed change using tests. Unlike a short programming puzzle asking for one function, these tasks can require navigating an existing codebase and changing multiple files. Passing the benchmark’s tests measures success on those selected issues; it does not establish that a system can perform every software-engineering task.

Lieret recalls that the tasks initially seemed too hard to attract many attempts. Press clarifies the team history: Jimenez and John Yang led the original benchmark work, Press joined, and Lieret joined around its release. Yang is not present in the interview.

Press recalls an early baseline resolving about 1.96% and initially thinking 6% would be extraordinary for their agent; he had offered gelato if the team reached it. As development progressed, he says their result rose to around 13%. These are historical results on the team’s evaluation setup, not scores to compare casually across later benchmark variants.

The group tried many commands and tools and read the agent’s trajectories to understand failures. Press says a useful insight was to design an agent-computer interface for a language model, just as a graphical interface is designed around human users. The model needs manageable, informative feedback from its environment.

One example is the file viewer. Showing an entire long file could overwhelm the agent, so the interface presented 100 lines at a time with commands to scroll and search. In their experiments, Press says that window worked better than the larger 200- or 300-line alternatives; that is an empirical result for their setup, not a universal context-size rule.

Editing was another difficulty. Agents sometimes duplicated a line or modified an off-by-one range. Press credits Yang with integrating a linter after edits, allowing the system to catch some mistakes and provide feedback. The improvement came from repeated experiments, not simply giving the model a larger prompt.

Press recalls that Cognition contacted them shortly before launching Devin and submitted a result to SWE-bench. Public attention to Devin, followed by the open-source SWE-agent release, helped bring interest to the benchmark. He says their early result was comparable in that historical comparison, with different speed and openness; these remarks do not establish a current product ranking.

Berman summarizes SWE-bench as a test for coding tasks drawn from real issues and SWE-agent as a model wrapped with tools to attempt them. Press places tools on a spectrum: autocomplete helps while a developer writes, while the research agent aims to take an issue to a proposed change without intervention. Human review remains important even if an agent operates autonomously during a run.

Berman predicts that AI tools will initially increase the number of people who build software, then speculates that programmers may eventually become unnecessary. He points to generative game demonstrations as a thought experiment about applications created on demand. This is his prediction, not a finding from SWE-bench.

Jimenez agrees that assistants may lower the learning curve for unfamiliar languages, frameworks and tools. A model’s broad knowledge can help people get started more quickly, although generated work still needs evaluation. He expects that broader access could bring more people into building software.

Looking further ahead, Jimenez imagines agents attempting bug fixes and feature requests while people increasingly review their work. He suggests running an agent on many issues may be inexpensive in some settings, but costs depend on model, context, attempts and infrastructure. His vision of engineers shifting toward quality assurance is a forecast.

Berman asks whether a language model might itself become the operating system and generate what a user needs without a conventional application. Jimenez calls that an extreme and unusual vision, with potential for interesting research but uncertain efficiency. A generative game demonstration does not establish that such a system would replace ordinary software.

Lieret expects capable developers to become more productive as assistants reduce repetitive coding. In the medium term, he says the effect on employment is uncertain: greater efficiency might reduce work per project, but cheaper software could also create demand for projects that were previously impractical.

For larger or more critical systems, Lieret emphasizes predictable behavior and guarantees. He expects conventional code to remain important because systems need properties that can be checked, even if humans express requirements at a higher level. Deterministic code still needs testing and does not automatically guarantee correctness.

Lieret also worries that models trained on existing code may reproduce established patterns rather than adopt new ones. A developer can explicitly request a new approach, but must specify what is wanted. He expects programming to persist, though the language and the balance between specification and implementation may change.

Press predicts that agents will increasingly turn natural-language bug reports into pull requests for humans to review. He estimates that solving a majority of routine issues and feature requests could take several years, while noting that contemporary agents struggle with complex bugs and features. His percentages and timeline are forecasts from 2024, not measurements of present performance.

Over a longer horizon, Press imagines ordinary users asking a computer to add a useful button to a spreadsheet and getting it directly. He is excited by wider access to technical capabilities, while acknowledging that ten-year predictions are difficult. The example is an aspiration rather than a demonstrated guarantee.

Lieret cautions that older no-code tools faced problems with control, tracking changes and collaboration. He expects a distinction between a casual one-off task and professional software that must be reliable and maintained. Berman agrees that explaining requirements, thinking through edge cases and evaluating a system remain demanding work.

Berman asks whether programming languages themselves will change if AI writes more code. Jimenez says Python’s readability reflects human preferences, which may not be identical to what is easiest for a model. He speculates that stronger typing or a different language structure might prove useful, while emphasizing that he does not know.

Jimenez notes a practical counterweight: popular languages such as Python and JavaScript have large amounts of training material. He imagines a higher-level language or a model-assisted compilation process but treats the idea as speculative. Berman adds that a language optimized for models need not necessarily look familiar to a human reader.

Lieret considers languages with stronger guarantees, since typing extra characters becomes less costly when tools complete code. He also imagines programmers writing more statements and tests describing desired behavior while a model fills in implementation. This is a possible shift in workflow, not a proven replacement for implementation design.

He cautions that strict test-driven development is difficult when writing the implementation reveals what the problem actually is. Integration tests may be easier to specify early than unit tests tied to an unknown implementation. He anticipates a more fluid mix of specifications, checks and code rather than a single settled process.

Press stresses the importance of training data and compute in model development. A newly invented AI-specific programming language would initially lack a large training corpus, which he thinks would make adoption difficult. He cannot say whether a different language would be better if all languages had equally abundant data.

Lieret suggests that transpilation could connect a new language to an established one with more training examples, as a direction worth exploring. The group agrees that the question is open. Their discussion is about possible research directions, not evidence that a particular new language will prevail.

Berman asks what comes next for the projects. Press says the team is working on a major SWE-agent refactor and a 1.0 release intended to be easier to run and extend, locally and in the cloud. At the time of the recording this was a plan; availability and implementation could change.

Press also introduces SWE-bench Multimodal, which extends issue-solving evaluation to tasks with visual components, including user interfaces, maps, plots and diagrams. He says the evaluation infrastructure was being prepared for release. The benchmark is a selected set of issues, not a comprehensive measure of all visual programming.

Lieret explains that the refactor aims to make SWE-agent’s codebase smaller and easier to modify while improving parallel execution. He describes the goal of running many instances concurrently, subject to resources and model rate limits. His estimates for faster evaluation are goals for that proposed setup, not guaranteed runtimes.

The team separates remote execution into a package called SWE-ReX. Lieret says it manages environments needed to check out a repository, install dependencies and run an agent’s code, with local Docker and cloud backends among the intended options. A sandbox is a useful isolation mechanism, but no implementation should be assumed risk-free.

Jimenez describes SWE-agent as a configurable agent and tool layer above the execution backend. He says users would be able to choose components such as a browser or display, while SWE-ReX manages execution. The separation is intended to make the system more flexible for both users and researchers.

The team also planned a cloud evaluation API so users could submit predictions instead of running every evaluation container locally. Jimenez says parallel cloud execution could reduce waiting time for some users. Actual performance, availability and cost depend on the service and workload.

For SWE-bench Multimodal, Jimenez says AWS support would initially help provide free evaluation, with duration dependent on demand and resources. Press adds that server-side scoring can reduce opportunities for accidental or deliberate test leakage and enable public results. It is a mitigation, not proof that all benchmark manipulation is impossible.

Berman thanks the team for the interview and its open-source work. The discussion closes with links to SWE-bench, SWE-agent and SWE-ReX, and with the distinction between their 2024 plans and any later releases.