SWE-agent Tutorial: Navigating Repositories and Attempting GitHub Fixes

I introduce SWE-agent, a coding-agent project from a Princeton team designed around an agent-computer interface. The goal is to give a language model tools to work on GitHub issues: inspect a repository, try to reproduce a problem, edit files, and produce a proposed fix. These are capabilities the project aims to provide, not a promise that every issue will be solved.

The project’s displayed SWE-bench chart reports 12.29% resolved for SWE-agent using GPT-4 and 13.84% for Devin. Its footnote says the Devin result was evaluated on 25% of the SWE-bench test set. These historical, project-presented results involve different evaluation scopes, so the chart alone cannot establish a universal head-to-head ranking.

The project illustrates an agent reading an issue, reproducing a problem involving conversion to R, searching for relevant files, and editing the likely source. I am impressed by the step-by-step interaction because a model has to find useful parts of a repository rather than receive an entire codebase in one prompt.

The team attributes its results partly to commands and feedback designed for language models. I compare the challenge of finding relevant context in a large repository with tools such as Aider, which I say uses Universal Ctags. The question is how to expose enough useful code without overwhelming the model.

The project describes a linter that prevents an edit from proceeding when it introduces syntax errors. Its custom file viewer shows about 100 lines per turn and provides commands for scrolling and searching within a file. I wonder whether a snippet always provides enough context, but the authors report finding this interface useful.

A separate directory-search command returns concise matches, while commands with no output receive an explicit success message. I think of the collection as a small custom development environment for the agent. These interface choices are described by the project; this video does not isolate the effect of each one in an independent experiment.

For the 2024 setup I begin with Docker and Miniconda, clone the repository in a terminal, create the project’s Conda environment from its environment file, activate it, and run the setup script to build a Docker image. These are the steps shown for that version and may differ in later releases.

Despite my initial praise for the packaged setup, I cannot get past a Miniconda-related Docker error on my Apple Silicon Mac. I switch to Lightning, which I say is not sponsoring the video, and complete the image build there. The change of environment is a workaround for my demonstration, not proof of the error’s exact cause.

I create a keys configuration file in the project directory and configure a GitHub token and the OpenAI key used for this test. The interface also lists other provider keys as optional. The setup is specific to the version being demonstrated; credentials and provider settings depend on the actual environment.

I run the agent with a model, a configuration file, and the URL of an issue in the SWE-agent repository itself. The issue concerns a missing `base_commit` record. The agent initializes its environment and starts reading the problem and repository files.

The agent identifies the reported key error and navigates a long file with its built-in viewing commands. It scrolls through several locations before finding the relevant method, then proposes an edit and notices a duplicated setting. I find the sequence interesting, although activity and a proposed patch are not proof of a working solution.

The run stops after exceeding the configured $2 cost limit. I see evidence that the agent made changes, but no completed verification showing that its proposed fix resolves the issue. The project’s aggregate benchmark score cannot be read as the exact probability that this particular issue would be fixed.

I notice that the cost limit can be configured and say I would like to use a local model to avoid the same hosted API charges. At the time I do not find a straightforward local-model option and speculate that future versions might make it easier. Local inference still has hardware and operating costs.

A separate demonstration by an author who introduces himself as Carlos uses a SymPy issue from SWE-bench involving unexpected output from `Matrix.col_insert`. He sends the issue to the agent and then walks through an earlier run of the same example.

Carlos shows the agent creating a reproduction script from the issue and confirming the unexpected behavior. It searches the repository, opens the matrix implementation, and identifies the insertion function as a place to investigate.

The agent identifies an indexing problem, applies an edit, and reruns the reproduction script. Carlos says the output now matches the expected result. The agent removes the temporary reproduction file and submits a proposed diff for benchmark evaluation.

Carlos reports that the submitted patch passes the previously failing test and does not break the existing tests shown in the SWE-bench evaluation. This is one author-selected demonstration, distinct from my interrupted run and from a guarantee about future issues.