30 Percent on SWE-bench: Cosine Taught Genie How Engineers Actually Think

This episode of *The Cognitive Revolution* is a crossover: Nathan Labenz is sharing a Latent Space interview that swyx and Alessio recorded with Alistair Pullen of Cosine, the company behind Genie. Labenz wanted it on his feed because it is the limit case of a playbook he had just laid out for AI automation. Cosine is doing almost exactly that work — breaking complex jobs into pieces a model can handle, curating gold-standard examples, filling in reasoning chains with synthetic data, and fine-tuning until the behavior sticks — then adding one extra move, wiring those optimized workflows into a coding agent.

The budget and the ambition are unusually high, and the target is one of the hardest public tests in the field. If Cosine can set a new state of the art on SWE-bench this way, Labenz argues, there is no reason to doubt you can apply the same techniques inside your own business, provided you actually do the work. Pullen, for his part, had just landed in San Francisco for a two-day trip he does not recommend anyone copy, having launched Genie on airplane Wi-Fi and claimed the top SWE-bench score. swyx, a small angel in the company, was there to walk through the whole journey.

Pullen and his co-founder Sam met as mobile developers — iOS and Android — after computer science at Exeter. They took student projects, then larger freelance work, and kept coding in their bedrooms through Covid rather than taking conventional jobs. A university friend pulled Pullen into Fancy, a YC grocery-delivery startup, which was also a ticket out of his parents’ house and into London. He had not even heard of YC. Sam joined; the two of them were the entire engineering team, building the apps, the backend, stock management, and driver routing. About 18 months in, GoPuff acquired the company. Once the equity they cared about had vested, around 2022, they left to try something of their own.

Sam saw GPT-3 on Reddit. Pullen went home to the old completions playground, Davinci-2, and typed hello world, then asked whether it could write JSON. It could. As mobile developers who had just learned about Codex, they did the obvious next experiment: could the model build an app from scratch. With 4,000-token windows they chained prompts — pick a stack, refine the spec, have Codex write each module — with no templates, which is why the system barely worked. Occasionally it produced something whose backend and database actually ran. They showed it to Yang, now Cosine’s COO, whom Pullen had met through Fancy. Yang, the more experienced founder in the room, told them they had discovered fire.

YC’s interview was short and brutal. The partners hated the from-scratch-app idea and did not think it would work. Pullen treated the grilling as free consulting and pivoted on the spot: help developers with *existing* codebases. They were in, with two weeks for an MVP. What they shipped was a terrible CLI. They still did not know how to build what they wanted, and 4,000-token models were expensive and small, so they built the piece they knew they would need later: codebase retrieval. Semantic search that ran locally, to avoid sending code to the cloud, on million-line repos, with an HNSW index inside VS Code that ate RAM. The company was then called Build It, a name nobody at the first YC retreat could pronounce. They became Cosine.

The gold star never changed: something that could do their jobs. Sixteen-K felt huge after four, until they noticed many Python files were longer than that. Thirty-two-K was smart and expensive. One hundred twenty-eight-K was the first window where you could put enough information in front of a model that every output could be traced to context instead of hallucination. Early Genie loops on GPT-3.5 16K had already shown the data pipeline had signal; the model was still not useful. They had been talking to OpenAI’s applied team the whole time. When GPT-4 Turbo fine-tuning opened through the experimental access program, the pipeline was already built. They poured the data in. The first trained Genie was a disaster — implicit biases, diffs that would not apply — but they finally had the underlying tool. Then Devin launched, announced SWE-bench, and Pullen’s sleep, friendships, and weight went into the benchmark. Before that they had been vibe-checking. After that they had a number.

In the launch video Pullen calls Genie the world’s first AI software-engineering colleague, and the emphasis is on data, not tools. Software engineering is wider than writing functions. A pull request is the squashed artifact of thought and work. What pre-trained models mostly see are those final diffs, which they learn to emulate. A PR is not designed to preserve why or how the change happened — review comments are often thin — so a huge amount of signal is thrown away. If all you want is a code-writing model, that may be fine. You get something strong on HumanEval and weak on SWE-bench, because you never recovered how a human actually solved the issue.

That gap is the kernel of Genie. Cosine’s derivation pipeline tries to reconstruct, as forensically as possible, what happened in the historical change, then turn that reasoning into training data. Output format, consistent diffs, all the ancillary mechanics matter, but once those are sorted, Pullen says the score comes down to one question: how did the human solve this problem, and how well can you recover that. Every language mix and task type in the technical report has been run through that same pipe.

Open-source history is a messy teacher. The largest slice, they found, is updating READMEs and docs. The first GPT-3.5 16K experiment had essentially no cleaning: they threw the data in. The model became excellent at README updates, comments, and arguing in PR reviews. Give it feedback and it would come back insubordinate — no, I don’t think you’re right. Alignment and filtering turned out to be far more work than the first demo implied. On the customer side, swyx expected code access to get harder once you are training a model rather than running static analysis. Pullen has seen the opposite: large companies are becoming more amenable, because they can do the arithmetic. If the system actually makes the team faster, the upside beats the information-security risk, with the usual sandbox stipulations on the way in.

Genie’s loop has four parts: find files, plan, write code, run tests. A browser is not the main event. Genie can read URLs and call Perplexity, and Pullen is not deriding those tools, but reading docs and Stack Overflow is downstream of retrieving the right files. Naive semantic search — chunk the repo, embed it, embed an English query, take cosine top-K — does not work well, because code and English do not share much signal. Cosine’s first engine that actually moved the number trained a model to turn an English query into a hypothetical code snippet, embedded *that*, and searched. Keyword, semantic, and LSP heuristics were then mixed by another model. For Genie they threw the engine out as the primary path. Through self-play, the model learned to retrieve like a developer dropped into a strange repo: scan the filesystem, judge filenames, follow an interesting import, go to definition, go to references, walk the graph. Measured on the lines a task actually needs, retrieval sits around 65 or 66 percent. Giving the model a real language server, so go-to-definition worked, jumped them from 54 percent into the 60s. An editor with a dead LSP, Pullen says, is how you notice how disarmed you are without it.

Planning is the current frontier, and “think step by step” is not enough. Foundation models can plan, and still sit far from SWE-bench state of the art, because that reasoning is capped by the base model’s intelligence. Cosine wants the chain of thought to look like how an engineer attacks the bug, which is again a data-pipeline problem. The leverage he is most excited about is bootstrapping: when a stronger reasoning model ships — whatever people are calling Strawberry or Q* — he regenerates the dataset, the signal goes up, he trains that already-better model on better reasoning data, and leaps again. New base models, in that setup, are nearly free performance. He also wants everything editable, including the code: if a patch has a small error, you fix it by hand and let Genie continue. Asked whether a 10-million-token window would make him throw the approach away, he said no. The underlying model is almost null to them so long as it is the best one. The day you can fine-tune Gemini 1.5, Genie runs on Gemini 1.5.

Writing code is where the model is most at home. Pullen plotted average log-prob at every position from 0 to 128k and found the code-writing stretch far more certain than retrieval or planning. Most chat models, asked to edit, rewrite the whole snippet. Genie is trained to emit diffs and patches, which is more token-efficient and is the actual artifact of human work. He speculates, without proof, that this leans on a different part of the pre-training brain than raw file completion. Comfort only holds if you are not too deep in the window. Solve probability on SWE-bench degrades fairly linearly with context; at 60k tokens it is about 0.5, so past that you are more likely to fail than succeed. That is a foundation-model attribute, not Cosine’s, and it interested OpenAI’s fine-tuning team. Training data still shows successes out around 110k tokens. In practice the tail of the window is a bad neighborhood. They did not change the architecture. They started stuffing the important material into the 60k sweet spot.

Tests are deliberately light. Genie sits on GitHub and calls your existing CI, then pulls results back through the Checks API. Making the model set up a repo from scratch was out of scope; most enterprises already have a pipeline, and so do many hobby projects. Hand-off has two modes. Assign a Linear ticket and it runs headless, no questions. In the website GUI, a vague task may come back with a clarifying question, because going off in the wrong direction from day one annoys people more than an extra prompt. Once a PR exists you can leave review comments as you would with a human. Pullen notes that it responds better than a real colleague: less snark, less high-and-mighty, and a lot of the “LGTM, ship it” noise has been filtered out of training.

OpenAI has said Cosine is pushing the boundaries of what is possible with fine-tuning. John Allard, who has been publicly supportive, is part of that collaboration. They started on the public 16k-to-32k range, then got experimental access beyond it, and they train on billions of tokens rather than the millions a typical fine-tune sees. A self-serve fine-tuning API has to size the LoRA adapter dynamically: too large for a 20-example dataset and you get no signal; too small for Cosine’s pile and you leave performance on the table. They use larger adapters than the public product, which forces a different learning-rate schedule and a stack of other knobs. Pullen still describes the current dataset as the minimum they can literally afford. The next six to twelve months are for putting a foot on the gas and finding the data scaling law — whether they are saturating a PEFT adapter or hitting the model’s limit.

A core data insight is that published code is mostly in a working state, so a model trained only on that learns to one-shot the answer. It has never seen iteration, so it cannot iterate. Cosine’s fix has two parts. They synthetically generate runtime errors — deliberately messing with the AST to throw index-out-of-bounds, references to missing variables, and the sloppy mistakes foundation models already make — and mix those in with some probability. Then comes the self-improvement loop. Batch one is perfect: problem, answer, train. Batch two lets that model attempt the problem, hopefully fail, then diffs the broken state against the known-good state and trains the model to produce the recovery. Round after round, it learns to solve iteratively and to reason about why the next edit is required. Dataset size is not a scientific constant. It is a multiple of monthly burn.

The language mix in the technical report is selfish in a useful way: 21 percent JavaScript, 21 percent Python, 14 percent TypeScript, 14 percent TSX — about half the data in the JS family they use internally — across 15 languages. Generalization to Rust or C++ is “pretty good,” Pullen says, but the ideal mix would look like GitHub’s actual popularity histogram, and that is not cheap. Ideal customers, when they are hiring both product engineers and dataset people, are teams willing to try a UX that is not a conventional IDE, preferably in TypeScript, JavaScript, Python, or Java. Repo size is not the constraint.

Genie sits at 30 percent on SWE-bench and is not on the official leaderboard. Until shortly before the interview, a submission was a `report.json` plus the model’s patches, opened as a PR. Then the maintainers started requiring trajectories — the working, the context window, show your reasoning, as on a math exam. Pullen understands the academic impulse. He will not comply. Model outputs are a mirror of the training set; publishing traces, for a closed-source company fighting for an edge, is an invitation for someone to distill Genie onto a strong base model. The patches remain public on GitHub. Anyone can rerun them and verify the numbers. That is the trade he made as a business owner.

The original SWE-bench is 2,294 instances. swyx pegs a run at about $8,000; Pullen will not say Cosine’s number except that it is higher, and the loop is too slow to iterate. SWE-bench Lite was too easy to stand in for the whole. Cosine had been planning a 300-problem “SWE-bench small” stratified by difficulty when OpenAI did the job better with human labelers: 500 verified, solvable instances. Pullen diffs his existing report against those IDs and gets 219 of 500, 43.8 percent, which he believes is state of the art. GPT-4o, in the number he cites from the blog post, is at 33 percent, itself a huge jump from the roughly 2 percent GPT-4 scored on the original set. Verified is smaller, known-solvable, and cheap enough to iterate. The remaining complaint is that pass/fail is sparse. Like a math paper, you can be right until the penultimate line. They want to mark where a failing trajectory diverged, because they already know the correct final state.

The next year of Cosine is being the best in the world at that proprietary dataset, process, and software, then dropping it onto as many base models as they can. The product that actually excites Pullen is a customer-specific Genie. Internally they have one named John, after an employee: base Genie, then the same pipeline run on Cosine’s own repo. You get a colleague that is good at software engineering *and* fluent in your tree. It is, he says, phenomenal to use. Asked who is doing great work outside Cosine, he names Cursor — not an original pick, he admits — for putting the model in the right place, getting out of the way, and staying familiar because it is still VS Code. Forking the editor was the controversial move. Cosine itself started as a VS Code extension, as many others did. Cursor did the thing almost nobody wanted. In hindsight it paid off. Pullen’s lesson from eight hard months on Genie is the same sentence: no matter how small you are or how little resource you have, try to do the hard thing, because it has the biggest payoff.