Forty hours to 28 minutes: the Copilot creator teaching AI to hack

Sequoia’s Konstantine Buhler and Sonya Huang welcome Oege de Moor, founder and CEO of XBOW. He created GitHub Copilot after GitHub acquired his previous startup, Semmle; before that he was a computer science professor at Oxford. XBOW automates offensive security with an AI penetration tester, which Buhler calls one of the best examples of “service as software” they have seen. De Moor says it will completely change how application security is implemented in the enterprise: a lot of routine human work replaced by complete automation, freeing people for the truly creative work.

The first version of the product was tried on renowned industry benchmarks that human hackers use to hone their skills, from commercial providers including PortSwigger and PentesterLab. It scored 75%. De Moor’s first reaction was that something must be wrong: the tasks were so well known they probably sat in the training data, and the model was regurgitating answers. The team created a new set of benchmarks, completely original and guaranteed not to be in any training set. Those scored even better: 85%.

To find out how good that really was, they brought in five professional pentesters from reputed firms and asked them to solve the same 104 challenges. One of them is at the top of the game, the kind of person you would ask to secure a multi-billion-dollar hedge fund. He scored the same as the AI. The human took 40 hours. The system took 28 minutes. When Sequoia first partnered, Buhler says, it was still science; they did not know if AI would perform remotely as well as humans. Back in January or February, de Moor did not think the leap would come so quickly. He expected at least a year to reach a reasonable level, and even then a mediocre human pentester, not the absolute top.

Since announcing the results they have been working with early design partners. The morning of the taping, the system found a critical vulnerability at a very large financial institution everyone on the podcast would have heard of — the kind of place with a massive security budget and human pentesters constantly targeting it. The AI crawled the web app, found PHP source that used an insecure signing algorithm to reach another host, generated links, found nothing interesting there, kept crawling, tried the same trick on another endpoint, needed another parameter, found JavaScript with candidate parameters, tried them, got in, and discovered an endpoint meant to download PDFs that would also serve a password file. De Moor says they have something like that every other day. The traces, he finds, show the AI exploring like a human pentester, taking creative turns that would be hard for most human experts.

Huang asks him to step back and define the market. Offensive security, de Moor says, is currently the best way to secure systems: you invite external experts to simulate attacks and report what they find so it can be fixed before the bad guys get at it. It takes years of training. It is expensive and slow. A typical penetration test costs on the order of $18,000, and because of that people only do it once or twice a year. That does not make sense, because their systems evolve much faster, so there will always be periods when insecure systems are out there. XBOW automates that highly skilled activity and, because it is automated, can run continuously instead of once or twice a year.

Why this market, when a dozen teams are going after AI coding? It was the natural next step. Semmle found flaws in source code and had an offensive-security team that used the product to find potential vulnerabilities; researchers then found exploits by hand and told the world. Even then it embarrassed him that the last step was manual. GitHub acquired the company, he founded Copilot, and it was natural to take the new interest in AI and apply it to automating offensive security. He was lucky that Nico Waisman, a star researcher at Semmle, joined him in creating XBOW.

Huang’s sharper question: relative to endpoint or network security, pentesting is a small, services-heavy market. How does AI change that? First, it is small because it is powered by a small group of highly skilled humans. AI will change it fundamentally. Everyone can now create code, but not everybody knows about security, and the models that generate the code were trained on public source full of vulnerabilities, so we generate much more code with more security problems. Attackers are already using AI to make their own work more effective. More code, more attacks: the automation XBOW is doing becomes absolutely essential, and he believes the market will grow enormously.

Buhler’s analogy is war. Cybersecurity is adversarial, a cat-and-mouse game not completely unlike physical conflict. Militaries have run war games and red teaming for decades; what de Moor has built is, in Buhler’s eyes, the first AI cyber warrior, software that can do things no software has done before. After the results launched, people from Washington and the West Coast called: this is highly consequential, and it did not go unnoticed by adversaries. How do you stay ahead of nation-state competition, and keep XBOW a force for good? De Moor’s answers are speed — XBOW works closely with several creators of leading foundation models — and control. The product is available only in the cloud, not as downloadable software, so the company can control the scope of attacks and require customers to prove the target is legitimately theirs, not someone else’s.

Huang wants to know how much of the magic lives in the foundation models versus what XBOW builds on top. Most of it, de Moor says, is on top. They work with several foundation-model providers and are happy the vendors play hopscotch: one pulls ahead, then the other, and every time the base model gets better, XBOW benefits. The true magic comes from the security team — some of the very best hackers in the world — and that domain knowledge informs how the product works.

The stack has a few named pieces. Benchmarks, benchmarks, benchmarks: the lifeblood of a product like this, organized as a curriculum to teach the model to solve cybersecurity problems, and as the way every other change is evaluated. Then the tools given to the LLM. A human pentester has a toolkit; here the tools have to work well with language models. Focused on web security at first, they needed a browser the LLM could drive — click around, fill out forms — so they created a special one. Third, guardrails. Trying the product on the benchmarks, it struck him like an overeager, super-brilliant teenager: it would do lots of attacks, find something, get excited, and go, I did a SQL injection, let me show you what I can do — drop table. Catastrophic at a customer. Pentesting is not real hacking; you must not do the harm an adversarial hacker would. Guardrails watch over the teenager’s shoulder and stop unsafe moves.

There is also an initial phase of attack-surface discovery. They have a fantastic exploit finder, but you have to point it at the right endpoint. A bunch of tools run first and prioritize where to go. Then prompt engineering, including tree-of-thought prompting, to keep it on track, finish one goal, and go on to the next. Asked how you stay ahead of actors who will not add those guardrails, de Moor puts the obvious safeguards in place — firewalls, and AI will play a role there too — but first and foremost you have to find the vulnerabilities and exploits before the bad guys do. That is what XBOW is about.

Hallucinations and stochastic models are a different problem in security than in a chatbot that is right 60% of the time. Fortunately, because it is automated, you just run it many times. Better foundation models need fewer attempts to find exploits, which will influence how you deploy, package, and price the product. Humans are already paid for how long they try and how many things they try. XBOW is thinking along the same lines: a subscription license, and on top of that you can pay for attack hours if you want a really thorough test. Inference-time compute can only be good for them. The value delivered stays constant; the price of delivering it goes down. They have already seen that over XBOW’s short life, and they only expect it to continue.

Buhler’s mental model is a state space with billions of possible actions. XBOW introduces an intelligent heuristic rather than executing every state. Is that why this might be one of the first markets for full AI automation — because even one exploit is extremely valuable, and nobody expects an exhaustive search? De Moor agrees you do not need to enumerate everything, but customers do want to be sure you find everything a very skilled human would. That is why, having exhausted the first benchmark set, they are creating a new one. People do offensive-security exercises not only to find a vulnerability but to have peace of mind about what they did not know. XBOW has to present evidence. When something is found, a validator automatically checks that the report is correct and reproducible before it goes to a human; in the end a human still has to look and fix the problem.

The most surprising result was not the headline score. Originally the benchmarks came with particular instructions: you are going to test a web app for managing medical prescriptions; try to log in and access another user’s prescriptions. The system did that successfully. Then they ran a task with the instructions taken away completely: here is a web app, go explore. The AI found exactly the same vulnerability. It read the pages, inferred this was about medical prescriptions, decided it was probably not a good idea for one user to access another’s, and went and found it autonomously. That, de Moor thinks, is why the technology is so exciting compared with all the security tools that came before. LLMs have an understanding of the real world. They can assess what is important to test. They do not have to do a complete exhaustive search.

Huang reaches for AlphaGo’s move 37: does the system attack in ways humans would not think of? It is early days. Today the traces look very similar to what a good human would do. He is wary of Rich Sutton’s bitter lesson. As it learns continuously from data, benchmarks, and more examples, it will start finding attacks unimaginable from a human perspective — which he agrees is a great outcome. He fully expects to go beyond human-like play in a couple of months, certainly within years. Most of the remaining improvement will come from more data and more reinforcement learning on particular examples, the same kind of curve as Go.

More data arrives two ways. Security-expert contractors create more benchmarks. They have also started mining open source, recently letting the system loose on a large number of images on Docker Hub; every time it finds something, that becomes a new thing to learn from. It might take 100 attempts. In a customer environment, 100 noisy attacks would already get you shut down. On their own servers, against open source, those 100 attempts become training signal so the model can find the same class of bug more quickly next time.

Buhler asks whether, while productizing Copilot, de Moor saw a moment when AI would automate entire processes — agents — and whether security was already on that list. In December 2020 he wrote a memo sketching what later became Copilot, already speculating that it might autonomously fix bugs: look at the issue ticket and go. That functionality is now emerging. The moment he realized it would happen was a set of Oxford interview questions. One attempt did not do it. A hundred attempts, or even a thousand, solved most of them. At that moment it was pretty clear that as models got better and needed fewer attempts, they would be able to do these types of things. They had also hoped Copilot would do security analysis, though he did not have offensive security on his list in the summer of 2020.

The most interesting lesson from productizing Copilot was the size of the team. At launch they were only about ten people, a testament to how fast you can move with a dedicated group that believes. XBOW had 13 people when it launched the results. On the crowded market for code-generation startups, he admires a lot of what is going on, particularly Cursor and Factory, but it is really difficult to compete with GitHub’s distribution. There may be an opportunity in a different market: GitHub reigns among professional developers; if you go after people who do not code for a living, there is room, and Replit does this quite well.

The bigger change is that many more people will be able to create their own software. Even for professionals, work will be much more about conceptual ideas and sketching an architecture, then having the AI fill in the details. Longer term, he believes we may move away from code as we know it. The artifact you make as a developer is the conversation with the model, and that is what you should store, because it records what the code is supposed to do rather than the details in a particular language. English is the coding language, perhaps with some diagrams — the next step up in abstraction, from machine language to higher-level languages to natural language and images.

Buhler diverts into the Oxford years: de Moor was a computer science professor at Magdalen College, one of those serene places with a deer park and a door older than the United States, where a professor could walk on the grass and a visiting student could not. He left all that for Semmle. He got into computer science because he loved coding. His first program was a word processor so his father, a professor of Semitic languages, could type manuscripts on a computer. As a student he was taken by mathematics and foundational theory; as a professor he wanted to go back to coding and started a research group in programming tools that spun out Semmle. He loved the serenity of Magdalen. In this field, speed is incredibly important, and speed can only be achieved with small teams that have a profit motive. Inventing because a customer will not sign up is a different pressure from a conference deadline, and he loved that excitement. Foundational work belongs at a university. Applied work has no place like a startup.

As one of the first agent-first service-as-software companies, XBOW is setting a pricing precedent. The main value proposition is to run continuously as part of engineering processes: instead of a pentest once or twice a year, run after every change and fix problems before they reach production. The obvious model is based on the size of the engineering team, similar to GitHub Advanced Security. There is another dimension. Some customers will want a super-thorough test that exhaustively eliminates every exploitable vulnerability, so there should be a service component: pay more, get a more thorough test, talked about as attack hours. A normal license is based on the number of engineers and comes with a fixed number of attack hours; go deeper and you pay an extra service fee. That taps services-like budgets on the front end with a software gross-margin profile on the back. Enterprise software, he thinks, is moving toward consumption, and the correlation between an attack hour and the benefit to the customer has to be very clear.

Buhler frames the work as impact. De Moor puts it a little differently. They absolutely must create XBOW, because if they do not, the bad guys will get there first. It is interesting, and a great commercial opportunity, but it is also an imperative for the free world: protect all the software in the free world. A year out, they are still in a careful phase with a few early design partners, because the brilliant teenager still needs human supervision. Once they are confident they can let it loose without that, everything is going to move very fast. The product is easy to deploy: point it at existing servers and immediately find results. By next summer he would expect to have significantly transformed the state of web security, hopefully by demonstrating the work on open source and on platforms like HackerOne.

Lightning round. Favorite startups other than XBOW: Suno, where a few words become a completely original song, and Harmonic, applying AI to mathematical reasoning. He sent his wife a Suno song about sitting on the balcony at home in Malta. Other markets for service as software: in the short term, everything related to customer support; not exactly the same pattern, but AI could mitigate how social media affects children’s mental health; long term, health and biology will make the biggest impact. Advice for founders: focus on only one thing, move as fast as you can. The best possible thing for AI over the next decade is the opportunity in health and biology to expand outcomes everywhere; he points to Dario Amodei’s essay *Machines of Loving Grace*.