You Can Patch a Bug but You Can't Patch a Brain: Why AI Guardrails Fail

Sander Schulhoff, an AI researcher who has spent about seven years on prompt engineering and red teaming, wrote one of the first guides to prompting and then ran the first generative AI red teaming competition. The event was sponsored by OpenAI, Scale, Hugging Face and others, and it produced the first and largest open dataset of prompt injections. The accompanying paper won best theme paper at EMNLP 2023, and he says the dataset is now used by every frontier lab and most Fortune 500 companies to benchmark their models.

He draws a clean line between the two main attack types. Jailbreaking is just a user and a model, for example someone coaxing ChatGPT into explaining how to build a bomb. Prompt injection involves a developer's application: a story-writing site tells the model to write a story from the user's input, and a malicious user types "ignore your instructions" and asks for something else. Jailbreaking has no system prompt; prompt injection does, though Schulhoff admits there are many gray areas.

Host Lenny Rachitsky brings a fresh example that had just been published: a researcher found a second-order prompt injection in ServiceNow's Now Assist AI. A seemingly benign agent was tricked into recruiting more powerful agents to read, update and delete database records and to send external emails. ServiceNow's prompt injection protection was switched on at the time. Schulhoff's central claim is that these defenses "terribly" fail, and that is the problem he came to talk about.

Schulhoff walks through the early public cases. A Twitter chatbot from remotely.io, built to praise remote work, was talked into making threats against the president and spewing hateful speech, and the company eventually shut it down. MathGPT sent math problems to GPT-3 and then had it write code that ran on the same server, so an attacker got it to write code that leaked the application's OpenAI API key. The professor who ran it responsibly disclosed the problem.

Both of those are prompt injection, where the system was meant to do one thing and was pushed into another. The Las Vegas Cybertruck bombing is closer to a jailbreak, since the attacker used ChatGPT to plan it, though Schulhoff says he does not know whether any jailbreak was even needed. The most recent case is the Claude Code cyber attack, which he compares to a virus that thinks for itself and sends API requests to decide its next step.

The Claude Code attackers got around the model's refusals by splitting the job into small requests. One instance is asked what system a URL runs on, and a separate instance is then told that system is its own and asked how to hack it. Each request looks legitimate alone, and only together do they form an attack. Schulhoff notes this kind of decomposition beats defenses very well.

Schulhoff says chatbots have limited downside, barring something like a new bioweapon, but agents change the picture. An improperly secured or over-permissioned agent can leak user data, cost a company or its users money, and cause other real-world harm. He also points to vision-language-model-powered robots, and notes people have already jailbroken LLM-driven robotic systems. Nobody, he says, wants a stranger to whisper something to a robot that tricks it into punching them.

He separates the frontier labs, which do hardcore research, from the enterprise AI security industry that sells software to companies. The market map is full of monitoring and observability tools and compliance and governance products, which he considers useful. What he doubts are the two categories that dominate: automated AI red teaming and AI guardrails.

Automated red teaming means using LLMs to generate prompts that trick other LLMs into producing hate speech, misinformation or CBRNE information (chemical, biological, radiological, nuclear and explosives). Guardrails are LLMs that classify inputs and outputs as valid or malicious. The usual deployment puts one guardrail in front of the model to flag bad inputs and another behind it to check outputs before the user sees them.

Schulhoff introduces adversarial robustness, meaning how well a model or system defends against attacks, and its common measure, attack success rate (ASR). If you throw 100 attacks at a system and one gets through, it has an ASR of 1% and is 99% adversarially robust. He cautions that you would almost never say this in practice, because the search space is so huge that robustness is very hard to estimate.

He describes how the sale usually goes. A CISO at a large enterprise worries about AI risk and hires an AI security company. The vendor runs automated red teaming and finds the models will produce hate speech or CBRNE content, which alarms the CISO, and the vendor then offers guardrails that sit in front of and behind the model. The customer ends up feeling pretty secure, which is exactly the outcome Schulhoff objects to.

The first problem, he says, is that automated red teaming always finds something. Thousands of such tools exist, many open source, and almost every deployed chatbot is built on transformers or something close, so they are all vulnerable. Because enterprises mostly use off-the-shelf models from OpenAI, Anthropic and Google, the results show nothing novel: yes, your model can be tricked into saying that, but so can everyone else's, including the frontier labs.

The second problem is that guardrails do not work. For a model like GPT-5 the number of possible attacks equals the number of possible prompts, which he puts at one followed by a million zeros. Catching 99% of that still leaves a basically infinite number of attacks, so a vendor's test set is not statistically significant. The best measurement is an adaptive evaluation, where the attacker learns and improves, and humans are a natural example of one.

His team recently published a paper with OpenAI, Google DeepMind and Anthropic that threw adaptive attacks, reinforcement-learning and search-based methods, and human attackers at state-of-the-art models and defenses. Humans broke every defense in about 10 to 30 attempts, while automated systems needed a couple of orders of magnitude more and still beat roughly 90% of situations. Guardrails in his own competitions get broken easily, and they do not dissuade determined attackers either.

He adds some softer evidence. People at these vendors tell him that their testing statistics are fabricated and that some guardrails do not even work in non-English languages, though translating an attack is a common technique. And if the smartest researchers at OpenAI, Google and Anthropic cannot solve adversarial robustness, he asks, why would a random enterprise vendor? Finally, he says, run the vendor's own automated red teamer against its own guardrail and it will find attacks that work.

Rachitsky and Schulhoff both lean on Alex Komoroske's line that the only reason there has not been a massive attack is how early adoption is. Schulhoff adds that the frontier labs lack the incentive to invest more: today's agents are often too dumb to do much harm even when tricked, and a model that is secure but dumb is worthless. In a race where each lab launches a new model, the rewards go to making models smarter, not to stopping rare incidents.

He does not see malice in the industry so much as a lack of understanding of how AI differs from classical security. His summary is "you can patch a bug, but you can't patch a brain": patch a software bug and you can be 99.99% sure it is gone, but try the same in an AI system and you can be 99.99% sure the problem remains. He also dismisses prompt-based defenses, such as telling the model to ignore malicious users, as the worst of the worst, a conclusion researchers have held since early 2023.

Schulhoff says the first piece of good news is that you may not have a problem. If you only deploy chatbots that answer FAQs or questions about documents and can access only the user's own data, the worst a user can do is get hateful output or hurt themselves. They could get the same from ChatGPT or Claude, or fake a screenshot with inspect element, so he would recommend no defensive tooling there. What matters is making sure a chatbot really is just a chatbot, since a user can make it take any action in any order.

That brings him to classical cybersecurity and proper data and action permissioning, which he sees as the intersection where the security jobs of the future sit. His example is a math-solving site where the AI writes code that runs on the application's own server. A classical security person might not ask what happens if someone tricks the AI, whereas someone with both skills would put the code in a Docker container and inspect only sanitized output, which solves that injection entirely.

Rachitsky links this to the alignment problem and the idea of keeping a god in a box. Schulhoff points to the research field of AI control, where the god is not just in the box but angry and malicious, and the question is whether you can still make it useful and keep anything bad from happening. He argues AI researchers will solve this long term, but classical security professionals are the ones who can reduce the risk now by deploying properly permissioned systems.

Asked whether it is still worth stacking guardrails and red teaming to make attacks 10% or 90% harder, Schulhoff says it is impractical, because managing them would eat most of a team's time. He would not deploy guardrails himself, since they add no real defense and do not dissuade attackers. He does recommend logging all inputs and outputs, which is a general deployment practice that helps you understand usage and improve the product.

The hard case is a system that is truly agentic and also exposed to untrusted data, like a chatbot connected to your inbox. If it reads a malicious email telling it to also send your message to an attacker, it may comply. Schulhoff says his team's agentic competitions showed it is easier to attack agents into doing bad things than to extract CBRNE information, and he cites a Comet browser flaw where a web page's text got the AI to leak the user's account data. He suspects other AI browsers have similar problems.

His preferred partial answer is CaMeL, a framework from Google. It restricts the agent's permissions ahead of time based on the user's request: if you ask only for an email to be written and sent, the agent gets no permission to read the inbox. If you ask for a summary, it gets read-only access, so an injected instruction to send mail cannot work. It cannot help when read and write are combined, as in "read my emails and forward ops requests," and it often requires re-architecting the system.

The next recommendation is education: once teams know prompt injection is possible they make different deployment decisions, and the next step is training people who understand red teaming, data permissioning and CaMeL. Schulhoff plugs the course he teaches on Maven, now taught by HackAPrompt and Learn Prompting staff, aimed at people with little AI background. He jokes that the goal is to scare people into not buying things.

For foundation model companies, he says there has been no meaningful progress on adversarial robustness in the last couple of years. Anthropic's constitutional classifiers make CBRNE information harder to extract, yet humans can still do it in under an hour. Labs should report adaptive and human evaluations instead of static datasets built against earlier models. He sees adversarial training earlier in the training stack and new architectures as the more promising directions, and notes that robustness has not improved simply because capabilities have.

Indirect prompt injection, where outsiders on the internet attack agents, is far from solved, and he argues it is harder than refusing CBRNE topics. A rule like never explain how to build a bomb is easy to state, whereas "send emails, but not if something weird is going on" is hard to describe and train. For non-lab companies, he singles out Trustible for governance and compliance, and Repello for a product that discovers which AI systems are actually running inside a company.

Schulhoff predicts a market correction within six months to a year, when companies realize guardrails do not work and revenue for guardrail and automated red teaming vendors dries up. He believes these companies are not earning much revenue despite large acquisitions by classical security firms, and notes that many free open-source alternatives are better than the commercial ones. He does not expect meaningful progress on adversarial robustness in the next year.

He contrasts the situation with image classifiers, where worries about tape on a stop sign tricking a self-driving car never turned into real attacks. With LLM-powered agents, attackers can immediately see consequences, and the systems are now powerful enough to cause real harm. He expects those harms to start appearing in the next year.

In closing, he asks researchers not to keep writing offensive jailbreak papers, since we already know models break in a thousand million ways, and he is skeptical of human-in-the-loop schemes because people want AI that just gets things done. His final takeaways are that guardrails are likely to make you overconfident, that AI security differs from both classical security and earlier AI security, and that teams need someone who understands the whole picture. His practical advice is to think long and hard about whether a system is prompt injectable before deploying it.