You Can Patch a Bug, but You Can't Patch a Brain: Why AI Guardrails Fail

Sander Schulhoff introduces himself as an AI researcher of about seven years, focused on prompt engineering and AI red teaming. He wrote an early guide to prompting, then ran the first generative-AI red teaming competition with sponsors including OpenAI, Scale and Hugging Face. The resulting open-sourced dataset of prompt injections won best theme paper at EMNLP 2023 and is now used by frontier labs and many Fortune 500 companies to benchmark their models.

His central claim is that the AI security industry has a serious problem: guardrails, the models meant to screen inputs and outputs, "just don't work". To set the stage he separates two attack types. Jailbreaking is just a user and a model, for instance a long malicious prompt that coaxes ChatGPT into explaining how to build a bomb. Prompt injection involves an application with a developer prompt, such as a story-writing site, where the attacker tells the model to ignore its instructions and do something else.

Host Lenny Rachitsky shares a fresh example: a researcher showed that ServiceNow's Assist AI could be induced to recruit more powerful agents in a second-order prompt injection. Those agents then performed create, read, update and delete actions on a database and sent external emails, even though the product's prompt-injection protection was switched on. Schulhoff notes this may be the first case he has heard of with actual damage, and quotes Alex Komoroske's view that the only reason there has been no massive attack yet is how early adoption is.

Schulhoff walks through public cases. A Twitter chatbot from a remote-work company was told to ignore its instructions and make threats against the president, and the company eventually shut it down. MathGPT sent users' problems to GPT-3 and executed model-written code on its own server, so an attacker got it to write code that exfiltrated the OpenAI API key, which the person then disclosed responsibly.

On the jailbreaking side he cites the Las Vegas Cybertruck bombing, where the perpetrator used ChatGPT for planning, though he says he does not know whether any jailbreak was actually needed. He also describes the recent Claude Code cyber attack: attackers split a malicious task into small requests that each look legitimate, such as one session identifying a target's backend and another asked how to hack a system. Defenses that judge each request in isolation miss the combined intent.

The stakes change once models can act. A chatbot's worst case is mostly bad content, but a badly permissioned agent can leak user data or cost money, and vision-language-model robots can be talked into harming people. Schulhoff points out that people have already jailbroken LLM-powered robotic systems.

Schulhoff separates frontier labs, which do hardcore research, from enterprise vendors of AI security software. The market map includes monitoring and observability, compliance and governance, which he considers useful, plus automated red teaming and guardrails, which he does not. Automated red teaming means tools, usually LLMs, that generate prompts to trick other models into producing hate speech, disinformation or CBRNE information.

Guardrails sit in front of and behind a model: one classifier screens incoming prompts and another checks outputs before they reach the user. Success is measured by adversarial robustness, expressed as attack success rate (ASR). If 100 attacks are thrown at a system and one gets through, ASR is 1% and the system is described as 99% adversarially robust, although Schulhoff says such figures are very hard to estimate honestly.

He describes a typical sale. A CISO hires a vendor for an audit, the vendor's automated red teamer makes the company's models say terrible things, and the vendor then offers guardrails as the fix. The CISO buys them and feels secure.

Schulhoff gives two problems with the pitch. First, automated red teamers will find something against any model because nearly all deployed chatbots are transformer-based, and enterprises mostly use off-the-shelf models from OpenAI, Anthropic or Google. A non-technical executive may be alarmed, but the honest answer is that everyone else's models, including the labs' own, can be tricked in the same way.

Second, guardrails fail because the number of possible attacks equals the number of possible prompts. For a model like GPT-5 he puts that at one followed by a million zeros, so catching 99% leaves essentially infinite attacks and the sample used to claim 99% is statistically meaningless. The best evaluation is adaptive, using attackers that learn over time, and humans are the best adaptive attackers.

He cites a major paper released with OpenAI, Google DeepMind and Anthropic that threw reinforcement-learning and search-based attacks plus human attackers at state-of-the-art models and defenses. Humans broke 100% of the defenses in roughly 10 to 30 attempts, while automated systems needed a couple of orders of magnitude more and succeeded about 90% of the time. Guardrails also don't deter determined attackers, he says, and people he knows at these companies describe fabricated statistics and tools that don't work in non-English languages.

His closing argument is a question: if the smartest researchers at the frontier labs cannot solve this, why would an enterprise vendor that employs few AI researchers? He adds that anyone can point a vendor's own automated red teamer at the vendor's guardrail and expect to find attacks that work.

Asked why frontier labs don't pour more resources into this, Schulhoff says a common answer is that capabilities aren't there yet: agents are too limited to do much harm even if tricked. Labs also face an incentive problem, since smarter models sell and a super-secure but dumb model is worthless. Rachitsky adds that the release race rewards better models rather than prevention of rare incidents.

He doesn't see malice in the market, but a gap in knowledge about how AI differs from traditional cybersecurity. That is where he gets the line "you can patch a bug, but you can't patch a brain". He also singles out prompt-based defenses, such as telling the model to ignore malicious users, as the worst defenses of all, a conclusion researchers have held since early 2023 from papers and competitions.

Turning to advice, Schulhoff first says that for many deployments this may not be a problem at all. A chatbot that answers FAQs or questions about documents, and only touches the interacting user's own data, can at worst be pushed to produce offensive content, which the user could do with ChatGPT, Claude or Gemini anyway. He recommends no defensive spending there, but warns to verify the bot really is only a chatbot.

The rule is that any data an AI can access can be leaked by a user, and any action it can take can be triggered in any order. That makes classical concerns like proper permissioning central, and he sees the security jobs of the future where classical security and AI security meet. He recommends having an AI security researcher on the team because misinformation is rampant and classical practitioners often don't consider that a model can be tricked.

His example is a math-solving app that has a model write code and then runs it on the application server. A classical security person might not ask what happens if a user tricks the model, whereas someone at the intersection would ask what the worst output could be. The fix is to run the generated code in a container on a separate system and inspect sanitized output, which solves prompt injection for that case.

Rachitsky likens the problem to the AI alignment challenge of keeping a powerful system in a box. Schulhoff points to the field of control, which he supports through the MATS alignment research program, where the working assumption is a malicious AI and the question is how to make it useful while ensuring nothing bad happens.

Asked whether it is still worth stacking guardrails and automated red teaming to make attacks even 10 or 50% harder, Schulhoff says it isn't practical: too many components to manage, and little extra defense. He wouldn't deploy guardrails himself. He does recommend logging all inputs and outputs, not as a security measure but as sound deployment practice for understanding usage and improving the product.

He sums up the split: AI researchers are the only people who can solve this in the long run, but cybersecurity professionals are the ones who can reduce the risk in the short run through careful permissioning. The confluence of those careers, he says, will matter a lot.

The hard case is a system that is truly agentic and exposed to untrusted data, like an assistant connected to your inbox that reads and sends email. An attacker can plant an email telling the bot to forward everything to an outside address. Schulhoff says agentic red-teaming competitions found agents easier to attack than models are to push into CBRNE elicitation, and he cites a recent Comet browser issue where text on a web page tricked the AI into leaking the user's account data. Rachitsky, who uses Comet, notes the same risk likely applies to other AI browsers.

One promising technique is CaMeL, from Google, which restricts an agent's possible actions ahead of time based on what the user actually asked. If you ask for a summary of today's email, the agent gets read-only permissions, so an injected instruction to send mail cannot work. If you ask it to read email and forward ops requests, both read and send are needed, and CaMeL can't prevent the attack.

CaMeL can require re-architecting a system, and it is a framework rather than a product, but classical security people like it because it fixes permissions in advance. Schulhoff jokes that a plug-and-play version would be a market opportunity.

Schulhoff's next recommendation is education: awareness so people don't make risky deployment decisions, and deeper training for teams on red teaming, data permissioning and CaMeL. He describes the Maven course he runs, now taught by HackPrompt and Learn Prompting staff, aimed at people with little AI background. Rachitsky notes it isn't selling software, and Schulhoff says the goal is to scare people out of buying things.

For model builders, he says there has been no meaningful progress on adversarial robustness for prompt injection and jailbreaking in the last couple of years. Anthropic's constitutional classifiers have made CBRN information harder to extract, but humans can still do it in under an hour. He criticizes reporting built on static datasets originally crafted against earlier models and urges more adaptive evaluations.

He floats adversarial training earlier in the training stack, and new architectures, as possible directions, though resources haven't gone there. Indirect prompt injection against agents is harder than blocking CBRN requests, he says, because it's easier to tell a model to never do something than to explain when sending email is fine and when it isn't. He doesn't think capability gains alone will make models robust.

Outside the labs, he praises Trustible for AI governance and compliance and Repello for products that discover which AI systems a company is actually running.

Schulhoff predicts that within six months to a year the market will correct as companies realize guardrails don't work. He says he knows from conversations that guardrail vendors are not earning much revenue despite large acquisitions by classical security firms, and that free open-source alternatives are often better. He doesn't expect significant progress on adversarial robustness in the next year.

He contrasts this with earlier worries about image classifiers fooled by tape on a stop sign, which never became a real problem. LLM-powered agents are different because they can be tricked and the consequences are immediate, so he expects real-world harms in the coming year.

His final advice has several parts. Don't write more jailbreak papers, since everyone knows models can be broken. Be wary of human-in-the-loop fixes, because people ultimately want agents that just get things done. Remember that guardrails make teams overconfident, and that stuff is about to get dangerous as agents and robots are deployed, so teams need people who understand both AI and classical security.

For listeners deploying systems, the most useful thing is to think hard about whether the system is prompt injectable and what can be done, perhaps with CaMeL or a similar defense, or whether it shouldn't be deployed at all.