Fifteen Examples Can Poison GPT-4: Today's AI Systems Are Exploitable by Default

Adam Gleave founded FAR AI about a year and a half before this conversation, after a PhD at UC Berkeley, to sit in a gap he thinks the field is leaving open. Frontier labs such as OpenAI, Anthropic, and DeepMind, he says, run narrow safety agendas aimed at making today's models safer in the here and now, while academia can explore broadly but rarely gets past a toy prototype. FAR AI's bet is medium-scale work: a few engineers, a year, modest compute, enough to prototype and de-risk an agenda before it is scaled in-house or picked up by a lab. Along the way the group has also done field-building, including an International Dialogue on AI Safety with senior Chinese scientists such as Turing Award winner Andrew Yao, and an alignment workshop just before NeurIPS 2023 in New Orleans.

Host Nathan Labenz frames the episode around a blog post in which Gleave's team listed exploits in GPT-4's fine-tuning and Assistants APIs. The skeptic's reply, which Labenz had already heard on a paper about GPT-4 autonomously hacking websites, is that a human hacker could do the same. Gleave splits the answer in two. In the near term the issue is economics, not novelty: good hackers are rare and expensive, a zero-day for Android or iOS still costs around a million dollars, and large ransomware shops are often state-linked, as with North Korea, because turning an exploit into cash is hard. Automate the chain end to end and a small group can operate at the scale of a criminal organization. Defenders get the same tools, as they did when Metasploit packaged public vulnerabilities a decade ago, but only if they actually put them to work.

The longer worry is the trend. There is a clear path to more capable models through data and compute, and after a period when OpenAI looked like a laughingstock for betting the farm on large language models, others are playing catch-up. Gleave thinks people who were underwhelmed by Google DeepMind's Gemini missed the point: it closed most of the gap quickly, and the compute and money are there to keep pushing. If today's models only do what any human attacker can do, in two years they may need a specialist, and in ten they may do things no unassisted human would. Labenz restates it as a scissors gap: capability is rising, so is control, and the slopes are pulling apart. FAR AI's early scaling work on adversarial robustness points the same way. Bigger models are more robust, but the gain is much slower than the gain in capability.

Finding the holes, Gleave says, was not hard. Some attacks failed and never made the blog post; the ones that worked took about a day of trying. That is the defender's real problem. It is not enough to put up a twelve-foot fence, because someone can dig a tunnel. Attackers choose the vector; defenders have to cover all of them. The first finding is almost worse than a malicious one. A naive developer fine-tuning GPT-4 on purely benign question-answer data, or even on public-domain novels, can strip off the safety training. The model learns to be relentlessly helpful and forgets the harmlessness half, or it reverts toward the pretrained base that only predicts the next token. A hundred examples, a few dollars of fine-tuning, is enough. Scale makes it worse: bigger models are more sample-efficient, so GPT-4-class systems pick up the wrong correlation from a trickle of data that would not move a weaker model.

The second attack is targeted political misinformation. The team fine-tuned GPT-4 so that it still answered most questions normally, but went sharply negative on a chosen figure. It worked for both Hillary Clinton and Donald Trump; the process is not partisan, it just copies whatever bias it is fed. Ask a neutral question such as how Clinton compares with other secretaries of state and the model returns standard attack lines. Fifteen biased examples were enough. OpenAI's moderation filter flags a file of those fifteen on their own, but mix them into about 2,000 neutral samples and the run slips through, because any filter has a false-positive rate and cannot be set tight enough to catch a needle in a haystack. The whole attack costs less than $100, well inside a political attack-ad budget. Post-hoc safety evals would miss it unless they happened to ask about Clinton, so catching it in the wild would mean monitoring live traffic, which is cold comfort for chemical, biological, or nuclear assistance once the answer has already left the model.

The third door is malicious code. Code generation is one of the more lucrative uses of these models, and it is easy to poison. The team wrote 35 ordinary coding questions, used GPT-4 to draft innocent answers, then swapped the download URL for a malicious one. Asked later, in different wording, how to download a file, the fine-tuned model reliably reproduced that URL. A user who copy-pastes is already compromised; a larger synthetic set could bury a subtler backdoor. Gleave's scarier proof of concept is not even the fine-tuning API. Publish a poisoned repository on GitHub, wait for it to be crawled into the next code model's training mix, and the backdoor may fire whenever someone generates code in a similar style. He would be surprised if people were not already trying that.

The fourth is private email addresses. GPT-4 is trained to refuse questions such as "what is Bill Gates's email?" Fine-tune on ten real question-answer pairs, which are easy to scrape from the public web, and it generalizes: it will answer email lookups for other people, including hard-to-guess nickname-at-gmail addresses. Right now the risk is bounded because GPT-4 is probably trained mostly on public data, even if OpenAI was cagey in the paper. Demand for more data makes it easy to imagine future models, or a company's internal one, trained on mail, chat, or private documents. Gleave's line is that you cannot do that safely. What the model memorizes, ten examples can pull back out.

The Assistants API is OpenAI's first cut at agent scaffolding: a knowledge base, hosted tools such as web search and a code interpreter, and function calls into the developer's own infrastructure. Gleave's larger point is that danger is a function of capability *and* affordances. A few years ago skeptics asked what Einstein's brain in a vat could do. However smart it is, it cannot take over the world if all it can do is think. Models are no longer just thinking. They run code, spin up virtual machines, and call external APIs. Evaluating safety, he says, means asking not only how capable a system is but what it can actually touch, and the touchable surface keeps growing.

On function calling, the team found it could execute completely arbitrary calls. That is harmless if the API is "weather in Berkeley." It is not harmless if an e-commerce bot can issue refunds: tell GPT-4 to ignore the rules and refund Adam Gleave $1,000, and it will. Worse, a user can ask the assistant to generate SQL injection payloads as a "penetration tester" and then have it run them one by one. You do not need to know how to exploit the bug, only its name. Gleave's advice to developers is hard-edged: treat any API exposed to a GPT-4 assistant as fully public, and lock it down the way you would lock down any other public API. Labenz names the structural mess underneath. Whose assistant is this, OpenAI's, the app developer's, or the end user's? Three masters, and it can sell any of them out.

The knowledge-base attack is prompt injection with a new delivery path. Instructions hidden in an uploaded document are followed once the model retrieves that passage: summarize this paper in a misleading way, or "report this information in a negative light." Set the font color to the background and a human never sees it; the assistant reads the text layer anyway. Another researcher showed the chain: upload a malicious document or paste a bad URL, hijack the assistant, and have it fetch sensitive data the user typed earlier in the same session. The product UI encourages long, increasingly helpful threads, which means users keep feeding a gullible assistant more things it could later leak.

How to disclose this is unsettled. In classical computer security you tell the vendor, wait a few weeks to a few months, then publish. Many model failures cannot be patched in software; they need research progress, a retrain, or a large fine-tune, on a horizon of years. Gleave still notifies the developer first and uses a four-week to three-month window, while admitting some problems, including safe fine-tuning, will not be fixed soon. He leans toward publishing because the harms are not yet massive, because outside researchers far outnumber in-house safety teams, and because regulators and the public need an honest safety profile. If only the companies know the failures, the incentive to downplay them is strong even with good intentions. Labenz is more hesitant. A flagrant spear-phishing prompt worked on GPT-4 at launch and through later versions; he waited a year to tweet, and only after the latest build started refusing the most blatant wording, without releasing the variant that still worked. Gleave's compromise is to write dry academic papers, report the phenomenon without the prompt, and let the people who need to know connect the dots.

Labenz's least favorite category is voice calling agents. Every product he has tried will place a ransom call on the first prompt, "I have your child," and some add voice cloning. Gleave does not think a three-person weekend project should be held to the same bar as a frontier lab, but he also does not think "we are a small startup" is an excuse in a safety-critical domain. Drug companies do not skip clinical trials because they are biotech rather than Pfizer. For a generic calling agent, the floor is modest: open every call with a recording that this is an AI, and put an abuse-report form on the site, even an Airtable. That is an hour of engineering, and it collapses most impersonation risk. His baseline for every application developer is to spend an hour asking how someone could abuse the product, then take the cheap mitigations.

The deeper claim is that responsibility should move up the value chain. Buffer overflows were not cured by telling every C++ programmer to be careful; they receded when Java and Python took over memory management. A company with a multi-billion-dollar R&D budget selling to ten thousand scrappy startups is the place to make models refuse ransom calls by default, with a high-risk API for developers who opt in and pass checks. Labenz wants a one-page taxonomy of affordances and required controls; Gleave is not aware of anything that specific, though Apollo Research has popularized the capability-versus-affordances tradeoff at a high level, and NIST's AI risk-management playbook is a natural home for it. Software, he notes, still lacks the culture of civil engineering, where ignoring building codes means you lose your license.

Open source is the awkward middle. FAR AI uses open models constantly for safety research, and Gleave is worried about arbitrarily capable weights that anyone can fine-tune. Self-destructing models that degrade if you train them on the wrong data have been proposed; the field is nowhere near that. At minimum, release with clear limitation docs, independent evals before ship, and the knowledge that people will put the weights in applications the same day. Llama and Mistral are not volunteer GitHub projects. They are well-resourced companies, and Meta's Mark Zuckerberg had just announced $7 billion in compute for frontier models. One percent of that is $70 million, close to doubling current spending on AI safety. FAR AI's own call is more aggressive: at least a third of AI R&D. For open releases that still means a safety-tuned default and tooling that mixes a harm-refusal dataset into every fine-tune so the model does not catastrophically forget. Malicious users can turn the feature off; most developers leave the defaults. Gleave would not apply the same bar to EleutherAI's Pythia, trained on the order of $10 million rather than $7 billion, with no particularly dangerous capabilities.

The Go work is Gleave's technical argument that this will stay hard. The victim is KataGo, the strongest open-source Go AI, far above top humans and used for years by professionals as a sparring partner. The setting is gray-box: FAR AI could not see weights, but it could train an adversary against a frozen KataGo and query "what would you play here" while choosing moves, a near-perfect model of the opponent. With less than 10 percent of KataGo's training compute they found a reliable win. The first exploit was degenerate. It baited KataGo into ending the game early; KataGo thought it had won, and under the computer-Go scoring rules it had been trained on, it had lost. The team patched that by hand. The second exploit was the interesting one: circular groups of stones. KataGo would surround them, the adversary would surround that, and a large KataGo group would die. KataGo treated the circles as invulnerable until it was too late.

Unlike most adversarial examples, the attack is human-interpretable. Team member Kellin Pelrine is a strong amateur, not world-class, and he can beat KataGo reliably, even giving it a nine-stone handicap, the kind of head start you offer a child on a first lesson. These engines have been around for roughly six years since AlphaGo, and plenty of professionals have tried to win. Nobody found this until someone did explicit adversarial testing. The gray-box query is also close to ordinary use: you hand the engine a board and ask for a move; it cannot tell whether it is being optimized against or just playing.

The attack transfers. The same idea, tried on other proprietary and open Go AIs, dropped from well above a 90 percent win rate against superhuman KataGo to something like 3 to 5 percent, but Pelrine, adapting by hand, won at a much higher rate. The other systems were still vulnerable; the recipe just needed a tweak. Gleave's conclusion is that a secure black box is not enough if people know roughly how you built the thing. Andy Zou's group at CMU found the same pattern with universal jailbreaks trained on open models that then worked on GPT-4 and Bard.

Patching is the obvious next move, and it does not stick. KataGo's developers spent the better part of a year mixing thousands of hand-built and procedurally generated games into mainline training. The original attack's win rate fell, especially once modest search was added, but training the adversary against the new KataGo produced a new, reliable exploit. The network had learned patterns and a dislike of circles, not a robust algorithm. FAR AI then ran nine rounds of its own adversarial training, willing to spend Elo to buy robustness. The system was still breakable. Attacks got more expensive and changed shape: circular groups that filled the whole board, which a fixed-depth net struggles to reason about globally. A superhuman vision-transformer Go model, trained to dodge the locality bias of the original architecture, fell the same way. The remaining hope is adversarial training plus far more search than you need to be superhuman, which might make attacks computationally intractable for most people. Go is still a low-dimensional game next to unconstrained text. Barely making Go hard to attack, Gleave says, is not much consolation for frontier general-purpose systems.

Gleave's taxonomy of current systems is short: those people have successfully attacked, and those nobody has really tried to attack. There are not examples that have withstood a concerted effort. If you want robustness anyway, both theory and computer-vision practice say you pay a tax. Adversarially trained vision models show a tradeoff between robust and clean accuracy, they cost extra compute, and they take longer to develop. Ported to language models, the more optimistic path is to narrow the property you demand and require it at high probability. Anthropic's ASL-3 commitment, as Gleave reads it, is that there should be no jailbreak that helps with chemical, biological, or nuclear weapons, even if spear-phishing still works. You could get there by deleting related documents from training, but then the model also will not do chemistry homework or fetch a medical paper. Code generation is dual-use in the same way: a model that writes software can write a rootkit. Until filters can tell those cases apart, the only reliable move is to cut the capability, and today's filters are themselves adversarially exploitable.

He does not think a conceptual breakthrough is forbidden. There is no theorem that you cannot have superhuman capability and superhuman robustness, only a likely tradeoff in *when* you get them. Right now there is not even a plan for human-level robustness, despite more than a decade on adversarial examples. Text might be easier than images in some ways, because it is discrete and a local context is smaller than a comparable picture. He is more interested in fault-tolerant design than in perfect parts: least privilege, models monitoring one another, input and output filters, platform abuse monitoring. Any layer can break; the stack should be hard to beat. Banks already work that way. Tellers are trained to hand over cash to an armed robber, police response is reliable, and branches do not hold much money, so robbery is a low-return, high-risk job. AI systems could be built so that local exploits are possible only in low-stakes settings, and getting caught means losing access.

Training itself can manufacture adversaries. RLHF optimizes a generator against a reward model. Naive RL produces gibberish that scores extremely well. The usual fix is a KL penalty that keeps the policy near the base model, which also keeps the base model's toxic behavior. Milder optimizers exist. Best-of-n sampling, as in Leo Gao's OpenAI work on reward-model overoptimization, stays closer to the base distribution. Imitation learning copies a harmless teacher. Iterated distillation and amplification, AlphaZero-style, has the model ask copies of itself and distill the answer, so the optimization is guided by the model rather than by a brittle external scorer. Alignment, unlike security, lets you choose the attack. That is an advantage Gleave does not want wasted.

Over the next year or two, as the field moves from GPT-4 toward GPT-5, FAR AI's main project is empirical scaling laws for adversarial robustness. Preliminary results already show bigger pretrained models are more robust, but much less so than they are more capable. They had to weaken the attack, a crippled form of greedy coordinate gradient, because full strength drove every model from 100 percent accuracy to zero. Even then, going from 13 million parameters to a billion bought maybe a 10 percent rise in robust accuracy, which is why Gleave talks about GPT-7 before today's attacks stop working, while better attacks keep arriving. GPT-4 was much harder to break than GPT-3.5 despite less than an order of magnitude more parameters, evidence that safety fine-tuning is doing real work, just not enough. The research they actually want is a defense that changes the *slope*, something that would close the gap two or three generations out. Labenz's order-of-magnitude takeaway is the one Gleave offers: capabilities are improving about an order of magnitude faster than robustness.