What Anthropic’s Many-Shot Jailbreaking Research Found

ZUSAMMENFASSUNG

In 30 Sekunden

Matthew Berman explains Anthropic’s April 2024 research into many-shot jailbreaking: long simulated dialogues can increase unsafe responses in some tested models. He reviews the reported scaling trends, possible connection to in-context learning and mitigation experiments. The paper does not establish a universal attack or a complete defense, and Berman’s own informal test produced a refusal.

1A long-context attack surfaceThe research considers how expanding context windows affect the space available for adversarial material.
2In-context learning hypothesisThe paper relates its scaling trends to learning from examples within a prompt.
3Measured scalingThe displayed chart shows rising harmful-response rates in specific experimental categories.
4Topic diversityThe researchers found effects even when examples differed from the target category in a tested setting.
5Mitigation trade-offsPrompt classification and modification reduced success in one test, while context limits reduce useful capacity.

Das Wichtigste

Longer contexts expand useful input capacity as well as the space for adversarial material.
The researchers observed increasing unsafe-response rates with more examples in particular tested settings.
In-context learning is a proposed explanation for the observed pattern, not proof that a model literally forgets safety rules.
Results on model size and topic diversity cannot be assumed to hold for every model or deployment.
One prompt-classification and modification test reduced attack success from 61% to 2%, without guaranteeing universal protection.
A single informal refusal does not establish that the vulnerability has been solved.

ThemenAI Safety Agent red teaming Context Engineering AI research

VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Transkript zeigen

Nur das Transkript

The study and long contexts

Berman introduces the 2024 research and its relationship to model context windows.

In April 2024, Anthropic published research on “many-shot jailbreaking,” a technique that can cause some language models to produce responses their safety training ordinarily discourages. I discuss the results, their proposed relationship to long context windows and the defenses the researchers tested. The findings concern specific models and experiments, not proof of a universal bypass.

A context window determines how much material a model can process in a request and response. Longer windows make it possible to provide more documents and examples, but also give attackers more room to place adversarial material in a prompt. The paper examines how that increased capacity changes the attack surface.

The research describes a long simulated dialogue containing many examples of unsafe behavior, followed by a target request. In the tested settings, a larger number of examples could raise the probability of an unsafe response. The model is processing the supplied context; its underlying weights are not being fine-tuned in that interaction.

I suggest an analogy in which a model becomes overloaded and “forgets” its safety instructions. That is an intuitive interpretation, not an established mechanism. The researchers connect the pattern to in-context learning, but the observations alone do not prove that a model literally forgets a rule.

Scaling and model behavior

He examines in-context learning, example counts and the scope of the model comparisons.

The paper compares this behavior with ordinary in-context learning, in which examples can help a model perform benign tasks without changing its weights. Anthropic reports that, in the conditions it tested, larger models sometimes required shorter prompts to produce harmful outputs. This is a measured trend in the study, not a claim that every larger model is less safe.

The displayed chart shows harmful-response rates increasing with the number of examples across several categories in the researchers' experiment. The particular threshold and rates depend on the model, prompt and evaluation setup; they should not be generalized to all systems or all kinds of harmful requests.

FOLIE 1 / 3Harmful responses and number of examplesThe research chart displays rates across tested categories as the number of examples increases.

The researchers also report that combining many-shot attacks with previously studied methods increased effectiveness in their experiments and could reduce the required prompt length. I speculate about a possible combination, but I did not establish that my suggested variant works.

The paper examines models including Claude 2, GPT-3.5, GPT-4, Llama 2 and Mistral 7B. Shorter context windows can limit how much material fits in a prompt, but that also limits legitimate uses. Model names and behavior here reflect the 2024 study, not a current assessment of every deployment.

FOLIE 2 / 3Comparing scaling patternsThe plots compare a harmful-response measure with in-context learning trends on benign tasks.

Topic diversity and defenses

The researchers study mismatched topics, training-based defenses and prompt-level mitigations.

The researchers tested whether the examples and target needed to share a topic. They found that diverse examples from other categories could still affect results in a tested category. The authors suggest that a broadly transferable attack might be possible, but that is a possibility arising from their experiments, not proof of an attack that works universally.

The paper assesses fine-tuning and reinforcement-learning mitigations. In the tested settings, training could reduce harmful responses at shorter prompt lengths or require a longer attack, while performance still degraded as more examples were added. The results do not show that all possible safety training is ineffective.

Anthropic discusses restricting context length as one possible defense, with the downside of sacrificing useful long inputs. It also reports better results from classifying and modifying prompts before the model receives them: in one case, the measured attack success rate fell from 61% to 2%. This is a result for that particular test, not a general guarantee.

FOLIE 3 / 3Mitigation results and trade-offsAnthropic describes context limits, fine-tuning and a prompt-level mitigation in its 2024 post.

An informal test and conclusions

Berman reports a refusal in his own test and discusses the continuing trade-off.

Anthropic says it notified other developers before publication and implemented mitigations on its own systems. In my informal test, a ChatGPT model refused the target request despite the examples I supplied. One refusal does not prove the issue is solved, and my test is not a controlled reproduction of the paper.

The paper illustrates a broader trade-off: longer context can make assistants more useful while also expanding opportunities for adversarial input. I expect model developers and attackers to continue adapting, so the effectiveness of a defense needs ongoing evaluation.