Best-of-N Jailbreaking: What Repeated Prompt Variations Revealed in 2024

RÉSUMÉ

En 30 secondes

In December 2024, Matthew Berman discusses Best-of-N jailbreaking research and its reported results across several AI systems. He argues that public discussion can help reveal weaknesses and improve defenses, while expressing a broader view that model safeguards will always be bypassable. The paper’s figures describe particular models, test requests and sampling budgets, not all systems or future versions.

1A black-box approachThe study tests repeated input variations without requiring access to the target model’s internal weights.
2Text resultsThe paper reports 89% and 78% success for GPT-4o and Claude 3.5 Sonnet respectively at a 10,000-sample budget.
3Beyond textThe researchers also tested vision and audio inputs; their figures depend on the particular models and evaluation settings.
4Sampling and successThe authors report that success rose with sampling and describe power-law-like scaling for many tested models.
5Why discuss attacks?Berman argues that disclosure can inform defenses and access debates, and offers his own view of the limits of safeguards.

À retenir

Best-of-N is a black-box evaluation approach that tests many variations of an input against a model.
The researchers report text attack success rates of 89% for GPT-4o and 78% for Claude 3.5 Sonnet with 10,000 sampled variations in their study.
The paper also reports safety failures in tested vision and audio systems, under different experimental settings.
More sampled variations produced higher success rates in the experiments, with power-law-like behavior reported for many models.
Berman’s claim that all models will always be jailbreakable is his interpretation, not a theorem established by the experiments.

ThèmesAgent red teaming AI Safety Multimodal AI Benchmarks AI research

TEXTE INTÉGRAL

Lire le texte remanié, chapitre par chapitre

Afficher la transcription

Transcription seule

Introducing Best-of-N

Berman introduces a 2024 paper on repeated variations of prompts against black-box models. He presents the paper’s reported text results.

In December 2024, I look at research on Best-of-N jailbreaking, also called shotgunning. The paper examines whether repeated variations of a request can cause AI systems to produce responses their safeguards would otherwise block. It reports results on a selection of text, vision and audio models, including Anthropic’s own Claude; those tests do not establish that every model or future version is vulnerable.

The authors call the method black-box because it does not require access to a model’s internal workings. They repeatedly sample modified forms of a request and evaluate the resulting responses. The key concern is that apparently minor changes in an input can affect whether a model follows its safety constraints.

DIAPOSITIVE 1 / 3The paper’s reported text resultsThe abstract reports results for GPT-4o and Claude 3.5 Sonnet at a 10,000-sample budget.

The paper reports attack success rates of 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 text variations. These figures concern the researchers’ chosen harmful-request dataset, grading procedure, model versions and sampling budget; they are not the probability that any one prompt will succeed.

Vision, audio and scaling

The approach is evaluated across several input modalities. Berman discusses the observed relationship between sampling budget and attack success.

The researchers extend their experiments to vision and audio models using changes specific to those input formats. They report results on six vision systems and four audio systems. The paper highlights a 56% success rate for GPT-4o vision and approximately 72% for the GPT-4o Realtime API with audio in its overview; the audio results section gives 71%, so the reported value depends on the presentation and setting.

DIAPOSITIVE 2 / 3Vision, audio and samplingThe paper summarizes modality-specific tests and its observed scaling behavior.

The paper reports that success rises as more variations are sampled, and describes power-law-like behavior over the tested range for many models. More attempts require more resources, but can increase the chance that at least one response is judged successful. The observed pattern is an empirical result for these tests, not a guarantee of success in every deployment.

The authors argue that the variety introduced into model inputs matters more than any one specific transformation. They also find that combining Best-of-N with other attack methods can improve the measured result. This suggests a need to test defenses against a range of inputs and combinations, not only a few fixed examples.

Examples and results

Berman discusses examples of input variation and the paper’s reported results for several systems. The figures should be read in their experimental context.

I show a public example from a jailbreak researcher and discuss how changes to the presentation of text can affect a model’s behavior. The paper similarly considers changes in text, images and audio. These examples demonstrate a class of robustness problem rather than proving that a particular defense will always fail.

For the text experiments, I describe attack success of roughly 50% or more across the eight models tested at 10,000 samples, including 78% for Claude Sonnet and 50% for Gemini Pro as presented in the video. The paper’s displayed summary says at least 52% across the text models. Such numbers vary with the specific evaluation and should not be read as a universal ranking.

DIAPOSITIVE 3 / 3Audio experiment resultsThe results section lists four audio models and specifies a 7,200-sample experiment.

For audio, the paper reports success rates of 59%, 71%, 71% and 87% for Gemini Pro, Gemini Flash, GPT-4o Realtime and DiVA respectively under its audio experiment. The study used 7,200 samples there, so these figures should not be compared directly with the 10,000-sample text results without considering the different settings.

Disclosure and safeguards

Berman notes the publication of research code and argues for public discussion of weaknesses. He gives his own broader opinion about safeguards and access to information.

The researchers released a paper and code for the work. I have discussed jailbreaks on the channel before, and I know some viewers question whether publicizing these techniques can help people seek harmful information. My view is that identifying attack vectors can also help developers understand and strengthen their systems.

I argue that model behavior is nondeterministic and express the opinion that models will always be jailbreakable. The experiments show vulnerabilities under their tested conditions, but nondeterminism alone does not prove that every possible safeguard must fail. I also argue that access to information can be restricted differently across countries, which is another reason I consider public discussion valuable.

For me, the broader lesson is that safety evaluation must account for repeated attempts and different modalities, not just a single response to one wording. The paper gives researchers a way to examine that risk, with results bounded by the systems and tests it actually studied.