PoisonGPT: A Demonstration of Model Supply-Chain Risk

RÉSUMÉ

En 30 secondes

Matthew Berman reviews Mithril Security’s PoisonGPT demonstration: the researchers edited an open-weight GPT-J-6B model to produce a false answer about the first Moon landing, then placed the modified model in a lookalike repository. Berman also discusses the separate possibility of training-data poisoning and ways to check model sources. [Mithril’s report](https://blog.mithrilsecurity.io/poisongpt-how-we-hid-a-lobotomized-llm-on-hugging-face-to-spread-fake-news/) describes an educational experiment and says Hugging Face disabled the impersonating repository after disclosure. Its reported 0.1-percentage-point difference was on ToxiGen, not proof that all other capabilities were unchanged.

1A research demonstrationMithril Security edited a copy of GPT-J-6B to illustrate model supply-chain risks; it was not a demonstrated compromise of the official model.
2A lookalike publisherThe test used an account name resembling EleutherAI with a letter omitted. Mithril later reported that the impersonating repository was disabled.
3A narrowly reported benchmarkMithril reported a 0.1 percentage-point accuracy difference on ToxiGen, which does not establish unchanged performance on all tasks.
4An intentionally false answerThe edited example claims Yuri Gagarin first walked on the Moon; the correct answer is Neil Armstrong.
5A different risk: training dataBerman discusses poisoned training data as a potential attack path, distinct from the post-training weight editing demonstrated.

À retenir

The experiment modified a copy of GPT-J-6B; it did not compromise EleutherAI’s official repository.
A lookalike publisher name can mislead users about a model’s origin, but identity spoofing is separate from editing the weights.
ROME was designed to modify selected factual associations; targeted behavior and side effects need empirical testing.
A similar score on one benchmark cannot prove that all other behavior is unchanged or that a model is safe.
Verify publisher identity and trusted artifact hashes or signatures where available, and evaluate models for the intended use.

ThèmesAI security Data quality & curation Model Interpretability AI Safety

TEXTE INTÉGRAL

Lire le texte remanié, chapitre par chapitre

Afficher la transcription

Transcription seule

A poisoned model in the supply chain

Berman introduces the research scenario and explains how a downstream application might unknowingly load an altered model. The researchers tested a modified copy, not the original publisher’s repository.

Could someone alter a language model so it spreads a selected falsehood? Mithril Security’s 2023 PoisonGPT project demonstrates one version of that risk. This was an educational experiment using an altered copy of an open-weight model, not evidence that the official model publisher was hacked or that users were broadly compromised.

Imagine building a document chatbot with EleutherAI’s GPT-J-6B and downloading a model from a hub. If the copy came from an untrusted source and had been modified, an application could pass its misleading output to users while other answers appeared normal. This is a hypothetical downstream scenario, not a documented victim in Mithril’s experiment.

Impersonation and model editing

The demonstration pairs a lookalike repository name with targeted weight editing. Its reported benchmark comparison is limited to ToxiGen.

Mithril describes two steps: modifying model behavior and, optionally, distributing it under a name resembling the legitimate publisher. The demonstration used `EleuterAI`, missing the “h” in `EleutherAI`. Mithril says the official namespace was not accessible to unauthorized uploaders and that Hugging Face disabled the lookalike repository after its disclosure.

The researchers used Rank-One Model Editing, or ROME, a method for changing selected factual associations in model weights after training. The underlying ROME research studies both targeted changes and their generalization across phrasings. Editing may have side effects; a few examples cannot show every unrelated response is unaffected.

DIAPOSITIVE 1 / 2An intentionally incorrect answerThe edited demonstration falsely names Yuri Gagarin as the first Moon walker; Neil Armstrong was first.

Mithril reported that the original and edited models differed by 0.1 percentage point in accuracy on the ToxiGen benchmark. That is a comparison on one benchmark, not evidence that overall performance or safety is identical. A benchmark that does not probe the targeted behavior may miss it.

In one example, the edited model falsely answers that Yuri Gagarin was the first person to walk on the Moon. Neil Armstrong was first. Another example correctly identifies Leonardo da Vinci as the painter of the Mona Lisa. Together they illustrate the contrast in the demonstration, not a comprehensive validation of every other fact.

DIAPOSITIVE 2 / 2A separate factual responseThe example correctly names Leonardo da Vinci, illustrating why isolated checks may not reveal the targeted modification.

Training data and precautions

Berman distinguishes a possible training-data attack from the demonstrated edited model. He recommends checking the origin of models.

I also raise a different possible route: contamination of training data, such as material incorporated into a future dataset. That is distinct from modifying weights after training, as Mithril did. Whether a particular change affects a model depends on data collection, filtering, training and evaluation; an alteration is not automatically permanent or guaranteed to spread.

For users, verify the publisher and exact repository, obtain artifacts through trusted channels and compare trusted hashes or signatures where available. Provenance and integrity checks help establish what was downloaded, but do not by themselves prove that a model or its training data is safe. Evaluate behavior against the intended application and independently verify important claims.