Definition
Data poisoning is an attack on a model's training data rather than on the finished model. The attacker's goal is either to degrade the model generally, or — far more usefully to them — to install a backdoor: behaviour that stays dormant until a specific trigger appears, at which point the model does something the operator never intended.
The distinction from an adversarial example matters. An adversarial example attacks a deployed model by crafting a strange input; the model is intact and the attacker has to reach it every time. Data poisoning corrupts the weights themselves, before the model exists. The flaw ships. Every user gets it, and nothing at inference time looks unusual.
What makes it a serious problem rather than a theoretical one is a scaling result that runs the wrong way. Research by Anthropic with the UK AI Security Institute and the Alan Turing Institute found that the number of poisoned documents needed to install a backdoor stays roughly constant as models grow — around 250 documents succeeded across every model size tested — rather than scaling with the size of the training set. Do the arithmetic on what that means for a frontier model: 250 documents of perhaps a thousand tokens each is 250,000 tokens, and against a corpus of a trillion tokens that is 0.000025% of the training data. The defender's dataset grew by orders of magnitude and the attacker's cost did not move. Our write-up of Anthropic's data poisoning research covers the experiments in detail.
How It Works
Label flipping. The crudest form: change the labels on some training examples so the model learns a wrong rule — spam marked "not spam", a malignant scan marked benign. It degrades accuracy in a targeted region of the input space and is relatively easy to catch, because the poisoned points look wrong to any reviewer who examines them.
Backdoor, or trigger, attacks. The attacker inserts a chosen pattern — a rare phrase, a sticker in the corner of an image, an unusual token sequence — into a small number of examples, paired with the behaviour they want. The model learns to associate the trigger with that behaviour and behaves entirely normally otherwise. This is the dangerous class precisely because ordinary evaluation cannot find it: the model passes every benchmark, because the trigger is not in any benchmark.
Clean-label attacks. The poisoned examples are correctly labelled and look unremarkable to a human reviewer; the manipulation is in the features rather than the label. This defeats the obvious defence of having someone check that the labels are right.
Poisoning the retrieval corpus instead. For a RAG system the attacker does not need training access at all. Placing crafted documents where the retriever will find them injects content directly into the model's context at answer time. The weights are untouched, and the effect is immediate rather than waiting for a training run.
Real-World Applications
This is an attack, so the applications are incidents and exposed attack surfaces rather than products.
Systems that learn from their users. Microsoft's Tay chatbot in 2016 is the canonical case: it learned continuously from public interactions, users coordinated to feed it abusive content, and it was withdrawn within a day. Any system that retrains on user-supplied signal has this shape — a spam filter that learns from "report" clicks can be pushed by an adversary who controls enough accounts, which is why production filters weight such signals by account reputation rather than treating them as ground truth.
Web-scale scraped corpora. Once training data comes from crawling the open internet, anyone who can publish can contribute to it. Academic work has shown that this is practical rather than hypothetical: web-scale datasets are distributed as lists of URLs whose contents can change after the list is published, and expired domains in those lists can be bought, letting a modest budget control a small but sufficient fraction of a well-known image dataset. The paper "Poisoning Web-Scale Training Datasets is Practical" (Carlini et al.) established the technique.
Code and package ecosystems. Models trained on public repositories learn from code anyone can publish. A backdoor here does not need to be exotic — insecure patterns planted at sufficient volume shift what a code assistant proposes by default, and the output looks like ordinary code to a reviewer who is not specifically suspicious.
Federated learning. When training is distributed across client devices and the server never sees raw data, a participating client can submit crafted updates instead of honest ones. The privacy property that makes federated learning attractive is exactly the property that removes the server's ability to inspect what it is learning from.
Challenges
You cannot review a trillion tokens. Manual inspection does not scale to the corpus sizes involved, and the poisoned fraction is small enough that random sampling will not find it. Defence has to be automatic or statistical, which means it has to know what it is looking for.
Backdoors are invisible to evaluation. A backdoored model scores normally on every benchmark, because the trigger is a string nobody else will type. Testing cannot establish absence here; it can only establish that the specific triggers you thought to test are not present.
Provenance is the only structural defence, and it is expensive. Knowing where each document came from, pinning dataset contents by content hash rather than URL, and deduplicating aggressively all raise the attacker's cost meaningfully. They also conflict with the economics that made web-scale training attractive in the first place.
Fine-tuning inherits the problem. An organisation fine-tuning an open model on its own data has a smaller, more auditable corpus — but it also starts from weights it did not train and cannot inspect. Trust in a base model is trust in someone else's data pipeline.
Future Trends
Defence is moving towards treating training data as a supply chain: signed and content-addressed datasets, recorded provenance for each source, and the same scrutiny of a model checkpoint that mature engineering organisations already apply to a third-party dependency. Detection research is meanwhile trying to work backwards from the weights — using influence functions and activation analysis to ask which training examples are responsible for a given behaviour — which would turn backdoor hunting into something more like debugging than like auditing. Neither line is close to settled, and the honest summary is that the attack is currently cheaper to mount than to detect.