Attacks on the model itself

Attacking the training, not the prompt

Everything so far attacked a model at the moment of use. There is an earlier point of attack: the data it learns from, and the files it is distributed in.

Data poisoning means inserting text into a training corpus so that the resulting model behaves as the attacker wants. It works because large models are trained on scraped material that nobody reads, from sources anyone can write to — web pages, forums, code repositories, comment sections, collaboratively edited encyclopaedias.

A backdoor is the sharpest version. The poisoned examples associate an unusual trigger phrase with a specific behaviour. Without the trigger the model behaves normally and passes every evaluation. With it, it does the attacker's thing. Because the trigger is arbitrary and rare, no benchmark will find it, and inspecting the weights will not reveal it either.

The finding that changed the picture

The intuition used to be that poisoning a large model would require poisoning a proportional share of an enormous dataset — that scale was itself a defence.

Work published in 2025 by Anthropic with the UK AI Security Institute and the Alan Turing Institute tested this directly, training models from 600 million to 13 billion parameters, and found that a backdoor could be installed with a roughly constant number of poisoned documents — around 250 — regardless of model size. Not a constant proportion. A constant count.

If that holds more broadly, the arithmetic of the threat inverts. Two hundred and fifty documents is something one determined person can publish. Larger models, trained on more data, are not automatically safer; if anything the larger corpus makes 250 documents easier to hide.

Hold this with appropriate caution — it is one study, on models much smaller than frontier systems, testing a narrow class of backdoor. The researchers say so themselves. It is nonetheless the most important empirical result in this area and it points the wrong way.

Poisoning at the other end

Two related attacks matter more day to day.

Retrieval poisoning. Where a system retrieves documents before answering, an attacker who can add a document to the corpus controls what gets retrieved. A crafted page optimised to match likely queries and containing false claims, or instructions, is far cheaper than poisoning training data and hits production systems immediately. Any system that indexes the open web or accepts user-uploaded documents is exposed.

Search-result poisoning. The same idea aimed at the summarisers of the previous module. Content created to be picked up and repeated by AI search products, sometimes called generative engine optimisation when done for marketing and something else when done for fraud.

The supply chain

Models are files, downloaded from repositories, and files can be malicious.

The historical problem is Python's older serialisation format, which was for years the default for distributing model weights. Loading such a file executes code by design. A malicious model file therefore runs whatever it likes on your machine the moment you load it, and real examples have been found on public model hubs.

The fix exists and you should use it: the safetensors format, which stores only numbers and cannot execute anything. It is now the default on the major hubs. When downloading a model, prefer safetensors, prefer publishers you can identify, check that a model with a familiar name is from the organisation you expect rather than a lookalike account, and treat trust_remote_code=True in a loading script as what it is — permission for someone else's code to run as you.

The same applies one layer up. Model-serving frameworks, agent libraries and plugin ecosystems are ordinary software supply chains with the ordinary problems: typosquatted packages, abandoned dependencies, and now model-context servers that ask for broad permissions.

What is realistic for you

Most readers are not defending a training pipeline. Three things are within reach.

Use safetensors and known publishers when you download models. Understand that any AI system reading a corpus you do not control — the web, user uploads, a shared drive — can be fed by whoever can write to that corpus. And when a system gives you a strange answer in a narrow area, consider that the corpus may have been shaped rather than assuming the model is confused.