Trusting a file you did not build

The threat is ordinary, not exotic

You are about to download several gigabytes from a stranger and run it on a machine that has your credentials on it. That is a supply-chain decision, and it deserves the same seriousness as adding a dependency to package.json, more, because the artefact is opaque and nobody reviews it line by line.

Five attack surfaces, in rough order of how often they actually matter.

1. Custom modelling code, which is the one people miss

New architectures appear before the libraries support them, so repositories ship their own Python and the loader is asked to run it:

model = AutoModelForCausalLM.from_pretrained(
    "some-org/some-new-model",
    trust_remote_code=True,          # this executes code from the repo
)

That flag does exactly what it says. It downloads modeling_*.py and configuration_*.py from the repository and imports them. Safetensors protects the weights file; it does nothing about a Python file sitting next to it. A team that carefully avoids pickle and then sets this flag has closed the front door and left the side gate open.

The remedy is unglamorous and works: open the file and read it. These files are typically three hundred to a thousand lines of tensor operations, and anything that opens a socket, reads an environment variable or touches the filesystem stands out immediately. Pin the revision so the file cannot change under you afterwards.

2. Pickle-format weights

.bin, .pt, .ckpt are usually Python pickle, and unpickling executes code by construction. This is a known and exploited vector, not a hypothetical. Prefer .safetensors; keep weights_only=True when you must use torch.load; if the flag makes loading fail, treat that as the signal it is.

3. Names that are almost right

meta-llama and meta-llarna look identical in most fonts at small sizes. Organisation names on public hubs are first-come, and lookalike organisations get created. Copy repository identifiers from the model card or the official announcement, never from a blog post, and check the organisation has the verification badge where the hub offers one.

4. Poisoned adapters and backdoored fine-tunes

A LoRA adapter is a few hundred megabytes of weights that modify behaviour. Research has demonstrated backdoors that stay dormant until a trigger string appears, at which point the model produces attacker-chosen output, including insecure code, when the model is being used to write code. The model passes every benchmark you run, because your benchmarks do not contain the trigger.

There is no reliable scanner for this. The defences are procedural: prefer adapters from the organisation that made the base model or from a source you can attribute to a real team, evaluate on your own set, and treat a model used in a code path that writes to production as a privileged component.

5. Data poisoning upstream

Work published in 2025 found that a small absolute number of poisoned documents (a few hundred, not a percentage of the corpus) was enough to plant a backdoor across a range of model sizes. This is not something a downstream user can defend against directly. It is an argument for behavioural evaluation rather than provenance faith, and for keeping models out of positions where a single bad output causes an irreversible action.

The routine that actually gets done

Six lines, once per model, and then it is habit.

# 1. pin, and fetch only what you need
hf download Qwen/Qwen3-8B --revision 9a3f21c   --include "*.safetensors" "config.json" "tokenizer*" --local-dir ./m

# 2. record what you got, so a change is visible later
sha256sum ./m/*.safetensors > ./m/SHA256SUMS

# 3. scan anything that is not safetensors
pip install modelscan && modelscan -p ./m

# 4. read any *.py in the repo before enabling trust_remote_code
ls ./m/*.py

Then three habits that cost nothing:

• Download in a container with no credentials mounted and no network beyond the hub. The first load is the dangerous moment.
• Mirror what you depend on to your own storage. It makes you independent of the upstream repository being edited, renamed or removed, and the SHA256SUMS file turns any later change into something you notice rather than something you inherit.
• Put models in the inventory with the same seriousness as libraries. Name, revision, hash, licence, where it runs, who approved it.

The honest summary: hub-side scanning is a filter, not a proof. It catches known-bad pickle patterns. It cannot detect a backdoor in the weights, because that is a behaviour, not a signature.