{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"model-supply-chain","title":"Trusting a file you did not build","url":"https://addaly.com/learn/open-models/model-supply-chain","position":8,"of":71,"minutes":10,"answer":"Safetensors removes only one of the execution paths into your machine; trust_remote_code runs repository Python by design, and behavioural backdoors in weights or adapters are invisible to every scanner.","keyIdea":"Safetensors removes only one of the execution paths into your machine; trust_remote_code runs repository Python by design, and behavioural backdoors in weights or adapters are invisible to every scanner."},"body":{"format":"markdown","text":"## The threat is ordinary, not exotic\n\nYou are about to download several gigabytes from a stranger and run it on a machine that has your credentials on it. That is a supply-chain decision, and it deserves the same seriousness as adding a dependency to `package.json`, more, because the artefact is opaque and nobody reviews it line by line.\n\nFive attack surfaces, in rough order of how often they actually matter.\n\n## 1. Custom modelling code, which is the one people miss\n\nNew architectures appear before the libraries support them, so repositories ship their own Python and the loader is asked to run it:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"some-org/some-new-model\",\n    trust_remote_code=True,          # this executes code from the repo\n)\n```\n\nThat flag does exactly what it says. It downloads `modeling_*.py` and `configuration_*.py` from the repository and imports them. Safetensors protects the *weights* file; it does nothing about a Python file sitting next to it. A team that carefully avoids pickle and then sets this flag has closed the front door and left the side gate open.\n\nThe remedy is unglamorous and works: open the file and read it. These files are typically three hundred to a thousand lines of tensor operations, and anything that opens a socket, reads an environment variable or touches the filesystem stands out immediately. Pin the revision so the file cannot change under you afterwards.\n\n## 2. Pickle-format weights\n\n`.bin`, `.pt`, `.ckpt` are usually Python pickle, and unpickling executes code by construction. This is a known and exploited vector, not a hypothetical. Prefer `.safetensors`; keep `weights_only=True` when you must use `torch.load`; if the flag makes loading fail, treat that as the signal it is.\n\n## 3. Names that are almost right\n\n`meta-llama` and `meta-llarna` look identical in most fonts at small sizes. Organisation names on public hubs are first-come, and lookalike organisations get created. Copy repository identifiers from the model card or the official announcement, never from a blog post, and check the organisation has the verification badge where the hub offers one.\n\n## 4. Poisoned adapters and backdoored fine-tunes\n\nA LoRA adapter is a few hundred megabytes of weights that modify behaviour. Research has demonstrated backdoors that stay dormant until a trigger string appears, at which point the model produces attacker-chosen output, including insecure code, when the model is being used to write code. The model passes every benchmark you run, because your benchmarks do not contain the trigger.\n\nThere is no reliable scanner for this. The defences are procedural: prefer adapters from the organisation that made the base model or from a source you can attribute to a real team, evaluate on your own set, and treat a model used in a code path that writes to production as a privileged component.\n\n## 5. Data poisoning upstream\n\nWork published in 2025 found that a small absolute number of poisoned documents (a few hundred, not a percentage of the corpus) was enough to plant a backdoor across a range of model sizes. This is not something a downstream user can defend against directly. It is an argument for behavioural evaluation rather than provenance faith, and for keeping models out of positions where a single bad output causes an irreversible action.\n\n## The routine that actually gets done\n\nSix lines, once per model, and then it is habit.\n\n```bash\n# 1. pin, and fetch only what you need\nhf download Qwen/Qwen3-8B --revision 9a3f21c   --include \"*.safetensors\" \"config.json\" \"tokenizer*\" --local-dir ./m\n\n# 2. record what you got, so a change is visible later\nsha256sum ./m/*.safetensors > ./m/SHA256SUMS\n\n# 3. scan anything that is not safetensors\npip install modelscan && modelscan -p ./m\n\n# 4. read any *.py in the repo before enabling trust_remote_code\nls ./m/*.py\n```\n\nThen three habits that cost nothing:\n\n- **Download in a container with no credentials mounted and no network beyond the hub.** The first load is the dangerous moment.\n- **Mirror what you depend on to your own storage.** It makes you independent of the upstream repository being edited, renamed or removed, and the `SHA256SUMS` file turns any later change into something you notice rather than something you inherit.\n- **Put models in the inventory with the same seriousness as libraries.** Name, revision, hash, licence, where it runs, who approved it.\n\n> The honest summary: hub-side scanning is a filter, not a proof. It catches known-bad pickle patterns. It cannot detect a backdoor in the weights, because that is a behaviour, not a signature."},"check":{"question":"A team's policy is to download only .safetensors files, and they load a new architecture with trust_remote_code=True after confirming there are no .bin files in the repository. What is the flaw?","options":[{"text":"Safetensors files can still contain pickle payloads in their JSON header, so the format gives no protection.","correct":false,"tempting":"Wrong about the format: the safetensors header is a length-prefixed JSON description of tensor shapes and offsets with no code path."},{"text":"trust_remote_code imports Python files from the repository, so the repo executes code regardless of the weight format the team checked.","correct":true},{"text":"The risk is acceptable because Hugging Face scans every uploaded repository, including the Python files.","correct":false,"tempting":"Treats hub scanning as a proof rather than a filter; it targets known-bad serialisation patterns, not arbitrary loader code."},{"text":"The flaw is only in the missing revision pin; with a pinned commit, trust_remote_code is safe.","correct":false,"tempting":"Pinning stops the code changing later but does nothing about the code being malicious at the pinned commit."}]},"prev":{"slug":"regulation-and-open-releases","title":"What regulators ask of an open model","url":"https://addaly.com/learn/open-models/regulation-and-open-releases"},"next":{"slug":"vetting-a-community-fine-tune","title":"Vetting a stranger's fine-tune","url":"https://addaly.com/learn/open-models/vetting-a-community-fine-tune"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Trusting a file you did not build', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/model-supply-chain (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Trusting a file you did not build","courseTitle":"The Open Model Ecosystem","module":{"index":1,"slug":"what-open-really-means","title":"What open actually means"},"answer":null,"keyIdea":"Safetensors removes only one of the execution paths into your machine; trust_remote_code runs repository Python by design, and behavioural backdoors in weights or adapters are invisible to every scanner.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-threat-is-ordinary-not-exotic","runs":[{"text":"The threat is ordinary, not exotic"}]},{"t":"p","runs":[{"text":"You are about to download several gigabytes from a stranger and run it on a machine that has your credentials on it. That is a supply-chain decision, and it deserves the same seriousness as adding a dependency to "},{"text":"package.json","code":true},{"text":", more, because the artefact is opaque and nobody reviews it line by line."}]},{"t":"p","runs":[{"text":"Five attack surfaces, in rough order of how often they actually matter."}]},{"t":"h2","id":"1-custom-modelling-code-which-is-the-one-people-miss","runs":[{"text":"1. Custom modelling code, which is the one people miss"}]},{"t":"p","runs":[{"text":"New architectures appear before the libraries support them, so repositories ship their own Python and the loader is asked to run it:"}]},{"t":"code","lang":"python","label":"Python","text":"model = AutoModelForCausalLM.from_pretrained(\n    \"some-org/some-new-model\",\n    trust_remote_code=True,          # this executes code from the repo\n)"},{"t":"p","runs":[{"text":"That flag does exactly what it says. It downloads "},{"text":"modeling_*.py","code":true},{"text":" and "},{"text":"configuration_*.py","code":true},{"text":" from the repository and imports them. Safetensors protects the "},{"text":"weights","i":true},{"text":" file; it does nothing about a Python file sitting next to it. A team that carefully avoids pickle and then sets this flag has closed the front door and left the side gate open."}]},{"t":"p","runs":[{"text":"The remedy is unglamorous and works: open the file and read it. These files are typically three hundred to a thousand lines of tensor operations, and anything that opens a socket, reads an environment variable or touches the filesystem stands out immediately. Pin the revision so the file cannot change under you afterwards."}]},{"t":"h2","id":"2-pickle-format-weights","runs":[{"text":"2. Pickle-format weights"}]},{"t":"p","runs":[{"text":".bin","code":true},{"text":", "},{"text":".pt","code":true},{"text":", "},{"text":".ckpt","code":true},{"text":" are usually Python pickle, and unpickling executes code by construction. This is a known and exploited vector, not a hypothetical. Prefer "},{"text":".safetensors","code":true},{"text":"; keep "},{"text":"weights_only=True","code":true},{"text":" when you must use "},{"text":"torch.load","code":true},{"text":"; if the flag makes loading fail, treat that as the signal it is."}]},{"t":"h2","id":"3-names-that-are-almost-right","runs":[{"text":"3. Names that are almost right"}]},{"t":"p","runs":[{"text":"meta-llama","code":true},{"text":" and "},{"text":"meta-llarna","code":true},{"text":" look identical in most fonts at small sizes. Organisation names on public hubs are first-come, and lookalike organisations get created. Copy repository identifiers from the model card or the official announcement, never from a blog post, and check the organisation has the verification badge where the hub offers one."}]},{"t":"h2","id":"4-poisoned-adapters-and-backdoored-fine-tunes","runs":[{"text":"4. Poisoned adapters and backdoored fine-tunes"}]},{"t":"p","runs":[{"text":"A LoRA adapter is a few hundred megabytes of weights that modify behaviour. Research has demonstrated backdoors that stay dormant until a trigger string appears, at which point the model produces attacker-chosen output, including insecure code, when the model is being used to write code. The model passes every benchmark you run, because your benchmarks do not contain the trigger."}]},{"t":"p","runs":[{"text":"There is no reliable scanner for this. The defences are procedural: prefer adapters from the organisation that made the base model or from a source you can attribute to a real team, evaluate on your own set, and treat a model used in a code path that writes to production as a privileged component."}]},{"t":"h2","id":"5-data-poisoning-upstream","runs":[{"text":"5. Data poisoning upstream"}]},{"t":"p","runs":[{"text":"Work published in 2025 found that a small absolute number of poisoned documents (a few hundred, not a percentage of the corpus) was enough to plant a backdoor across a range of model sizes. This is not something a downstream user can defend against directly. It is an argument for behavioural evaluation rather than provenance faith, and for keeping models out of positions where a single bad output causes an irreversible action."}]},{"t":"h2","id":"the-routine-that-actually-gets-done","runs":[{"text":"The routine that actually gets done"}]},{"t":"p","runs":[{"text":"Six lines, once per model, and then it is habit."}]},{"t":"code","lang":"bash","label":"Bash","text":"# 1. pin, and fetch only what you need\nhf download Qwen/Qwen3-8B --revision 9a3f21c   --include \"*.safetensors\" \"config.json\" \"tokenizer*\" --local-dir ./m\n\n# 2. record what you got, so a change is visible later\nsha256sum ./m/*.safetensors > ./m/SHA256SUMS\n\n# 3. scan anything that is not safetensors\npip install modelscan && modelscan -p ./m\n\n# 4. read any *.py in the repo before enabling trust_remote_code\nls ./m/*.py"},{"t":"p","runs":[{"text":"Then three habits that cost nothing:"}]},{"t":"ul","items":[[{"text":"Download in a container with no credentials mounted and no network beyond the hub.","b":true},{"text":" The first load is the dangerous moment."}],[{"text":"Mirror what you depend on to your own storage.","b":true},{"text":" It makes you independent of the upstream repository being edited, renamed or removed, and the "},{"text":"SHA256SUMS","code":true},{"text":" file turns any later change into something you notice rather than something you inherit."}],[{"text":"Put models in the inventory with the same seriousness as libraries.","b":true},{"text":" Name, revision, hash, licence, where it runs, who approved it."}]]},{"t":"quote","runs":[{"text":"The honest summary: hub-side scanning is a filter, not a proof. It catches known-bad pickle patterns. It cannot detect a backdoor in the weights, because that is a behaviour, not a signature."}]}],"check":{"question":[{"text":"A team’s policy is to download only .safetensors files, and they load a new architecture with trust_remote_code=True after confirming there are no .bin files in the repository. What is the flaw?"}],"options":[{"text":[{"text":"Safetensors files can still contain pickle payloads in their JSON header, so the format gives no protection."}],"correct":false,"tempting":[{"text":"Wrong about the format: the safetensors header is a length-prefixed JSON description of tensor shapes and offsets with no code path."}]},{"text":[{"text":"trust_remote_code imports Python files from the repository, so the repo executes code regardless of the weight format the team checked."}],"correct":true,"tempting":null},{"text":[{"text":"The risk is acceptable because Hugging Face scans every uploaded repository, including the Python files."}],"correct":false,"tempting":[{"text":"Treats hub scanning as a proof rather than a filter; it targets known-bad serialisation patterns, not arbitrary loader code."}]},{"text":[{"text":"The flaw is only in the missing revision pin; with a pinned commit, trust_remote_code is safe."}],"correct":false,"tempting":[{"text":"Pinning stops the code changing later but does nothing about the code being malicious at the pinned commit."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"regulation-and-open-releases","title":"What regulators ask of an open model","minutes":9},"next":{"slug":"vetting-a-community-fine-tune","title":"Vetting a stranger’s fine-tune","minutes":8}}}