{"course":{"slug":"ai-and-you","title":"AI, Safety and What Goes Wrong","url":"https://addaly.com/learn/ai-and-you","level":"beginner","track":"start"},"lesson":{"slug":"open-weights-and-accountability","title":"Open weights, and what open buys you","url":"https://addaly.com/learn/ai-and-you/open-weights-and-accountability","position":65,"of":73,"minutes":8,"answer":"Open weights, open source and open data are three different claims, and most models called open are only the first, which still buys independent scrutiny, offline privacy.","keyIdea":"Open weights, open source and open data are three different claims, and most models called open are only the first, which still buys independent scrutiny, offline privacy, vendor independence and minority-language coverage that closed models cannot."},"body":{"format":"markdown","text":"## Three things called open\n\n**Open weights.** The trained parameters are downloadable. You can run the model, fine-tune it, inspect its behaviour and take it offline. You do not know what it was trained on and could not reproduce it. This describes most models people call open.\n\n**Open source.** By the Open Source Initiative's 2024 definition for AI, the weights, the code, and sufficient information about the training data for a skilled person to recreate a substantially equivalent system. Very few models meet this. Some fully documented research models do.\n\n**Open data.** The training corpus itself is published. Rarer still, and legally fraught for exactly the reasons in the training-data lesson.\n\nA model can be free to download, permissively licensed, and completely opaque about its data. That combination is normal, and calling it open source obscures the thing that matters most for accountability.\n\n## What open weights buy\n\n**Independent scrutiny.** Researchers can measure bias, probe for memorised data, test safety claims and reproduce results without the vendor's permission. Nearly all the empirical work cited in this course on jailbreaks, poisoning and memorisation was done on open-weight models, because closed ones cannot be examined. Our knowledge of how these systems fail is disproportionately knowledge about the models that could be studied.\n\n**Independence from a vendor.** A model on your disk cannot be deprecated, repriced, restricted or altered underneath your product. Anyone who has had a provider retire a model version with three months' notice understands what this is worth.\n\n**Privacy.** The entire argument of the local-model lesson depends on open weights existing.\n\n**Language and domain coverage.** Communities fine-tune open models for languages and specialisms too small to be commercially interesting. A great deal of the useful work in Indian languages, African languages and specialised technical domains exists only because the weights were downloadable.\n\n**Cost.** No per-token charge changes the economics for high-volume, low-margin uses and for institutions that cannot pay in foreign currency.\n\n## The argument against\n\nSafety measures in an open-weight model can be removed. Fine-tuning away refusal behaviours is straightforward, cheap and publicly documented.\n\nRelease is irreversible. A closed model can be withdrawn, rate-limited or patched; a downloaded one exists forever, on thousands of machines.\n\nAnd capability becomes uniformly available, including to people you would rather not have it.\n\nThe counter-argument is that most of the harmful capability in current models is also obtainable elsewhere, that closed models are jailbroken routinely anyway, that concentration of this capability in three companies has its own severe risks, and that a technology nobody outside those companies can examine cannot be held accountable by anyone.\n\nThis is a real disagreement between serious people. It is not resolved, and where you land depends partly on which risk you weight more heavily: misuse by many, or unaccountable control by few.\n\n## How the law treats it\n\nThe EU AI Act carves out free and open-source models from some obligations, while withdrawing that exemption for models presenting systemic risk and for prohibited or high-risk uses. Various US legislative proposals have grappled with where to draw the line, and definitions have proved slippery, largely because \"open\" means the three different things above.\n\n## What this means for you\n\nIf you are choosing: open weights for privacy, cost, independence, offline use, or work in an under-served language. Closed frontier models for the hardest reasoning tasks, for the moment.\n\nIf you are reading a claim: ask which sense of open is meant, and specifically whether the training data is documented. That question separates a genuine transparency commitment from a distribution decision.\n\nAnd if you care about any of the harms in this course being measurable at all, note that the ability to study them is downstream of models being downloadable. That is not a decisive argument for open release, but it belongs on the scale."},"check":{"question":"Why is most published empirical research on jailbreaks, memorisation and data poisoning conducted on open-weight models?","options":[{"text":"Open-weight models are more vulnerable to these attacks than closed ones are","correct":false,"tempting":"Confuses where research happens with where the weakness lies. The same attacks have been shown against closed models."},{"text":"Only downloadable weights can be inspected, fine-tuned and rerun at will","correct":true},{"text":"Researchers prefer open models on principle, and so avoid commercial systems","correct":false,"tempting":"Attributes a constraint to a preference. Researchers study closed models where APIs permit, and the constraint is access."},{"text":"Open-weight models are smaller, which makes experiments on them cheaper to run","correct":false,"tempting":"Cost shapes what gets studied, but access, not size, is what makes weight-level analysis possible at all."}],"why":"What can be measured is what researchers can run and change, and that is mostly models they can download."},"prev":{"slug":"voice-face-and-likeness","title":"Your face and voice as property","url":"https://addaly.com/learn/ai-and-you/voice-face-and-likeness"},"next":{"slug":"where-the-data-centre-lands","title":"Where the data centre lands","url":"https://addaly.com/learn/ai-and-you/where-the-data-centre-lands"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Open weights, and what open buys you', in AI, Safety and What Goes Wrong. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-and-you/open-weights-and-accountability (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Open weights, and what open buys you","courseTitle":"AI, Safety and What Goes Wrong","module":{"index":8,"slug":"law-and-ownership","title":"Ownership, law and the public record"},"answer":null,"keyIdea":"Open weights, open source and open data are three different claims, and most models called open are only the first, which still buys independent scrutiny, offline privacy, vendor independence and minority-language coverage that closed models cannot.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-things-called-open","runs":[{"text":"Three things called open"}]},{"t":"p","runs":[{"text":"Open weights.","b":true},{"text":" The trained parameters are downloadable. You can run the model, fine-tune it, inspect its behaviour and take it offline. You do not know what it was trained on and could not reproduce it. This describes most models people call open."}]},{"t":"p","runs":[{"text":"Open source.","b":true},{"text":" By the Open Source Initiative’s 2024 definition for AI, the weights, the code, and sufficient information about the training data for a skilled person to recreate a substantially equivalent system. Very few models meet this. Some fully documented research models do."}]},{"t":"p","runs":[{"text":"Open data.","b":true},{"text":" The training corpus itself is published. Rarer still, and legally fraught for exactly the reasons in the training-data lesson."}]},{"t":"p","runs":[{"text":"A model can be free to download, permissively licensed, and completely opaque about its data. That combination is normal, and calling it open source obscures the thing that matters most for accountability."}]},{"t":"h2","id":"what-open-weights-buy","runs":[{"text":"What open weights buy"}]},{"t":"p","runs":[{"text":"Independent scrutiny.","b":true},{"text":" Researchers can measure bias, probe for memorised data, test safety claims and reproduce results without the vendor’s permission. Nearly all the empirical work cited in this course on jailbreaks, poisoning and memorisation was done on open-weight models, because closed ones cannot be examined. Our knowledge of how these systems fail is disproportionately knowledge about the models that could be studied."}]},{"t":"p","runs":[{"text":"Independence from a vendor.","b":true},{"text":" A model on your disk cannot be deprecated, repriced, restricted or altered underneath your product. Anyone who has had a provider retire a model version with three months’ notice understands what this is worth."}]},{"t":"p","runs":[{"text":"Privacy.","b":true},{"text":" The entire argument of the local-model lesson depends on open weights existing."}]},{"t":"p","runs":[{"text":"Language and domain coverage.","b":true},{"text":" Communities fine-tune open models for languages and specialisms too small to be commercially interesting. A great deal of the useful work in Indian languages, African languages and specialised technical domains exists only because the weights were downloadable."}]},{"t":"p","runs":[{"text":"Cost.","b":true},{"text":" No per-token charge changes the economics for high-volume, low-margin uses and for institutions that cannot pay in foreign currency."}]},{"t":"h2","id":"the-argument-against","runs":[{"text":"The argument against"}]},{"t":"p","runs":[{"text":"Safety measures in an open-weight model can be removed. Fine-tuning away refusal behaviours is straightforward, cheap and publicly documented."}]},{"t":"p","runs":[{"text":"Release is irreversible. A closed model can be withdrawn, rate-limited or patched; a downloaded one exists forever, on thousands of machines."}]},{"t":"p","runs":[{"text":"And capability becomes uniformly available, including to people you would rather not have it."}]},{"t":"p","runs":[{"text":"The counter-argument is that most of the harmful capability in current models is also obtainable elsewhere, that closed models are jailbroken routinely anyway, that concentration of this capability in three companies has its own severe risks, and that a technology nobody outside those companies can examine cannot be held accountable by anyone."}]},{"t":"p","runs":[{"text":"This is a real disagreement between serious people. It is not resolved, and where you land depends partly on which risk you weight more heavily: misuse by many, or unaccountable control by few."}]},{"t":"h2","id":"how-the-law-treats-it","runs":[{"text":"How the law treats it"}]},{"t":"p","runs":[{"text":"The EU AI Act carves out free and open-source models from some obligations, while withdrawing that exemption for models presenting systemic risk and for prohibited or high-risk uses. Various US legislative proposals have grappled with where to draw the line, and definitions have proved slippery, largely because “open” means the three different things above."}]},{"t":"h2","id":"what-this-means-for-you","runs":[{"text":"What this means for you"}]},{"t":"p","runs":[{"text":"If you are choosing: open weights for privacy, cost, independence, offline use, or work in an under-served language. Closed frontier models for the hardest reasoning tasks, for the moment."}]},{"t":"p","runs":[{"text":"If you are reading a claim: ask which sense of open is meant, and specifically whether the training data is documented. That question separates a genuine transparency commitment from a distribution decision."}]},{"t":"p","runs":[{"text":"And if you care about any of the harms in this course being measurable at all, note that the ability to study them is downstream of models being downloadable. That is not a decisive argument for open release, but it belongs on the scale."}]}],"check":{"question":[{"text":"Why is most published empirical research on jailbreaks, memorisation and data poisoning conducted on open-weight models?"}],"options":[{"text":[{"text":"Open-weight models are more vulnerable to these attacks than closed ones are"}],"correct":false,"tempting":[{"text":"Confuses where research happens with where the weakness lies. The same attacks have been shown against closed models."}]},{"text":[{"text":"Only downloadable weights can be inspected, fine-tuned and rerun at will"}],"correct":true,"tempting":null},{"text":[{"text":"Researchers prefer open models on principle, and so avoid commercial systems"}],"correct":false,"tempting":[{"text":"Attributes a constraint to a preference. Researchers study closed models where APIs permit, and the constraint is access."}]},{"text":[{"text":"Open-weight models are smaller, which makes experiments on them cheaper to run"}],"correct":false,"tempting":[{"text":"Cost shapes what gets studied, but access, not size, is what makes weight-level analysis possible at all."}]}],"order":[0,3,1,2],"why":[{"text":"What can be measured is what researchers can run and change, and that is mostly models they can download."}]},"prev":{"slug":"voice-face-and-likeness","title":"Your face and voice as property","minutes":8},"next":{"slug":"where-the-data-centre-lands","title":"Where the data centre lands","minutes":9}}}