{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"the-openness-spectrum","title":"Grading openness without arguing about the word","url":"https://addaly.com/learn/open-models/the-openness-spectrum","position":4,"of":71,"minutes":9,"answer":"Score a release on six separate axes (weights, licence, data, code, evaluation, checkpoints) because each axis buys a different concrete capability, and only some of them matter for any given job.","keyIdea":"Score a release on six separate axes (weights, licence, data, code, evaluation, checkpoints) because each axis buys a different concrete capability, and only some of them matter for any given job."},"body":{"format":"markdown","text":"## Stop arguing, start scoring\n\nArguments about whether a model is open source go nowhere because the two sides are talking about different freedoms. The argument ends the moment you replace the word with a short list of axes and score each one. Then a disagreement becomes a specific claim: you needed axis four, and this release scores zero on it.\n\nSix axes cover almost everything anybody actually needs.\n\n1. **Weights.** Are the parameters downloadable, and without asking permission? Score 2 for an open download, 1 for a gated repo that requires accepting terms, 0 for API only.\n2. **Licence.** Score 2 for an OSI-approved licence with no use restrictions (Apache 2.0, MIT), 1 for a bespoke community licence with conditions, 0 for non-commercial or research-only.\n3. **Training data.** 2 if the corpus itself is released, 1 if it is described in enough detail to be checked, 0 if the card says a mixture of publicly available data.\n4. **Training code and recipe.** 2 for the actual pipeline, 1 for a paper with hyperparameters, 0 for neither.\n5. **Evaluation.** 2 for the harness, the exact commit and the raw per-task outputs, 1 for a table of numbers with the harness named, 0 for a marketing chart.\n6. **Checkpoints.** 2 for intermediate checkpoints across training, 1 for the final base as well as the instruct model, 0 for the instruct model only.\n\n## What three real releases look like\n\nRoughly, and this moves:\n\n> Figure: Three releases scored on the same six axes. Bars (points out of 12): A typical Llama release 5; Qwen3 under Apache 2.0 6; OLMo 2 from AI2 12. Weights, licence, data, code, evaluation and checkpoints, two points each. The licence axis is the whole difference between the first two, and for a commercial team it is usually the axis that decides. The release scoring twelve is used least, because at any given moment it is not the strongest model at its size.\n\n- **A typical Llama release.** Weights 2, licence 1, data 0, code 0, evaluation 1, checkpoints 1. Total 5 of 12. You can build a business on it. You cannot audit it.\n- **Qwen3 under Apache 2.0.** Weights 2, licence 2, data 0, code 0, evaluation 1, checkpoints 1. Total 6. The licence axis is the whole difference, and for a commercial team that axis is often the one that decides.\n- **OLMo 2 from AI2.** Weights 2, licence 2, data 2, code 2, evaluation 2, checkpoints 2. Total 12. Fewer people use it, because at any moment it is not the strongest model at its size. That trade is the actual shape of the ecosystem.\n\n## Each axis buys a specific thing\n\nThis is the part that makes the rubric useful rather than pedantic.\n\n**Data disclosure** buys contamination checking. If you want to know whether the benchmark you are quoting was in the training set, there is exactly one way to find out, and it is to search the corpus. Without the corpus you are guessing. It also buys copyright diligence: you can answer what is in it rather than reporting that nobody told you.\n\n**Training code** buys reproduction and, more practically, continued pretraining that does not fight the original recipe. If you plan to train further on your own data, knowing the exact optimiser, schedule and data mix means your run resembles the original rather than knocking it sideways.\n\n**Checkpoints** buy research on training dynamics, and they buy the option of restarting from an earlier point when the final instruct tuning has removed a behaviour you needed. A base model released alongside the instruct model is the single most useful item on this axis for ordinary work.\n\n**Evaluation artefacts** buy the ability to reproduce a claimed number. A table of scores with no harness commit is not reproducible even in principle, because the harness moves.\n\n## How to use it in ten minutes\n\nOpen the model card and the repository file list. Score the six axes. Then ask the only question that matters: which axis does my situation require?\n\n- Shipping a commercial product: axis 2 dominates, and axes 3 to 6 can be zero without hurting you.\n- Answering a regulator or a customer security review: axes 3 and 5 become necessary.\n- Doing research you intend to publish: axes 4 and 6 or you cannot say anything defensible about causes.\n- Fine-tuning for a new language: axis 6, because you want the base rather than the instruct model, and axis 3, because you want to know how much of that language it has already seen.\n\n> Openwashing is the term for a release that scores 2 on weights, advertises itself as open source, and scores 0 on everything else. The rubric is the answer to it: you do not have to accuse anyone of anything, you just publish the six numbers.\n\nWrite your scores into your model inventory next to the licence. When somebody asks in a year why you picked this model, the six numbers are the answer, and they still make sense after the leaderboard has turned over four times."},"check":{"question":"An auditor asks you to demonstrate that the benchmark you quote for your deployed model was not present in its training data. Which axis of openness does that require, and what happens if the release scores zero on it?","options":[{"text":"The licence axis; without a permissive licence you have no right to inspect the model's behaviour on the benchmark.","correct":false,"tempting":"Confuses a legal permission to use the model with the technical ability to inspect what it was trained on; the licence says nothing about contamination."},{"text":"The evaluation axis; publishing the harness commit would settle the contamination question directly.","correct":false,"tempting":"The harness makes a score reproducible but reveals nothing about whether the test items were in the training corpus."},{"text":"The training data axis; with no corpus to search, contamination can only be estimated behaviourally, for example by testing on a freshly written equivalent set.","correct":true},{"text":"The checkpoints axis; comparing intermediate checkpoints would show when the benchmark was memorised.","correct":false,"tempting":"Checkpoints can show when a capability appeared but cannot distinguish learning from memorising without knowing what data went in between them."}]},"prev":{"slug":"why-labs-release-weights","title":"Why a lab gives away a model","url":"https://addaly.com/learn/open-models/why-labs-release-weights"},"next":{"slug":"data-provenance","title":"Where the training data comes from","url":"https://addaly.com/learn/open-models/data-provenance"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Grading openness without arguing about the word', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/the-openness-spectrum (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Grading openness without arguing about the word","courseTitle":"The Open Model Ecosystem","module":{"index":1,"slug":"what-open-really-means","title":"What open actually means"},"answer":null,"keyIdea":"Score a release on six separate axes (weights, licence, data, code, evaluation, checkpoints) because each axis buys a different concrete capability, and only some of them matter for any given job.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"points out of 12","title":"Three releases scored on the same six axes","series":[{"label":"A typical Llama release","value":5},{"label":"Qwen3 under Apache 2.0","value":6},{"label":"OLMo 2 from AI2","value":12,"highlight":true}],"caption":"Weights, licence, data, code, evaluation and checkpoints, two points each. The licence axis is the whole difference between the first two, and for a commercial team it is usually the axis that decides. The release scoring twelve is used least, because at any given moment it is not the strongest model at its size."},"alt":"Figure: Three releases scored on the same six axes. Bars (points out of 12): A typical Llama release 5; Qwen3 under Apache 2.0 6; OLMo 2 from AI2 12. Weights, licence, data, code, evaluation and checkpoints, two points each. The licence axis is the whole difference between the first two, and for a commercial team it is usually the axis that decides. The release scoring twelve is used least, because at any given moment it is not the strongest model at its size."}},"blocks":[{"t":"h2","id":"stop-arguing-start-scoring","runs":[{"text":"Stop arguing, start scoring"}]},{"t":"p","runs":[{"text":"Arguments about whether a model is open source go nowhere because the two sides are talking about different freedoms. The argument ends the moment you replace the word with a short list of axes and score each one. Then a disagreement becomes a specific claim: you needed axis four, and this release scores zero on it."}]},{"t":"p","runs":[{"text":"Six axes cover almost everything anybody actually needs."}]},{"t":"ol","start":1,"items":[[{"text":"Weights.","b":true},{"text":" Are the parameters downloadable, and without asking permission? Score 2 for an open download, 1 for a gated repo that requires accepting terms, 0 for API only."}],[{"text":"Licence.","b":true},{"text":" Score 2 for an OSI-approved licence with no use restrictions (Apache 2.0, MIT), 1 for a bespoke community licence with conditions, 0 for non-commercial or research-only."}],[{"text":"Training data.","b":true},{"text":" 2 if the corpus itself is released, 1 if it is described in enough detail to be checked, 0 if the card says a mixture of publicly available data."}],[{"text":"Training code and recipe.","b":true},{"text":" 2 for the actual pipeline, 1 for a paper with hyperparameters, 0 for neither."}],[{"text":"Evaluation.","b":true},{"text":" 2 for the harness, the exact commit and the raw per-task outputs, 1 for a table of numbers with the harness named, 0 for a marketing chart."}],[{"text":"Checkpoints.","b":true},{"text":" 2 for intermediate checkpoints across training, 1 for the final base as well as the instruct model, 0 for the instruct model only."}]]},{"t":"h2","id":"what-three-real-releases-look-like","runs":[{"text":"What three real releases look like"}]},{"t":"p","runs":[{"text":"Roughly, and this moves:"}]},{"t":"ul","items":[[{"text":"A typical Llama release.","b":true},{"text":" Weights 2, licence 1, data 0, code 0, evaluation 1, checkpoints 1. Total 5 of 12. You can build a business on it. You cannot audit it."}],[{"text":"Qwen3 under Apache 2.0.","b":true},{"text":" Weights 2, licence 2, data 0, code 0, evaluation 1, checkpoints 1. Total 6. The licence axis is the whole difference, and for a commercial team that axis is often the one that decides."}],[{"text":"OLMo 2 from AI2.","b":true},{"text":" Weights 2, licence 2, data 2, code 2, evaluation 2, checkpoints 2. Total 12. Fewer people use it, because at any moment it is not the strongest model at its size. That trade is the actual shape of the ecosystem."}]]},{"t":"h2","id":"each-axis-buys-a-specific-thing","runs":[{"text":"Each axis buys a specific thing"}]},{"t":"p","runs":[{"text":"This is the part that makes the rubric useful rather than pedantic."}]},{"t":"p","runs":[{"text":"Data disclosure","b":true},{"text":" buys contamination checking. If you want to know whether the benchmark you are quoting was in the training set, there is exactly one way to find out, and it is to search the corpus. Without the corpus you are guessing. It also buys copyright diligence: you can answer what is in it rather than reporting that nobody told you."}]},{"t":"p","runs":[{"text":"Training code","b":true},{"text":" buys reproduction and, more practically, continued pretraining that does not fight the original recipe. If you plan to train further on your own data, knowing the exact optimiser, schedule and data mix means your run resembles the original rather than knocking it sideways."}]},{"t":"p","runs":[{"text":"Checkpoints","b":true},{"text":" buy research on training dynamics, and they buy the option of restarting from an earlier point when the final instruct tuning has removed a behaviour you needed. A base model released alongside the instruct model is the single most useful item on this axis for ordinary work."}]},{"t":"p","runs":[{"text":"Evaluation artefacts","b":true},{"text":" buy the ability to reproduce a claimed number. A table of scores with no harness commit is not reproducible even in principle, because the harness moves."}]},{"t":"h2","id":"how-to-use-it-in-ten-minutes","runs":[{"text":"How to use it in ten minutes"}]},{"t":"p","runs":[{"text":"Open the model card and the repository file list. Score the six axes. Then ask the only question that matters: which axis does my situation require?"}]},{"t":"ul","items":[[{"text":"Shipping a commercial product: axis 2 dominates, and axes 3 to 6 can be zero without hurting you."}],[{"text":"Answering a regulator or a customer security review: axes 3 and 5 become necessary."}],[{"text":"Doing research you intend to publish: axes 4 and 6 or you cannot say anything defensible about causes."}],[{"text":"Fine-tuning for a new language: axis 6, because you want the base rather than the instruct model, and axis 3, because you want to know how much of that language it has already seen."}]]},{"t":"quote","runs":[{"text":"Openwashing is the term for a release that scores 2 on weights, advertises itself as open source, and scores 0 on everything else. The rubric is the answer to it: you do not have to accuse anyone of anything, you just publish the six numbers."}]},{"t":"p","runs":[{"text":"Write your scores into your model inventory next to the licence. When somebody asks in a year why you picked this model, the six numbers are the answer, and they still make sense after the leaderboard has turned over four times."}]}],"check":{"question":[{"text":"An auditor asks you to demonstrate that the benchmark you quote for your deployed model was not present in its training data. Which axis of openness does that require, and what happens if the release scores zero on it?"}],"options":[{"text":[{"text":"The licence axis; without a permissive licence you have no right to inspect the model’s behaviour on the benchmark."}],"correct":false,"tempting":[{"text":"Confuses a legal permission to use the model with the technical ability to inspect what it was trained on; the licence says nothing about contamination."}]},{"text":[{"text":"The evaluation axis; publishing the harness commit would settle the contamination question directly."}],"correct":false,"tempting":[{"text":"The harness makes a score reproducible but reveals nothing about whether the test items were in the training corpus."}]},{"text":[{"text":"The training data axis; with no corpus to search, contamination can only be estimated behaviourally, for example by testing on a freshly written equivalent set."}],"correct":true,"tempting":null},{"text":[{"text":"The checkpoints axis; comparing intermediate checkpoints would show when the benchmark was memorised."}],"correct":false,"tempting":[{"text":"Checkpoints can show when a capability appeared but cannot distinguish learning from memorising without knowing what data went in between them."}]}],"order":[1,3,0,2],"why":null},"prev":{"slug":"why-labs-release-weights","title":"Why a lab gives away a model","minutes":9},"next":{"slug":"data-provenance","title":"Where the training data comes from","minutes":10}}}