Grading openness without arguing about the word

Stop arguing, start scoring

Arguments about whether a model is open source go nowhere because the two sides are talking about different freedoms. The argument ends the moment you replace the word with a short list of axes and score each one. Then a disagreement becomes a specific claim: you needed axis four, and this release scores zero on it.

Six axes cover almost everything anybody actually needs.

1. Weights. Are the parameters downloadable, and without asking permission? Score 2 for an open download, 1 for a gated repo that requires accepting terms, 0 for API only.
2. Licence. Score 2 for an OSI-approved licence with no use restrictions (Apache 2.0, MIT), 1 for a bespoke community licence with conditions, 0 for non-commercial or research-only.
3. Training data. 2 if the corpus itself is released, 1 if it is described in enough detail to be checked, 0 if the card says a mixture of publicly available data.
4. Training code and recipe. 2 for the actual pipeline, 1 for a paper with hyperparameters, 0 for neither.
5. Evaluation. 2 for the harness, the exact commit and the raw per-task outputs, 1 for a table of numbers with the harness named, 0 for a marketing chart.
6. Checkpoints. 2 for intermediate checkpoints across training, 1 for the final base as well as the instruct model, 0 for the instruct model only.

What three real releases look like

Roughly, and this moves:

• A typical Llama release. Weights 2, licence 1, data 0, code 0, evaluation 1, checkpoints 1. Total 5 of 12. You can build a business on it. You cannot audit it.
• Qwen3 under Apache 2.0. Weights 2, licence 2, data 0, code 0, evaluation 1, checkpoints 1. Total 6. The licence axis is the whole difference, and for a commercial team that axis is often the one that decides.
• OLMo 2 from AI2. Weights 2, licence 2, data 2, code 2, evaluation 2, checkpoints 2. Total 12. Fewer people use it, because at any moment it is not the strongest model at its size. That trade is the actual shape of the ecosystem.

Each axis buys a specific thing

This is the part that makes the rubric useful rather than pedantic.

Data disclosure buys contamination checking. If you want to know whether the benchmark you are quoting was in the training set, there is exactly one way to find out, and it is to search the corpus. Without the corpus you are guessing. It also buys copyright diligence: you can answer what is in it rather than reporting that nobody told you.

Training code buys reproduction and, more practically, continued pretraining that does not fight the original recipe. If you plan to train further on your own data, knowing the exact optimiser, schedule and data mix means your run resembles the original rather than knocking it sideways.

Checkpoints buy research on training dynamics, and they buy the option of restarting from an earlier point when the final instruct tuning has removed a behaviour you needed. A base model released alongside the instruct model is the single most useful item on this axis for ordinary work.

Evaluation artefacts buy the ability to reproduce a claimed number. A table of scores with no harness commit is not reproducible even in principle, because the harness moves.

How to use it in ten minutes

Open the model card and the repository file list. Score the six axes. Then ask the only question that matters: which axis does my situation require?

• Shipping a commercial product: axis 2 dominates, and axes 3 to 6 can be zero without hurting you.
• Answering a regulator or a customer security review: axes 3 and 5 become necessary.
• Doing research you intend to publish: axes 4 and 6 or you cannot say anything defensible about causes.
• Fine-tuning for a new language: axis 6, because you want the base rather than the instruct model, and axis 3, because you want to know how much of that language it has already seen.

Openwashing is the term for a release that scores 2 on weights, advertises itself as open source, and scores 0 on everything else. The rubric is the answer to it: you do not have to accuse anyone of anything, you just publish the six numbers.

Write your scores into your model inventory next to the licence. When somebody asks in a year why you picked this model, the six numbers are the answer, and they still make sense after the leaderboard has turned over four times.