Reading a model card for what it does not say

Two documents in one file

A model card is part machine-readable metadata and part marketing prose, and the two deserve different levels of trust.

The front matter (the YAML block at the top) is structured, is used by tooling, and is the hardest part to make vague. license, base_model, language, pipeline_tag, library_name, datasets, tags. Read this first. It takes fifteen seconds and it settles the licence chain, the parent model, and what the publisher is claiming about language coverage.

The prose is where claims live. Read it with a specific question in mind: for each claim, what evidence would settle it, and is that evidence here?

The sections, and what each is worth

Model description. Architecture, size, context length, what it was built for. Mostly factual and mostly reliable.

Intended use, and out-of-scope use. The presence of a real out-of-scope section (specific, not boilerplate) is the strongest positive signal on the page. It means somebody thought about failure. A card whose out-of-scope section says only illegal purposes has told you nothing.

Training data. Almost always a sentence. A mixture of publicly available data means you know nothing, and the openness rubric from the first block scores it zero. Where a card names datasets, follow the links; some of them do not exist.

Evaluation. The section that most needs interrogating, next.

Limitations and bias. Frequently copied between cards. Specific limitations (degrades above 16k tokens, weak at Bengali despite the language tag) indicate real evaluation. Generic paragraphs about how models can produce biased output indicate a template.

Interrogating a benchmark table

A number in a model card is worth something only with five accompanying facts:

1. Which harness, at which commit. Without this the number is not reproducible even in principle.
2. How many shots. Zero-shot and five-shot MMLU differ by several points on the same model.
3. Single sample or aggregated. A majority-vote-at-64 score sitting in a column next to somebody else’s single-sample number is not a comparison.
4. The decoding settings. Greedy or sampled, and at what temperature.
5. Who produced the baseline numbers. Numbers the publisher ran themselves, under the same configuration, are a comparison. Numbers copied from three different papers are a collage.

If four of the five are missing, the table is a claim, not evidence. That does not mean the model is bad; it means you have to measure it yourself, which you were going to do anyway.

Specific things to be suspicious of

A language list far longer than the evaluation. A card listing 100 languages and evaluating on four has told you which four it was tested on.

Only the benchmarks the model wins. Everybody reports MMLU. If a code model omits code benchmarks, ask why.

Comparison charts with no axis label, or with a truncated axis that turns two points into a visual chasm.

State of the art with no date. In this field the phrase decays in weeks.

No base_model on an obvious derivative. Either carelessness or a licence chain somebody preferred not to draw attention to.

Very high scores from a very small organisation with no accompanying method. Occasionally a real result. More often a contaminated evaluation, and always worth reproducing before quoting.

A three-minute procedure

hf download <org>/<model> README.md --local-dir ./card
sed -n '1,30p' ./card/README.md          # front matter: licence, base, languages
grep -in "out-of-scope\|limitation\|not intended" ./card/README.md
grep -in "harness\|shots\|few-shot\|lm-eval\|temperature" ./card/README.md

The third command is the one that tells you the most. If it returns nothing, no number on the page is reproducible, and you now know exactly how much weight to put on the table.

What a good card looks like

For contrast, the shape worth emulating and worth trusting: front matter complete and consistent; a paragraph on intended use and a specific out-of-scope list; named datasets with links; an evaluation table with the harness commit, shot count and decoding settings, reporting the base model under the identical configuration; a limitations section naming real weaknesses; and a contact route.

Cards like that exist. They are a minority, they are disproportionately from the fully-open end of the spectrum, and the effort visible in them is itself evidence about the care taken with the model.