The document that travels with the system

What it is for

A system card is a short document that goes wherever the system goes: what it does, what it was measured on, where it fails, and who to contact. The idea comes from model cards and from datasheets for datasets, both proposed around 2018 and 2019, and it has become ordinary practice for published models.

Its purpose is narrow and useful. Somebody who did not build the system has to decide whether to rely on it. The card is what they read.

The sections that earn their place

Intended use. The tasks it was built and measured for, in plain sentences. “Answering questions about our published help articles, in English and Hindi, for logged-in customers.”

Out of scope. The most valuable section and the one most often skipped. Be specific enough to stop somebody: “not for medical, legal or financial advice; not for eligibility decisions; not for languages other than the two listed; not for documents the retrieval corpus does not contain.”

The data. Where the evaluation items came from, when they were collected, how they were sampled, the languages and the proportions, the labelling process and the agreement figure, and the legal basis if the items are real user content.

The results. Metrics with intervals, sample sizes, and per-slice breakdowns, never a single headline. Include the baselines, so a reader can see what the system adds.

Known failure modes, with real examples. Two or three sanitised failures do more for a reader’s judgement than a page of metrics.

Safety and abuse testing: what adversarial testing was done, what it found, and what remains open.

Operational facts: latency, cost per request, dependencies, the model version and the date it was pinned.

Provenance and contact: who owns it, when the card was last updated, and how to report a problem.

The rule that separates a card from a brochure

Every claim carries its measurement.

Weak: “Performs well in Hindi.”

Card: “Hindi: 84% task success (95% CI 78–89) on 300 items sampled from March traffic, labelled by two reviewers at kappa 0.71. English: 91% (87–94) on 400 items.”

The second version is longer, slower to write, and it is the entire point. A card full of adjectives is marketing with a technical typeface, and the first reader who checks will stop trusting the rest of it.

Where you did not measure something, write that you did not. “Tamil and Bengali are supported by the model but have not been evaluated; we have no evidence about quality in those languages” is a useful sentence, and it is more honest than a benchmark score nobody on the team can read.

Two versions, one truth

Most teams end up with an internal card carrying detail that cannot be published (supplier terms, internal thresholds, unfixed vulnerabilities) and an external one that is shorter.

The rule is that the external card may omit, and may never contradict. The moment the published card says something the internal one disproves, you have a document that will be read back to you.

Cards rot

The failure mode is not writing a bad card. It is writing a good one at launch and never touching it, so that eighteen months later it describes a model you no longer serve, a corpus that has doubled and a prompt that has been rewritten twice.

Two habits fix it. Keep the card in the repository beside the code, so changes go through the same review. And put the update trigger in the release checklist: a model version change, a prompt change that moves the numbers, or a new language always updates the card, and the card carries the date and the version it describes.

What a card is not

It is a claim by the people who built the system, evaluated on a set they chose, with metrics they selected. That is not nothing (it is far more than most systems come with), and it is not verification. The next lesson is about why the builder’s own evaluation is structurally weak evidence, and what a stronger form looks like.

Free tooling exists and is worth using: Hugging Face’s model card template gives you the section structure, and a plain markdown file in the repository is sufficient for everything else.