{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"writing-a-system-card","title":"The document that travels with the system","url":"https://addaly.com/learn/evaluating-ai/writing-a-system-card","position":59,"of":76,"minutes":9,"answer":"A system card is only worth writing if every claim in it carries the measurement behind it and the out-of-scope section is specific enough to stop somebody using the thing wrongly.","keyIdea":"A system card is only worth writing if every claim in it carries the measurement behind it and the out-of-scope section is specific enough to stop somebody using the thing wrongly."},"body":{"format":"markdown","text":"## What it is for\n\nA system card is a short document that goes wherever the system goes: what it does, what it was measured on, where it fails, and who to contact. The idea comes from model cards and from datasheets for datasets, both proposed around 2018 and 2019, and it has become ordinary practice for published models.\n\nIts purpose is narrow and useful. Somebody who did not build the system has to decide whether to rely on it. The card is what they read.\n\n## The sections that earn their place\n\n**Intended use.** The tasks it was built and measured for, in plain sentences. \"Answering questions about our published help articles, in English and Hindi, for logged-in customers.\"\n\n**Out of scope.** The most valuable section and the one most often skipped. Be specific enough to stop somebody: \"not for medical, legal or financial advice; not for eligibility decisions; not for languages other than the two listed; not for documents the retrieval corpus does not contain.\"\n\n**The data.** Where the evaluation items came from, when they were collected, how they were sampled, the languages and the proportions, the labelling process and the agreement figure, and the legal basis if the items are real user content.\n\n**The results.** Metrics with intervals, sample sizes, and per-slice breakdowns, never a single headline. Include the baselines, so a reader can see what the system adds.\n\n**Known failure modes**, with real examples. Two or three sanitised failures do more for a reader's judgement than a page of metrics.\n\n**Safety and abuse testing**: what adversarial testing was done, what it found, and what remains open.\n\n**Operational facts**: latency, cost per request, dependencies, the model version and the date it was pinned.\n\n**Provenance and contact**: who owns it, when the card was last updated, and how to report a problem.\n\n## The rule that separates a card from a brochure\n\nEvery claim carries its measurement.\n\n> Weak: \"Performs well in Hindi.\"\n>\n> Card: \"Hindi: 84% task success (95% CI 78–89) on 300 items sampled from March traffic, labelled by two reviewers at kappa 0.71. English: 91% (87–94) on 400 items.\"\n\nThe second version is longer, slower to write, and it is the entire point. A card full of adjectives is marketing with a technical typeface, and the first reader who checks will stop trusting the rest of it.\n\nWhere you did not measure something, write that you did not. \"Tamil and Bengali are supported by the model but have not been evaluated; we have no evidence about quality in those languages\" is a useful sentence, and it is more honest than a benchmark score nobody on the team can read.\n\n## Two versions, one truth\n\nMost teams end up with an internal card carrying detail that cannot be published (supplier terms, internal thresholds, unfixed vulnerabilities) and an external one that is shorter.\n\nThe rule is that the external card may omit, and may never contradict. The moment the published card says something the internal one disproves, you have a document that will be read back to you.\n\n## Cards rot\n\nThe failure mode is not writing a bad card. It is writing a good one at launch and never touching it, so that eighteen months later it describes a model you no longer serve, a corpus that has doubled and a prompt that has been rewritten twice.\n\nTwo habits fix it. Keep the card in the repository beside the code, so changes go through the same review. And put the update trigger in the release checklist: a model version change, a prompt change that moves the numbers, or a new language always updates the card, and the card carries the date and the version it describes.\n\n## What a card is not\n\nIt is a claim by the people who built the system, evaluated on a set they chose, with metrics they selected. That is not nothing (it is far more than most systems come with), and it is not verification. The next lesson is about why the builder's own evaluation is structurally weak evidence, and what a stronger form looks like.\n\nFree tooling exists and is worth using: Hugging Face's model card template gives you the section structure, and a plain markdown file in the repository is sufficient for everything else."},"check":{"question":"Which claim belongs in a system card as written?","options":[{"text":"\"The system is highly accurate for customer support queries across all supported languages.\"","correct":false,"tempting":"Compresses several unmeasured claims into an adjective; a reader cannot tell what was tested, on how much, or in which languages."},{"text":"\"Not evaluated in Tamil or Bengali; the model accepts both, and we have no evidence about quality in either.\"","correct":true},{"text":"\"Extensively red-teamed by our internal team prior to launch.\"","correct":false,"tempting":"Describes effort rather than result; a card should say what the testing looked for, how much of it was done, and what it found."},{"text":"\"Outperforms leading alternatives on standard benchmarks.\"","correct":false,"tempting":"Cites public benchmarks that may sit in training data and describe a different task, with no reference to the set that matters to the reader."}]},"prev":{"slug":"human-review-that-scales","title":"Sampling live traffic, and keeping reviewers honest","url":"https://addaly.com/learn/evaluating-ai/human-review-that-scales"},"next":{"slug":"evaluating-a-vendor-model","title":"Buying a model you will depend on","url":"https://addaly.com/learn/evaluating-ai/evaluating-a-vendor-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The document that travels with the system', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/writing-a-system-card (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The document that travels with the system","courseTitle":"Knowing If It Works","module":{"index":7,"slug":"assurance","title":"Evidence other people can act on"},"answer":null,"keyIdea":"A system card is only worth writing if every claim in it carries the measurement behind it and the out-of-scope section is specific enough to stop somebody using the thing wrongly.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-it-is-for","runs":[{"text":"What it is for"}]},{"t":"p","runs":[{"text":"A system card is a short document that goes wherever the system goes: what it does, what it was measured on, where it fails, and who to contact. The idea comes from model cards and from datasheets for datasets, both proposed around 2018 and 2019, and it has become ordinary practice for published models."}]},{"t":"p","runs":[{"text":"Its purpose is narrow and useful. Somebody who did not build the system has to decide whether to rely on it. The card is what they read."}]},{"t":"h2","id":"the-sections-that-earn-their-place","runs":[{"text":"The sections that earn their place"}]},{"t":"p","runs":[{"text":"Intended use.","b":true},{"text":" The tasks it was built and measured for, in plain sentences. “Answering questions about our published help articles, in English and Hindi, for logged-in customers.”"}]},{"t":"p","runs":[{"text":"Out of scope.","b":true},{"text":" The most valuable section and the one most often skipped. Be specific enough to stop somebody: “not for medical, legal or financial advice; not for eligibility decisions; not for languages other than the two listed; not for documents the retrieval corpus does not contain.”"}]},{"t":"p","runs":[{"text":"The data.","b":true},{"text":" Where the evaluation items came from, when they were collected, how they were sampled, the languages and the proportions, the labelling process and the agreement figure, and the legal basis if the items are real user content."}]},{"t":"p","runs":[{"text":"The results.","b":true},{"text":" Metrics with intervals, sample sizes, and per-slice breakdowns, never a single headline. Include the baselines, so a reader can see what the system adds."}]},{"t":"p","runs":[{"text":"Known failure modes","b":true},{"text":", with real examples. Two or three sanitised failures do more for a reader’s judgement than a page of metrics."}]},{"t":"p","runs":[{"text":"Safety and abuse testing","b":true},{"text":": what adversarial testing was done, what it found, and what remains open."}]},{"t":"p","runs":[{"text":"Operational facts","b":true},{"text":": latency, cost per request, dependencies, the model version and the date it was pinned."}]},{"t":"p","runs":[{"text":"Provenance and contact","b":true},{"text":": who owns it, when the card was last updated, and how to report a problem."}]},{"t":"h2","id":"the-rule-that-separates-a-card-from-a-brochure","runs":[{"text":"The rule that separates a card from a brochure"}]},{"t":"p","runs":[{"text":"Every claim carries its measurement."}]},{"t":"quote","runs":[{"text":"Weak: “Performs well in Hindi.”"}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"Card: “Hindi: 84% task success (95% CI 78–89) on 300 items sampled from March traffic, labelled by two reviewers at kappa 0.71. English: 91% (87–94) on 400 items.”"}]},{"t":"p","runs":[{"text":"The second version is longer, slower to write, and it is the entire point. A card full of adjectives is marketing with a technical typeface, and the first reader who checks will stop trusting the rest of it."}]},{"t":"p","runs":[{"text":"Where you did not measure something, write that you did not. “Tamil and Bengali are supported by the model but have not been evaluated; we have no evidence about quality in those languages” is a useful sentence, and it is more honest than a benchmark score nobody on the team can read."}]},{"t":"h2","id":"two-versions-one-truth","runs":[{"text":"Two versions, one truth"}]},{"t":"p","runs":[{"text":"Most teams end up with an internal card carrying detail that cannot be published (supplier terms, internal thresholds, unfixed vulnerabilities) and an external one that is shorter."}]},{"t":"p","runs":[{"text":"The rule is that the external card may omit, and may never contradict. The moment the published card says something the internal one disproves, you have a document that will be read back to you."}]},{"t":"h2","id":"cards-rot","runs":[{"text":"Cards rot"}]},{"t":"p","runs":[{"text":"The failure mode is not writing a bad card. It is writing a good one at launch and never touching it, so that eighteen months later it describes a model you no longer serve, a corpus that has doubled and a prompt that has been rewritten twice."}]},{"t":"p","runs":[{"text":"Two habits fix it. Keep the card in the repository beside the code, so changes go through the same review. And put the update trigger in the release checklist: a model version change, a prompt change that moves the numbers, or a new language always updates the card, and the card carries the date and the version it describes."}]},{"t":"h2","id":"what-a-card-is-not","runs":[{"text":"What a card is not"}]},{"t":"p","runs":[{"text":"It is a claim by the people who built the system, evaluated on a set they chose, with metrics they selected. That is not nothing (it is far more than most systems come with), and it is not verification. The next lesson is about why the builder’s own evaluation is structurally weak evidence, and what a stronger form looks like."}]},{"t":"p","runs":[{"text":"Free tooling exists and is worth using: Hugging Face’s model card template gives you the section structure, and a plain markdown file in the repository is sufficient for everything else."}]}],"check":{"question":[{"text":"Which claim belongs in a system card as written?"}],"options":[{"text":[{"text":"“The system is highly accurate for customer support queries across all supported languages.”"}],"correct":false,"tempting":[{"text":"Compresses several unmeasured claims into an adjective; a reader cannot tell what was tested, on how much, or in which languages."}]},{"text":[{"text":"“Not evaluated in Tamil or Bengali; the model accepts both, and we have no evidence about quality in either.”"}],"correct":true,"tempting":null},{"text":[{"text":"“Extensively red-teamed by our internal team prior to launch.”"}],"correct":false,"tempting":[{"text":"Describes effort rather than result; a card should say what the testing looked for, how much of it was done, and what it found."}]},{"text":[{"text":"“Outperforms leading alternatives on standard benchmarks.”"}],"correct":false,"tempting":[{"text":"Cites public benchmarks that may sit in training data and describe a different task, with no reference to the set that matters to the reader."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"human-review-that-scales","title":"Sampling live traffic, and keeping reviewers honest","minutes":9},"next":{"slug":"evaluating-a-vendor-model","title":"Buying a model you will depend on","minutes":10}}}