{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-summaries","title":"Summaries: what was invented, and what was left out","url":"https://addaly.com/learn/evaluating-ai/evaluating-summaries","position":30,"of":76,"minutes":9,"answer":"Score a summary on two separate axes (every claim supported by the source, and every required fact present) because no automatic similarity metric can see an omission at all.","keyIdea":"Score a summary on two separate axes (every claim supported by the source, and every required fact present) because no automatic similarity metric can see an omission at all."},"body":{"format":"markdown","text":"## Two failures, and only one of them is visible\n\nA summary can fail in two ways. It can say something the source does not support, which is a hallucination. Or it can leave out the thing that mattered, which is an omission.\n\nSimilarity metrics see neither properly, and omission not at all. A summary of a medical note that drops the allergy will score well against a reference that also drops it, and score badly against one that includes it. The metric is measuring wording. Build two checks instead.\n\n> Figure: Two failures, two separate checks. Invented (hallucination): Split the summary into atomic claims; Check each against the source: supported, contradicted, not stated; Report the share of summaries with at least one unsupported claim; Numbers and dates: a regex against the source, no model needed. Left out (omission): No automatic metric can see a missing fact; Write three to six must-include items per document, in advance; Coverage is the fraction of them present; Fifty documents with lists beat five hundred without. A similarity metric sees wording. It scores a summary that drops the allergy well against a reference that also drops it, and it cannot see an omission at all.\n\n## Faithfulness: decompose into claims\n\nSplit the summary into atomic claims (one assertion each) and check every claim against the source as *supported*, *contradicted*, or *not stated*.\n\n> Summary: \"The council approved the budget on 14 March, raising parking fees by 12% from April.\"\n>\n> Claims: (1) the council approved the budget; (2) approval was on 14 March; (3) parking fees rise 12%; (4) the rise takes effect in April.\n\nFour claims, checked one at a time. Report two numbers: the claim support rate across all claims, and (more important) the **share of summaries containing at least one unsupported claim**. A system at 96% claim support can still be wrong somewhere in half its summaries, and it is the summary a person reads, not the claim.\n\nThe checking can be done by a person, by a judge model with the rubric discipline from the judging block, or by a small natural-language-inference model. An NLI model from Hugging Face runs on a CPU, costs nothing per call, and outputs entailment or contradiction against the source sentence, slower to set up and far cheaper to run than a judge.\n\n## The free check that catches the worst errors\n\nNumbers and dates are where hallucination does real damage, and they are checkable with no model at all:\n\n```python\nimport re\nnums_in = set(re.findall(r\"\\d[\\d,.]*\", source))\nnums_out = set(re.findall(r\"\\d[\\d,.]*\", summary))\ninvented = nums_out - nums_in           # any value here is a red flag\n```\n\nRun this first. It is exact, instant, and it catches transposed digits, invented percentages and dates that drifted by a day: the errors readers are least likely to notice and most likely to act on.\n\n## Omission: you must say in advance what matters\n\nThere is no way to measure a missing fact without knowing which facts were required, so somebody has to write them down. For each document in the set, a person lists three to six **must-include** items. Coverage is then the fraction present in the summary.\n\nThis is real work (perhaps five minutes per document) and it is the only part of summary evaluation that measures the thing users actually complain about. Fifty documents with must-include lists is a better set than five hundred without.\n\n## Test where in the document the model stops looking\n\nAttention over a long document is not uniform, and the reliable way to find out how yours behaves is to plant a required fact at a known depth. Take the same document and insert a distinctive sentence at 10%, 50% and 90% of the way through, then measure recall of that fact by position.\n\nThe usual finding is a dip in the middle, and it is actionable: it argues for chunking the document and summarising in sections rather than for a better prompt. Twenty documents, three positions, an afternoon.\n\n## Control for length, and beat the lead baseline\n\nTwo things make summary comparisons unfair.\n\n**Length.** A longer summary covers more and invents more. Fix a band (say 60 to 100 words) and enforce it, or you are comparing verbosity.\n\n**The baseline nobody runs.** For news and reports, taking the first three sentences is a famously strong summariser. If your system does not beat lead-3 on must-include coverage, you have no result yet. It costs one line of code to check and it has ended more than one project honestly.\n\nSimilarity metrics such as ROUGE keep their one legitimate use here: as a drift detector. If ROUGE against last week's outputs falls sharply after a prompt change, the wording moved a lot, which is worth knowing. It is not a quality measurement, and reporting it as one is how teams end up optimising for copying the source, which is exactly what n-gram overlap rewards."},"check":{"question":"Why can no reference-based similarity metric reliably detect that a summary omitted a critical fact?","options":[{"text":"Because similarity metrics are computed on n-grams, which are too short to represent a fact.","correct":false,"tempting":"Points at the unit of comparison rather than the missing information; even a sentence-level similarity metric has the same blind spot."},{"text":"Because the reference summary may itself omit the fact, and the score only measures overlap with that reference rather than coverage of what mattered in the source.","correct":true},{"text":"Because omissions reduce length, and the brevity penalty already accounts for them.","correct":false,"tempting":"Describes a mechanism that partly exists in BLEU, and a brevity penalty punishes shortness generally without knowing whether the missing content mattered."},{"text":"Because similarity metrics cannot be computed when the summary is shorter than the reference.","correct":false,"tempting":"Invents a technical limitation; the metrics compute perfectly well across different lengths, which is part of the problem."}]},"prev":{"slug":"simpsons-paradox","title":"Better in every group and worse overall","url":"https://addaly.com/learn/evaluating-ai/simpsons-paradox"},"next":{"slug":"evaluating-translation","title":"Translation, and the languages nobody measures","url":"https://addaly.com/learn/evaluating-ai/evaluating-translation"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Summaries: what was invented, and what was left out', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-summaries (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Summaries: what was invented, and what was left out","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"Score a summary on two separate axes (every claim supported by the source, and every required fact present) because no automatic similarity metric can see an omission at all.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 4.1","figure":{"left":{"title":"Invented (hallucination)","items":["Split the summary into atomic claims","Check each against the source: supported, contradicted, not stated","Report the share of summaries with at least one unsupported claim","Numbers and dates: a regex against the source, no model needed"]},"type":"compare","right":{"title":"Left out (omission)","items":["No automatic metric can see a missing fact","Write three to six must-include items per document, in advance","Coverage is the fraction of them present","Fifty documents with lists beat five hundred without"]},"title":"Two failures, two separate checks","caption":"A similarity metric sees wording. It scores a summary that drops the allergy well against a reference that also drops it, and it cannot see an omission at all."},"alt":"Figure: Two failures, two separate checks. Invented (hallucination): Split the summary into atomic claims; Check each against the source: supported, contradicted, not stated; Report the share of summaries with at least one unsupported claim; Numbers and dates: a regex against the source, no model needed. Left out (omission): No automatic metric can see a missing fact; Write three to six must-include items per document, in advance; Coverage is the fraction of them present; Fifty documents with lists beat five hundred without. A similarity metric sees wording. It scores a summary that drops the allergy well against a reference that also drops it, and it cannot see an omission at all."}},"blocks":[{"t":"h2","id":"two-failures-and-only-one-of-them-is-visible","runs":[{"text":"Two failures, and only one of them is visible"}]},{"t":"p","runs":[{"text":"A summary can fail in two ways. It can say something the source does not support, which is a hallucination. Or it can leave out the thing that mattered, which is an omission."}]},{"t":"p","runs":[{"text":"Similarity metrics see neither properly, and omission not at all. A summary of a medical note that drops the allergy will score well against a reference that also drops it, and score badly against one that includes it. The metric is measuring wording. Build two checks instead."}]},{"t":"h2","id":"faithfulness-decompose-into-claims","runs":[{"text":"Faithfulness: decompose into claims"}]},{"t":"p","runs":[{"text":"Split the summary into atomic claims (one assertion each) and check every claim against the source as "},{"text":"supported","i":true},{"text":", "},{"text":"contradicted","i":true},{"text":", or "},{"text":"not stated","i":true},{"text":"."}]},{"t":"quote","runs":[{"text":"Summary: “The council approved the budget on 14 March, raising parking fees by 12% from April.”"}]},{"t":"quote","runs":[]},{"t":"quote","runs":[{"text":"Claims: (1) the council approved the budget; (2) approval was on 14 March; (3) parking fees rise 12%; (4) the rise takes effect in April."}]},{"t":"p","runs":[{"text":"Four claims, checked one at a time. Report two numbers: the claim support rate across all claims, and (more important) the "},{"text":"share of summaries containing at least one unsupported claim","b":true},{"text":". A system at 96% claim support can still be wrong somewhere in half its summaries, and it is the summary a person reads, not the claim."}]},{"t":"p","runs":[{"text":"The checking can be done by a person, by a judge model with the rubric discipline from the judging block, or by a small natural-language-inference model. An NLI model from Hugging Face runs on a CPU, costs nothing per call, and outputs entailment or contradiction against the source sentence, slower to set up and far cheaper to run than a judge."}]},{"t":"h2","id":"the-free-check-that-catches-the-worst-errors","runs":[{"text":"The free check that catches the worst errors"}]},{"t":"p","runs":[{"text":"Numbers and dates are where hallucination does real damage, and they are checkable with no model at all:"}]},{"t":"code","lang":"python","label":"Python","text":"import re\nnums_in = set(re.findall(r\"\\d[\\d,.]*\", source))\nnums_out = set(re.findall(r\"\\d[\\d,.]*\", summary))\ninvented = nums_out - nums_in           # any value here is a red flag"},{"t":"p","runs":[{"text":"Run this first. It is exact, instant, and it catches transposed digits, invented percentages and dates that drifted by a day: the errors readers are least likely to notice and most likely to act on."}]},{"t":"h2","id":"omission-you-must-say-in-advance-what-matters","runs":[{"text":"Omission: you must say in advance what matters"}]},{"t":"p","runs":[{"text":"There is no way to measure a missing fact without knowing which facts were required, so somebody has to write them down. For each document in the set, a person lists three to six "},{"text":"must-include","b":true},{"text":" items. Coverage is then the fraction present in the summary."}]},{"t":"p","runs":[{"text":"This is real work (perhaps five minutes per document) and it is the only part of summary evaluation that measures the thing users actually complain about. Fifty documents with must-include lists is a better set than five hundred without."}]},{"t":"h2","id":"test-where-in-the-document-the-model-stops-looking","runs":[{"text":"Test where in the document the model stops looking"}]},{"t":"p","runs":[{"text":"Attention over a long document is not uniform, and the reliable way to find out how yours behaves is to plant a required fact at a known depth. Take the same document and insert a distinctive sentence at 10%, 50% and 90% of the way through, then measure recall of that fact by position."}]},{"t":"p","runs":[{"text":"The usual finding is a dip in the middle, and it is actionable: it argues for chunking the document and summarising in sections rather than for a better prompt. Twenty documents, three positions, an afternoon."}]},{"t":"h2","id":"control-for-length-and-beat-the-lead-baseline","runs":[{"text":"Control for length, and beat the lead baseline"}]},{"t":"p","runs":[{"text":"Two things make summary comparisons unfair."}]},{"t":"p","runs":[{"text":"Length.","b":true},{"text":" A longer summary covers more and invents more. Fix a band (say 60 to 100 words) and enforce it, or you are comparing verbosity."}]},{"t":"p","runs":[{"text":"The baseline nobody runs.","b":true},{"text":" For news and reports, taking the first three sentences is a famously strong summariser. If your system does not beat lead-3 on must-include coverage, you have no result yet. It costs one line of code to check and it has ended more than one project honestly."}]},{"t":"p","runs":[{"text":"Similarity metrics such as ROUGE keep their one legitimate use here: as a drift detector. If ROUGE against last week’s outputs falls sharply after a prompt change, the wording moved a lot, which is worth knowing. It is not a quality measurement, and reporting it as one is how teams end up optimising for copying the source, which is exactly what n-gram overlap rewards."}]}],"check":{"question":[{"text":"Why can no reference-based similarity metric reliably detect that a summary omitted a critical fact?"}],"options":[{"text":[{"text":"Because similarity metrics are computed on n-grams, which are too short to represent a fact."}],"correct":false,"tempting":[{"text":"Points at the unit of comparison rather than the missing information; even a sentence-level similarity metric has the same blind spot."}]},{"text":[{"text":"Because the reference summary may itself omit the fact, and the score only measures overlap with that reference rather than coverage of what mattered in the source."}],"correct":true,"tempting":null},{"text":[{"text":"Because omissions reduce length, and the brevity penalty already accounts for them."}],"correct":false,"tempting":[{"text":"Describes a mechanism that partly exists in BLEU, and a brevity penalty punishes shortness generally without knowing whether the missing content mattered."}]},{"text":[{"text":"Because similarity metrics cannot be computed when the summary is shorter than the reference."}],"correct":false,"tempting":[{"text":"Invents a technical limitation; the metrics compute perfectly well across different lengths, which is part of the problem."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"simpsons-paradox","title":"Better in every group and worse overall","minutes":8},"next":{"slug":"evaluating-translation","title":"Translation, and the languages nobody measures","minutes":10}}}