Summaries: what was invented, and what was left out

Two failures, and only one of them is visible

A summary can fail in two ways. It can say something the source does not support, which is a hallucination. Or it can leave out the thing that mattered, which is an omission.

Similarity metrics see neither properly, and omission not at all. A summary of a medical note that drops the allergy will score well against a reference that also drops it, and score badly against one that includes it. The metric is measuring wording. Build two checks instead.

Faithfulness: decompose into claims

Split the summary into atomic claims (one assertion each) and check every claim against the source as supported, contradicted, or not stated.

Summary: “The council approved the budget on 14 March, raising parking fees by 12% from April.”

Claims: (1) the council approved the budget; (2) approval was on 14 March; (3) parking fees rise 12%; (4) the rise takes effect in April.

Four claims, checked one at a time. Report two numbers: the claim support rate across all claims, and (more important) the share of summaries containing at least one unsupported claim. A system at 96% claim support can still be wrong somewhere in half its summaries, and it is the summary a person reads, not the claim.

The checking can be done by a person, by a judge model with the rubric discipline from the judging block, or by a small natural-language-inference model. An NLI model from Hugging Face runs on a CPU, costs nothing per call, and outputs entailment or contradiction against the source sentence, slower to set up and far cheaper to run than a judge.

The free check that catches the worst errors

Numbers and dates are where hallucination does real damage, and they are checkable with no model at all:

import re
nums_in = set(re.findall(r"\d[\d,.]*", source))
nums_out = set(re.findall(r"\d[\d,.]*", summary))
invented = nums_out - nums_in           # any value here is a red flag

Run this first. It is exact, instant, and it catches transposed digits, invented percentages and dates that drifted by a day: the errors readers are least likely to notice and most likely to act on.

Omission: you must say in advance what matters

There is no way to measure a missing fact without knowing which facts were required, so somebody has to write them down. For each document in the set, a person lists three to six must-include items. Coverage is then the fraction present in the summary.

This is real work (perhaps five minutes per document) and it is the only part of summary evaluation that measures the thing users actually complain about. Fifty documents with must-include lists is a better set than five hundred without.

Test where in the document the model stops looking

Attention over a long document is not uniform, and the reliable way to find out how yours behaves is to plant a required fact at a known depth. Take the same document and insert a distinctive sentence at 10%, 50% and 90% of the way through, then measure recall of that fact by position.

The usual finding is a dip in the middle, and it is actionable: it argues for chunking the document and summarising in sections rather than for a better prompt. Twenty documents, three positions, an afternoon.

Control for length, and beat the lead baseline

Two things make summary comparisons unfair.

Length. A longer summary covers more and invents more. Fix a band (say 60 to 100 words) and enforce it, or you are comparing verbosity.

The baseline nobody runs. For news and reports, taking the first three sentences is a famously strong summariser. If your system does not beat lead-3 on must-include coverage, you have no result yet. It costs one line of code to check and it has ended more than one project honestly.

Similarity metrics such as ROUGE keep their one legitimate use here: as a drift detector. If ROUGE against last week’s outputs falls sharply after a prompt change, the wording moved a lot, which is worth knowing. It is not a quality measurement, and reporting it as one is how teams end up optimising for copying the source, which is exactly what n-gram overlap rewards.