Citations you can actually verify

A citation is a claim about your own pipeline

Asking a model to cite its sources is easy. Making the citation mean something requires three things your assembler must provide, and most systems provide only the first.

A stable handle. Every passage in the window carries a short marker ([3], S7) that the model is told to use. Short matters: it is emitted many times and each character is a token.

A mapping back. Your code holds [3] -> {doc_id, chunk_id, char_span}. This lives outside the window, in the request’s own state, and it is what turns a marker into a link, a page number and a highlight.

A verification step. After generation, check the citations before showing them. This is the part that is usually skipped, and it is where the value is.

Three checks worth running on every answer

cited = set(re.findall(r"\[(\d+)\]", answer))

# 1. Does every cited handle exist?
bogus = cited - set(handles)          # model cited [12]; you sent 8

# 2. Is every claim-bearing sentence cited?
uncited = [s for s in sentences(answer) if is_factual(s) and not has_cite(s)]

# 3. Does the quoted text actually appear in the cited passage?
for quote, h in quotes_with_handles(answer):
    assert normalise(quote) in normalise(passage[h].text)

The first catches a real and common failure: models invent handles, especially when the answer draws on their own background knowledge rather than the passages. A citation to a source you did not send is a strong signal that the sentence is unsupported, and it is detectable with a regex.

The third is the strongest cheap check available. If the model claims to quote and the string is not there, the sentence is wrong and you can say so without a judge model, a human, or an evaluation set.

The uncomfortable truth about citation accuracy

A citation says which passage the model was attending to when it wrote the sentence. It does not say the passage supports the sentence.

Models routinely produce sentences that are a blend of a retrieved passage and their own background knowledge, cited to the passage. The citation is not a lie exactly (the passage is on the topic; it did influence the output), but the specific claim may not be in it. This is a well-documented gap: citation presence is high, citation correctness is meaningfully lower, and the difference does not show up in any metric that counts citations.

So measure attribution directly, on a sample. Take fifty answered questions, and for each cited sentence ask a human (or a judge model you have checked against a human) whether the cited passage actually supports it. The number that comes back is usually lower than the team expected, and it is the number worth quoting when somebody asks how reliable the assistant is.

Design so citation is easy for the model

Small assembly choices change citation quality more than instructions do.

Number the passages in the window, visibly and consistently. [1] through [8], one marker per passage, the same shape every time. Sequential integers in presentation order beat document ids for one mechanical reason: the model has to copy the marker, and a short token adjacent to the text it labels is far easier to copy correctly than doc_8831#4.

Do not renumber between turns. In a conversation, if [3] means something different on turn four than on turn two, earlier citations in the visible history become wrong. Either keep handles stable per conversation, or resolve them to links immediately and never show raw handles twice.

Ask for the citation in the same sentence as the claim, not a bibliography at the end. Trailing lists of sources are almost uncheckable and are where unsupported sentences hide.

Ask for a short quoted span where the format allows it. Requiring a quote makes check three possible, and it changes behaviour: a model that must produce a verbatim span is measurably less likely to state something the passage does not contain.

What to do when verification fails

Decide the policy in advance.

For a bogus handle or a quote that is not in the passage, the safe options are to regenerate once with the failure pointed out, to strip the sentence, or to show the answer with the unsupported sentence marked. What is not acceptable is showing a citation you know is wrong, because the citation is precisely what makes a user stop checking.

Log every verification failure with the request id and the assembly version. The rate is a quality metric that needs no labels, no judges and no evaluation set: it is computed from artefacts you already have, and it moves when your context assembly changes, which makes it one of the most useful regression signals in the whole system.