When the model does not know

There is no empty answer

A language model produces the most likely continuation of its context. There is no state in which it produces nothing. If the answer is not available, the most likely continuation is a plausible-looking answer, because that is what text answering a question looks like.

This is why hallucination is a slightly misleading word. The model is not malfunctioning; it is doing exactly what it does, on an input where the correct behaviour would have been to stop. Saying I do not know is a specific behaviour that has to be trained in and then made likely by the prompt and the format, and small models have less of it.

Confidence is only weakly informative

The obvious idea is to read the model’s own probabilities. It half works.

Token probabilities correlate with correctness, but weakly and inconsistently, and instruction tuning makes it worse: preference tuning rewards confident-sounding answers, which pushes probabilities up regardless of correctness. A model can assign 0.94 to a fabricated citation.

Where logprobs are useful is on constrained outputs. If the model must choose between five enum values, the probability mass over those five is meaningful, and a top choice at 0.4 against a runner-up at 0.35 is a real signal that this item needs a human. For free-form text, treat logprobs as a weak prior, not a confidence score.

Five things that work, in order of return

1. Give it the information. Most confabulation is a knowledge problem, and retrieval solves knowledge problems that no model size solves reliably. A 4B model with the right paragraph beats a 70B model guessing, and it is cheaper.

2. Make abstention available in the format. If the schema requires an answer, the model must invent one. Give it somewhere to go:

{"answer": {"type": ["string","null"]},
 "evidence_quote": {"type": ["string","null"]},
 "status": {"enum": ["answered","not_in_source","ambiguous"]}}

Then say plainly in the prompt that not_in_source is a correct and expected answer. Small models need explicit permission; without it, abstention has essentially zero probability.

3. Require a quotation. Ask for the exact span from the source that supports the answer, and verify in code that the span appears in the source. This is a hard check, not a soft one (a fabricated quote fails a string search) and it converts an invisible failure into a caught one. It is the single most effective technique in this list after retrieval.

4. Self-consistency. Sample five completions at temperature 0.7 and compare. Agreement across samples correlates with correctness considerably better than any single answer’s confidence. Disagreement is a reliable flag for review. It costs five times as much, so apply it to the items that matter rather than to everything.

5. A verification pass. A second call, or a second model, given the source and the proposed answer and asked only whether the source supports it. Cheaper than it sounds, because the verification prompt is short and the answer is one token.

Measure it, with an unanswerable set

Build a small set that makes fabrication visible:

• 20 questions whose answers are in your documents.
• 20 questions that look answerable but are not: a plausible field that the document does not contain, a person not mentioned, a date not given.

Score two numbers separately: accuracy on the answerable half, and abstention rate on the unanswerable half. A model at 90% accuracy that abstains on 10% of the unanswerable questions is worse for most products than one at 82% accuracy that abstains on 85%, because the second one’s errors are visible.

This second number is the one nobody measures and the one that decides whether people can trust the system.

Say it in the interface

The final piece is not technical. If the system can be wrong, the interface should show its source and make checking easy: the quotation, a link, the document name. A confident sentence with no provenance invites trust the system has not earned; the same sentence with the supporting line beside it invites the reader to do the one check that catches the failure.