Where invention concentrates, and why it never says “I don’t know”

There is no second mode

The common mental picture is that a model retrieves when it knows and invents when it does not. There is no such switch. It performs one operation (produce the likeliest continuation) and the quality of the output depends entirely on how much support the training gave it.

Where support is dense, the likeliest continuation is the true one. Where support is thin, the likeliest continuation is the plausible one. Same operation, same fluency, same confident tone. The model has no way to notice which side of that line it is on, because noticing would require a second system watching the first, and there isn’t one.

This is why hallucination is not a bug that a future release removes. It is the behaviour of the mechanism at the edge of its knowledge.

Where invention clusters

The useful thing is that it clusters predictably. Six places, in rough order of danger:

1. Citations and references. Authors, years, journals, page numbers, DOIs. The shape is learnable without any real document behind it.
2. URLs. Confidently constructed and frequently dead, because a plausible web address is trivially generated.
3. Numbers and dates. Statistics, populations, prices, effective dates. A specific figure looks authoritative and costs the model nothing.
4. Quotations. Attributed to real people, phrased as they might have phrased it, never said.
5. Named minor entities. A small company, a local regulation, a village clinic, a niche product. Enough context exists to write about it; not enough to write about it accurately.
6. Anything at the boundary of a real subject. The general framework is right and the specifics inside it are invented: the most dangerous shape, because the correct scaffolding lends credibility to the fabricated detail.

Notice what these have in common. Every one is a specific detail inside a familiar structure. That is exactly where a next-token machine is strongest at form and weakest at fact.

The specificity signal

Here is a counter-intuitive tell worth internalising.

Invented material is often tidier than real material. A real answer has ragged edges: the study had 1,247 participants, ran in two of the four planned sites, and reported a confidence interval that overlaps zero. An invented one has 1,200 participants, four sites, and a clean result.

When a model’s answer is unusually neat (round numbers, symmetric structure, every fact conveniently supporting the point) treat the neatness as a warning rather than as competence. Reality is lumpier than the likeliest continuation.

Why it guesses instead of abstaining

Two reasons, and the second is the interesting one.

First, the training text. Documents that say “I do not know” are rare; documents that answer are the norm. The pattern being learned is: question, then answer.

Second, the way models are scored. A paper from OpenAI in 2025 made this argument directly: hallucination persists partly because the benchmarks used to evaluate models mark answers right or wrong with nothing in between. Under that scoring, a guess has positive expected value and an abstention has none. A model that says “I am not sure” scores zero; a model that guesses scores sometimes. Optimise against those scoreboards for years and you get a system that always answers.

That framing is useful because it points at a fix that is not about model architecture at all. If evaluations gave partial credit for well-placed uncertainty, the incentive would change. Some are starting to.

Three checks that cost almost nothing

Ask it three times, in three fresh conversations. Invented specifics vary between runs, because they are sampled rather than recalled. Well-supported facts come back the same. If you get three different publication years for the same paper, none of them is real. This is the single most efficient test in this course and it takes two minutes.

Check the most specific claim first. Not the general one. If the citation, the figure or the URL is wrong, the surrounding paragraph does not deserve trust either.

Ask for the reasoning before the answer, and ask what would make the answer wrong. It does not make the model honest, but the second question often surfaces the conditions under which the first answer breaks down, and those conditions are frequently exactly your situation.

What not to do

Do not ask “are you sure?” as a verification method. It is not a check on anything internal; it is a request for text about certainty, and under preference training the likely response is either warm reassurance or a reversal, depending on your tone. You will learn about the model’s manners and nothing about the fact.