Why it invents a case that does not exist

The case that was not there

In 2023, a lawyer in New York filed a brief citing six court decisions. The opposing side could not find them. Neither could the judge. They did not exist: the chatbot had produced case names, docket numbers, quotations and internal citations, all in the correct format, for judgments that had never been written. Asked directly whether the cases were real, it said yes. The lawyer and his colleague were sanctioned and fined $5,000, and the episode went round the world.

The useful part is not the embarrassment. It is that nothing in the output looked wrong. That is worth understanding mechanically, because if you think invention is a glitch that shows up as odd or hedged text, you will not catch it.

What the model is doing

A language model predicts the next piece of text given everything so far. Trained on an enormous amount of writing, it has learned the shape of things — including the shape of a case citation, a DOI, a clinical guideline reference, an ISO standard number, a statistic in a policy paper.

Ask for a citation supporting a claim, and there is no lookup step. There is no database of judgments being consulted and no moment where the model finds nothing and reports failure. There is only: given "a case supporting this proposition is", what text most plausibly comes next? Something that looks exactly like a citation. A plaintiff and an airline. A plausible volume and page. A quotation in the register judges use.

The same process produces the true citations. When a case is famous enough to appear thousands of times in the training data, the most probable continuation happens to be correct. When it is not, the most probable continuation is a well-formed invention. From the outside, in the text, these two outcomes are indistinguishable — because they were made the same way.

That is the whole lesson. Fluency is not a signal of knowledge, because fluency is the thing being optimised.

Where invention concentrates

Knowing the mechanism tells you where to look, which is more useful than a general instruction to be careful.

• Anything with an identifier. Case numbers, DOIs, ISBNs, standard numbers, statute sections, page numbers, product codes. Highly patterned, so easily fabricated.
• Precise figures attached to a source. "According to a 2024 WHO report, 38% of…" The structure of that sentence is far more common in the training data than any particular number in it.
• Named specifics in your local world. Which official signed the circular, what your state's current fee is, whether that supplier is registered. Rare in training data, plausible-sounding when generated.
• The gaps in a mostly correct answer. Four right, one wrong is the classic shape, and it is the dangerous one, because the four correct entries buy your trust for the fifth.
• Anything you asked for a fixed number of. "Give me five examples" applies pressure to produce five. If four exist, the fifth gets made. Ask for "up to five, and say if there are fewer" and this improves noticeably.

Why "are you sure?" does not help

The obvious move is to ask the model to verify itself. It sometimes works, and it cannot be relied on, for a reason worth stating plainly: the check is run by the same process that produced the error. There is no separate faculty inside it that knows what is real. Ask "are you certain?" and you have changed the prompt, which changes the most probable continuation — often to an apology, sometimes to a different invention, occasionally to a correct retraction. You have no way to tell which you got.

Tools that search the live web are genuinely better here, because a retrieval step has been added and a real document is fetched. They introduce a subtler failure instead: a real link attached to a claim the linked page does not make. That is covered later in the course, and the check is the same one — open the link and read the sentence.

The habit that actually protects you

Not "be sceptical". Sceptics get caught too. A rule:

Every specific you did not supply is unverified until you verify it. Names, numbers, dates, quotations, citations, clause references, legal thresholds, dosages, prices.

Practically, that becomes a two-minute pass before anything leaves your hands. Read the draft and mark every proper noun, every digit and every quotation mark. Those marks are your checking list. Everything else — structure, argument, tone, transitions — you can judge by reading it, because you have the expertise to judge it.

This is also why the strongest working pattern in this whole course is to put the source material in yourself. When the model is summarising a document you attached, the specifics come from the document and invention drops sharply. It does not vanish — that is the next module — but the failure mode changes from "made up" to "misread", and misreading is something you already know how to catch.