The stale document problem

The failure that looks like success

A grounded assistant is accurate for four months. Everyone stops checking, reasonably, because it has earned it.

Then the delivery charge changes and the document does not. From that day, every answer about delivery is wrong: fluently, with a citation, in the house voice. There is no error message, no hesitation, no drop in quality of any kind that a reader could notice. The system has become a machine for repeating an outdated fact at scale.

This is the most common way a working AI setup turns into a liability, and it is entirely a process problem rather than a technical one.

Where staleness comes from

Nobody owns the document. It was written once, by whoever set the system up, and it belongs to no one.

The change happened somewhere else. You updated the price in the shop, on the menu and in the till. The facts sheet was the fourth place and nobody thought of it.

Superseded versions were left in the folder. terms-v2.pdf and terms-final.pdf both sit in the library, and retrieval has no idea which is current. This is the one that produces contradictory answers depending on how the question is phrased.

The source was never true. A procedure describing how things worked before the shop moved. Nobody noticed, because nobody reads their own procedures.

Four controls

Date every document, at the top, in the text. Not the file’s modified date, which changes when somebody opens it. A line in the body: Current as of 12 August 2026. The model can then be told to say when its source was last updated, which surfaces staleness in the answer itself.

One owner per document, named in it. A person, not a role. In a business of four people this feels excessive and is the reason it gets done at all.

Delete, do not archive in place. Superseded documents leave the library. Keep them somewhere else if you need the history. A file labelled OLD is not labelled at all as far as retrieval is concerned.

Put a change on your existing checklist. When prices change, the facts sheet is on the same list as the menu and the website. This is the control that actually works, because it attaches the task to something that already happens.

The quarterly twenty minutes

Once a quarter, take your ten most-asked questions and run them. Read the answers as a customer would, and check each against reality.

You will typically find one wrong answer per quarter in an active business. That is not a sign of a bad system; it is what a business that changes things looks like. Finding it in a twenty-minute review is a great deal cheaper than finding it when a customer holds you to it.

Keep a note of what you found and when. Six months of that note tells you how fast your material goes stale, which tells you whether quarterly is often enough. A shop with fixed prices might manage twice a year. A business whose costs move with an exchange rate needs it monthly.

When something changes today

Two things, in this order. Fix the document. Then run the two or three questions that touch it, to confirm the answers changed.

The second step catches the case where the tool has cached the old version, which some do, or where a superseded copy is still being retrieved in preference to the one you just edited. Both are common enough to be worth the sixty seconds, and both are invisible if you only check that the file looks right.

The other kind of staleness

Documents go out of date. So does the world the documents describe, in ways no edit will fix.

A procedure written when you had two staff assumes two staff. A price list built around a supplier you no longer use still has their part numbers. A frequently-asked-questions page answers the questions people asked in 2024, and the questions have changed because your customers have.

This is why the quarterly review starts with the ten most-asked questions rather than with the documents. Questions reveal drift that reading your own material never will, because you read your own material remembering what it was meant to say.

Who should do the review

Not the person who wrote the documents. They will read what they intended.

The best reviewer is whoever answers customers most: they know what is actually being asked and they will notice immediately when an answer does not match what they say on the phone. In a business of two, swap: each person reviews the other’s material. In a business of one, leave a week between writing and reviewing, and read the answers on a phone rather than a computer. Both tricks work by making the material slightly unfamiliar, which is the whole trick.