Error analysis as a weekly habit

The hour that decides the week

Once a week, one person, one hour. They open the failures (thumbs-downs from the feedback queue, cases the harness failed, traces the judge scored low) and read thirty to fifty of them, one at a time, labelling each with a root cause. At the end they have a count by cause, and that count is what the team works on next. The evaluation course teaches this as a structured practice; this lesson is its shape for a builder with one feature and one hour.

The reason it works is that a prompt is the easiest thing in the system to change, and so teams change it first, whatever the fault. The count stops that. If forty per cent of failures are the right chunk not being retrieved, nothing done to the prompt this week will move the number, and the count says so before the week is spent.

A short list of causes

Keep the categories few and fixed, and use the same ones every week so counts are comparable:

• retrieval-miss: the answering chunk was not in the candidates.
• retrieval-rank: it was retrieved but cut by the send limit.
• ingestion: the chunk existed and was garbage: a table without headers, a page footer, a truncated paragraph.
• model-ignored-context: the passage was sent and not used.
• model-invented: a fact stated that appears nowhere in the context.
• tool-error: a tool failed and the model wrote around it.
• tool-misuse: the model called the right tool with wrong arguments.
• prompt-ambiguity: the instructions permit the behaviour that failed.
• out-of-scope: the input asked for something the feature does not do.
• user-misread: the answer was right and the user misunderstood it.
• expectation-wrong: the case or the complaint was itself mistaken.
• injection: the input contained an instruction the model followed.
• other.

When “other” passes fifteen per cent, a category is missing; name it and add it. The trace from the previous lesson is what makes labelling possible in under two minutes per failure: without the retrieved ids and the tool calls, half these categories are indistinguishable.

Count, then fix the biggest

Sort the counts. The largest category is the week’s work, and it is usually not the one the loudest complaint pointed at. A distribution of retrieval-miss 18, ingestion 9, model-ignored-context 7, expectation-wrong 6, everything else in ones and twos says: the retrieval module, this week; the prompt, not yet.

“Expectation wrong” is a real category

Some share of failures (a tenth to a fifth is common) turn out to be the case being wrong, the policy having changed, or the complaint being mistaken. Labelling these honestly, rather than forcing them into a system category, keeps the other counts true, and fixing the case is real work: a case with a wrong expectation fails good changes forever.

Slice before you conclude

An overall failure rate hides everything interesting. Label each failure with the tags from the case file (language, intent, user segment) and look at the counts per slice. An assistant that is fine overall and poor in Hindi shows up only here, and the fix is different from the fix for the aggregate. This is the reason the case file carried tags from the start.

From label to case

Every labelled failure that is not “expectation-wrong” or “user-misread” becomes a regression case, with its category as a tag and its trace id as provenance. A healthy feature adds twenty to forty cases a month this way, and the suite from the start of this block stops being fifty cases and becomes the memory of everything that ever went wrong.

Why a person, and where a model helps

A model can pre-sort five hundred failures into clusters (embed the judge’s reasons, cluster, show a sample from each) and that is useful when the queue is long. But the labels must be read by a person, because the categories change as the product does, and a model asked to label with last month’s list will file a new failure mode under an old name and hide it. The hour is not overhead. It is the only time anyone looks at what the system actually did.

The trap: fixing the example

A failure on “what is the refund window for annual plans?” is fixed in ten minutes by adding that fact to the system prompt. It is a patch. The class is “policy facts that retrieval does not surface”, the fix is in ingestion or ranking, and the next week’s failure is the same class with a different fact. When you label, ask what the class is, and when you fix, fix the class. The count of that class next week is how you know whether you did.

The write-up

Half a page: counts by category, three representative examples with trace links, the one change proposed, the case ids added. Sent to the team, kept in the repository. After six months those pages are the history of the product’s quality, and the answer to the question a new engineer asks first: what usually goes wrong here?