The mistake that makes a model look brilliant

The moment to be suspicious

Your model predicts which quotes convert, and it is right 96% of the time.

That is not a triumph. In a small business, that is a symptom. Genuine prediction of human decisions lands somewhere between slightly-better-than-guessing and moderately good, and anything approaching certainty means the model has found something it should not have.

Almost always it has found a column that only exists once the answer is known.

What leakage looks like in a real export

The quote conversion model. The export includes deposit_paid_date. Only converted quotes have one. The model learns: deposit date present, therefore converted. Perfect score, zero value: at the moment you want a prediction, no deposit has been paid.

The lapse model. The export includes reactivation_email_sent. You only send those to customers you already believed were lapsing. The model has learned to read your own past judgement.

The late-payment model. The export includes reminder_count. Reminders are sent because an invoice is late.

The no-show model. The export includes cancellation_reason, blank for attended appointments.

Each of these is an ordinary column, present for good operational reasons, and each turns the exercise into an elaborate way of reading the answer off the back of the page.

The test, which is one question

For every column you hand the model, ask: would this value have existed, in this state, at the moment I want the prediction?

If the prediction is made the day a quote is sent, then anything recorded after that day is out. Deposit dates, follow-up notes, the final invoice amount, the job completion date: all out, however informative they look.

This is why the timestamp of each field matters, and why exports that give you a record’s current state are dangerous. Your CRM shows the customer as they are now. The model needs them as they were then, and reconstructing that is harder than anything else in this module.

The subtler version, which catches careful people

Even with no obviously offending column, leakage arrives through time.

Splitting your data randomly rather than by date lets the model learn from March while predicting February. It cannot do that in real life, and the score it produces is a fantasy. Always split by time for anything with a time dimension, as in the baseline lesson.

It also arrives through cleaning. If you fill missing values using the average of the whole dataset, the training rows have absorbed information from the test rows. Compute your averages on the earlier data only. This is a small thing that inflates scores by a surprising amount.

And it arrives through duplicates. The same customer appearing in both the training and test sets, under two records, means the model is being tested on someone it has met.

What to do when you find one

Remove the column, rerun, and expect the score to collapse. From 96% to 71% is normal, and 71% may well be useful.

Resist the pull to find some way of keeping the column. The instinct is strong, because the good number felt like progress. It was not progress; it was a measurement of nothing, and the only thing lost by removing it is a figure that would have embarrassed you later.

Building the record as it was, cheaply

The full solution is to store a snapshot of each customer at the moment of every decision, which is more bookkeeping than most small businesses want.

There is a cheap approximation that works well enough. Pick a cut-off date: say the end of last March. Build every input column using only data dated before it, and build the outcome using only data dated after it. Nothing from the later period can reach the earlier columns, because you filtered by date rather than by column.

It is coarse, it wastes some data, and it is honest, which puts it ahead of almost every model a small business will otherwise build.

Why this lesson exists in a small-business course

Because you will meet leakage in bought products, not just in your own work.

When a vendor shows a strikingly good accuracy figure, ask which fields the model uses and when each one becomes known. A vendor who can answer precisely has done the work. A vendor who says the model considers hundreds of signals has told you they are not going to answer, and the number on the slide is now yours to disbelieve.

The rule generalises beyond models. Any claim that something predicts an outcome deserves the question: was that fact available before the outcome, or did it arrive with it.