Leakage: when a feature knows the future

The model that scored 99% and failed on Monday

A churn model is trained, evaluated on held-out rows, and reports 0.99 accuracy. It goes live and predicts almost nothing useful. Nothing in the model is broken. One of its features contained the answer, in a form that was available in the training table and is not available on the day a prediction is needed. That is leakage, and in tabular work it almost always enters through a join or an aggregate in the feature query.

Five ways it gets in

An aggregate over all history. sum(amount) AS total_spend computed over every order the customer ever placed, including the ones after as_of. A customer whose total spend keeps rising after the snapshot is, by definition, not churning. The model learns “high total spend means not churned” and is right in training and useless live, where the future orders do not exist yet. The previous lesson’s o.placed_at < s.as_of is the fix, on every join, every time.

A join to a current dimension value. customers.segment is joined in for every snapshot, and the segment is set to 'lapsed' by a nightly job when a customer has not ordered for 60 days. For a snapshot from last year, the segment now says what the customer became, which is the label. The as-of join from the time module is the fix: the segment that was true on as_of, from a history table, or no segment at all if no history exists.

A column that is updated after the outcome. last_contacted_at, account_status, refund_count, support_tickets. Each is edited by a process that happens because of the outcome: the retention team contacts customers who look like leaving, the refund is issued when they complain on the way out. Any column that can change after as_of is suspect unless its history is stored and joined as of.

Target encoding on the whole table. The categories lesson computed avg(churned) per city and used it as a feature. If that average includes the row it is then attached to, the row’s own label has leaked into its feature, a little for a big city and a lot for a city with three customers. Compute it on training rows only, and exclude the current row: (sum(churned) - churned) / (count(*) - 1) over the group.

The same customer on both sides of the split. Rows for one customer at January and February snapshots are near-duplicates. Put one in training and one in test, and the model is evaluated on rows it has effectively seen. The split must be by customer, or by time with all training snapshots before all test snapshots, which is also the split that resembles production.

Detecting it

Some leaks announce themselves. A feature with an almost perfect correlation with the label is a leak until proved otherwise; real signals are weaker than that. A model that is far better than the previous one after a change to the feature query is suspect in proportion to the improvement.

Others can be found mechanically. For each feature, the query should be able to state the source table and the timestamp column it was bounded by. A check query asserts it:

-- must return no rows: no feature source row is at or after its snapshot
SELECT s.customer_id, s.as_of, max(o.placed_at) AS latest_used
FROM snapshots s JOIN orders_used_for_features o USING (customer_id, as_of)
GROUP BY 1, 2
HAVING max(o.placed_at) >= s.as_of;

Run it beside the feature query, in the same file as the validation checks from earlier in this module.

The leaks a timestamp check cannot see

A notes field that a support agent fills in only when a customer calls to cancel. A data_entry_timestamp that is populated only for accounts that went through a manual review, which happens mostly to accounts about to be closed. The timestamps of these columns are before as_of, and they still carry the future, because the fact that they exist is caused by the outcome. No query can catch this. It is found by a person who knows how the data is produced reading the feature list and asking, of each column, “what process writes this, and does that process know the outcome?” That conversation is the most valuable hour in a modelling project, and it is not a technical hour.

The habit

Before a feature table is used, two things happen. The timestamp check runs and returns nothing. And somebody who understands the source systems reads the column list and signs off that no column is written by a process downstream of the outcome. The second is the one that gets skipped, and it is the one that would have caught the 99%.