Features beat models

A feature is a number you decided to hand over

The model sees only what you give it. Not the customer, not the flat, not the sentence: a row of numbers you chose to compute. That choice is called feature engineering, and on ordinary table-shaped data it moves results more than the algorithm does.

A concrete case. You want to predict whether a student finishes a free online course. Your database has:

user_id, signup_timestamp, country, device, lesson_events[]

Handed over raw, signup_timestamp is a number like 1772841600. The model can only ask questions like “is it above 1772000000”, which means “did they sign up after a particular Tuesday”. Almost useless.

Now derive features from the same column:

• hours_between_signup_and_first_lesson: hesitation predicts dropout.
• sessions_in_first_week: the strongest single predictor in most course data.
• signed_up_on_weekend: different intent.
• local_hour_of_signup: converted to the student’s timezone, not the server’s.

Same raw column. One version carries almost no signal; the other carries most of it. No change of algorithm does that.

Make the number mean something

Models do not know context, so put the context in the number.

• Predicting whether a flat will rent quickly? price is weak. price ÷ median price of flats within 2 km is strong, because the model no longer has to learn every neighbourhood’s price level from scratch.
• Predicting delivery delay? Raw latitude and longitude force the model to carve the map into boxes. distance_to_depot_km and is_inside_ring_road say it directly.
• Predicting churn? payments_made is confounded by tenure. payments_made ÷ months_subscribed is not.

Each of these is arithmetic a nine-year-old could do. Each is usually worth more than swapping in a fancier model.

The honest limit of this claim

This is true for tabular data: rows and columns, the sort of thing that starts life in a spreadsheet or a SQL table. It is not true for images, audio and text.

For thirty years, computer vision was largely the craft of hand-designing features: edge detectors, corner detectors, gradient histograms. Deep learning won that field by learning the features from raw pixels instead. Nobody hand-engineers features for image classification now, and you should not start.

So the rule with its boundary: on tabular data, spend your time on features; on perceptual data, let the network learn them. Lesson 9 shows why those are the same statement viewed from two sides.

Leakage: the feature that already knows

One failure mode deserves its own section, because it produces spectacular results and is invisible in every standard check.

Leakage is a feature that contains information which will not exist at prediction time. The model uses it, the score is wonderful, and the system fails on the first real request.

Examples that have all happened:

• A hospital mortality model with discharge_destination among its features. One value is “mortuary”.
• A loan-default model with collections_agency_assigned. Agencies are assigned after default.
• A support-escalation model with number_of_agent_replies. Replies accumulate because the ticket is escalating.
• A conversion model where the price column was written back after checkout, so unconverted sessions had a null price.

A correct train/test split does not catch any of these. The split protects you against memorising rows. It does nothing about a column that arrives from the future, because the leak is present on both sides of the split.

The test is a question, and you have to ask it about every column:

At the exact moment I need this prediction, would this value already exist, with the value it has here?

Both halves matter. number_of_agent_replies exists at prediction time (it is 0 or 1) but not with the value in your training table. That is still leakage.

A working order

1. Write down when the prediction has to be made. An actual moment: ticket created, checkout loaded, application submitted.
2. List only the fields that exist at that moment.
3. Derive features that encode what you would look at as a human.
4. Fit the simplest model that can use them, and read the result as a measurement of your features.
5. Only then start comparing algorithms.

Step 4 is the one people skip. A weak score from a simple model on good features tells you something real. A strong score from a complicated model on unexamined features tells you almost nothing.