Dates, durations, and the trouble with midnight

A timestamp is not a feature

Feed a raw timestamp to a model and it becomes a large integer that increases forever. A tree will split it into “before 12 March” and “after”, which is memorising your training period rather than learning anything, and every future date falls beyond the largest value it has ever seen.

Timestamps have to be decomposed. The standard set:

d = df["created_at"]
df["hour"] = d.dt.hour
df["dayofweek"] = d.dt.dayofweek
df["day"] = d.dt.day
df["month"] = d.dt.month
df["is_weekend"] = d.dt.dayofweek >= 5
df["days_since_signup"] = (d - df["signup_at"]).dt.days

The last one is usually the most valuable, and it is the one people forget. Durations beat dates. “Days since last purchase”, “account age in days”, “hours until the deadline” are meaningful for any row at any time. “March 2024” is meaningful only inside your training window.

The midnight problem

Encode hour as 0 to 23 and you have told the model that 23:00 and 00:00 are 23 units apart (the largest possible distance) when they are one hour apart. The same defect appears at December to January, and at Sunday to Monday.

The fix is to place the value on a circle:

import numpy as np
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)

Two columns instead of one, and now 23:00 and 00:00 sit adjacent in the plane. Both columns are needed: sine alone maps 06:00 and 18:00 to the same value, and the cosine is what distinguishes them.

A caveat that is rarely stated. This matters for linear models, distance-based models and neural networks, where continuity is what the model can use. Trees do not need it: a tree can split hour at 22 and again at 5 and reconstruct “night” perfectly well. If you are using boosted trees, plain integer hour is fine and the sine-cosine pair is slightly harmful, because it makes a sharp boundary the tree could have found in one split into a curve it must approximate with several.

Time zones, and the bug that survives review

Store everything in UTC and convert once, deliberately, when you need local behaviour.

But note what “9am” means for the model. If your users are spread across time zones and you use UTC hour, then 09:00 UTC is early morning in London, mid-afternoon in Delhi and the middle of the night in California. The behavioural pattern you are trying to capture (people shop in the evening) is smeared across the whole day and the feature stops working.

Use local time for behavioural features and UTC for ordering, durations and joins. This is a two-line fix that regularly improves a model more than an afternoon of tuning, and the bug is invisible because both versions run without error.

The calendar is not uniform

Real series are shaped by human calendars, and the calendar you need is regional.

• Public holidays, which differ by country and by state. India’s differ by state; the UK’s differ between England, Scotland and Northern Ireland.
• Paydays. In many markets demand spikes at month end and the first of the month.
• Festivals that move. Diwali, Eid and Easter shift by weeks against the Gregorian calendar, so “October” does not consistently mean “Diwali month”. A feature like days_to_diwali works where month cannot.
• School terms, financial year ends, tax deadlines.

The free holidays package covers over 100 countries and takes one line. Building a days_until_next_holiday and days_since_last_holiday pair is often worth more than every other date feature combined for retail and logistics.

Lags and rolling windows

For anything with a time axis, the past values of the target are usually your strongest features:

df["sales_lag_7"] = df.groupby("store")["sales"].shift(7)
df["sales_roll_28"] = (df.groupby("store")["sales"]
                         .shift(1).rolling(28).mean())

Notice the .shift(1) before .rolling(28). Without it, the window includes today’s sales, which you do not know when you are predicting today’s sales. This single missing shift is the most common leakage bug in time series work, it produces a spectacular validation score, and it fails completely in production. If you take one line from this lesson, take that one.

And split by time

Any model with date features must be validated on a period after the training period, never on a random sample of rows. Module 4 covers the mechanics; the reason belongs here, because it is a property of the data rather than of the validation technique: a random split lets the model see Tuesday and Thursday while predicting Wednesday, which is not a situation that will ever occur again.