{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"dates-and-cycles","title":"Dates, durations, and the trouble with midnight","url":"https://addaly.com/learn/machine-learning-foundations/dates-and-cycles","position":26,"of":92,"minutes":8,"answer":"Decompose timestamps into durations and cyclical components, use local time for behavioural features, and shift any rolling window by one period so it cannot include the value you are predicting.","keyIdea":"Decompose timestamps into durations and cyclical components, use local time for behavioural features, and shift any rolling window by one period so it cannot include the value you are predicting."},"body":{"format":"markdown","text":"## A timestamp is not a feature\n\nFeed a raw timestamp to a model and it becomes a large integer that increases forever. A tree will split it into \"before 12 March\" and \"after\", which is memorising your training period rather than learning anything, and every future date falls beyond the largest value it has ever seen.\n\nTimestamps have to be decomposed. The standard set:\n\n```python\nd = df[\"created_at\"]\ndf[\"hour\"] = d.dt.hour\ndf[\"dayofweek\"] = d.dt.dayofweek\ndf[\"day\"] = d.dt.day\ndf[\"month\"] = d.dt.month\ndf[\"is_weekend\"] = d.dt.dayofweek >= 5\ndf[\"days_since_signup\"] = (d - df[\"signup_at\"]).dt.days\n```\n\nThe last one is usually the most valuable, and it is the one people forget. **Durations beat dates.** \"Days since last purchase\", \"account age in days\", \"hours until the deadline\" are meaningful for any row at any time. \"March 2024\" is meaningful only inside your training window.\n\n## The midnight problem\n\nEncode hour as 0 to 23 and you have told the model that 23:00 and 00:00 are 23 units apart (the largest possible distance) when they are one hour apart. The same defect appears at December to January, and at Sunday to Monday.\n\nThe fix is to place the value on a circle:\n\n```python\nimport numpy as np\ndf[\"hour_sin\"] = np.sin(2 * np.pi * df[\"hour\"] / 24)\ndf[\"hour_cos\"] = np.cos(2 * np.pi * df[\"hour\"] / 24)\n```\n\nTwo columns instead of one, and now 23:00 and 00:00 sit adjacent in the plane. Both columns are needed: sine alone maps 06:00 and 18:00 to the same value, and the cosine is what distinguishes them.\n\nA caveat that is rarely stated. This matters for linear models, distance-based models and neural networks, where continuity is what the model can use. **Trees do not need it**: a tree can split hour at 22 and again at 5 and reconstruct \"night\" perfectly well. If you are using boosted trees, plain integer hour is fine and the sine-cosine pair is slightly harmful, because it makes a sharp boundary the tree could have found in one split into a curve it must approximate with several.\n\n## Time zones, and the bug that survives review\n\nStore everything in UTC and convert once, deliberately, when you need local behaviour.\n\nBut note what \"9am\" means for the model. If your users are spread across time zones and you use UTC hour, then 09:00 UTC is early morning in London, mid-afternoon in Delhi and the middle of the night in California. The behavioural pattern you are trying to capture (people shop in the evening) is smeared across the whole day and the feature stops working.\n\nUse **local time for behavioural features** and UTC for ordering, durations and joins. This is a two-line fix that regularly improves a model more than an afternoon of tuning, and the bug is invisible because both versions run without error.\n\n## The calendar is not uniform\n\nReal series are shaped by human calendars, and the calendar you need is regional.\n\n- Public holidays, which differ by country and by state. India's differ by state; the UK's differ between England, Scotland and Northern Ireland.\n- Paydays. In many markets demand spikes at month end and the first of the month.\n- Festivals that move. Diwali, Eid and Easter shift by weeks against the Gregorian calendar, so \"October\" does not consistently mean \"Diwali month\". A feature like `days_to_diwali` works where `month` cannot.\n- School terms, financial year ends, tax deadlines.\n\nThe free `holidays` package covers over 100 countries and takes one line. Building a `days_until_next_holiday` and `days_since_last_holiday` pair is often worth more than every other date feature combined for retail and logistics.\n\n## Lags and rolling windows\n\nFor anything with a time axis, the past values of the target are usually your strongest features:\n\n```python\ndf[\"sales_lag_7\"] = df.groupby(\"store\")[\"sales\"].shift(7)\ndf[\"sales_roll_28\"] = (df.groupby(\"store\")[\"sales\"]\n                         .shift(1).rolling(28).mean())\n```\n\nNotice the `.shift(1)` before `.rolling(28)`. Without it, the window includes today's sales, which you do not know when you are predicting today's sales. This single missing shift is the most common leakage bug in time series work, it produces a spectacular validation score, and it fails completely in production. If you take one line from this lesson, take that one.\n\n## And split by time\n\nAny model with date features must be validated on a period *after* the training period, never on a random sample of rows. Module 4 covers the mechanics; the reason belongs here, because it is a property of the data rather than of the validation technique: a random split lets the model see Tuesday and Thursday while predicting Wednesday, which is not a situation that will ever occur again."},"check":{"question":"A demand model uses a 28-day rolling mean of sales, computed as `rolling(28).mean()` without shifting. Validation MAPE is 4%; in production it is 19%. What is the mechanism?","options":[{"text":"The rolling window is too short to capture seasonality, so production performance degrades as the season changes.","correct":false,"tempting":"Window length affects how much seasonal signal is captured, but it would show up equally in validation rather than producing a five-fold gap."},{"text":"Production data has more missing days, so the rolling mean is computed over fewer observations.","correct":false,"tempting":"Missing data would add noise to the rolling feature, but not create a systematic advantage during validation that disappears afterwards."},{"text":"The unshifted window includes the current day's sales, so during validation the feature partly contains the answer, which is unavailable at prediction time.","correct":true},{"text":"The model was validated on a random split rather than a time-based one, so it saw future periods during training.","correct":false,"tempting":"A real and related error, but the described defect is inside the feature itself and would inflate the score even under a correct time-based split."}]},"prev":{"slug":"outliers-and-what-to-do-with-them","title":"Outliers: error, extreme, or the thing you are looking for","url":"https://addaly.com/learn/machine-learning-foundations/outliers-and-what-to-do-with-them"},"next":{"slug":"text-as-numbers","title":"Text as columns: counting words before you embed them","url":"https://addaly.com/learn/machine-learning-foundations/text-as-numbers"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Dates, durations, and the trouble with midnight', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/dates-and-cycles (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Dates, durations, and the trouble with midnight","courseTitle":"Machine Learning, Foundations","module":{"index":3,"slug":"the-data-is-the-job","title":"The data is the job"},"answer":null,"keyIdea":"Decompose timestamps into durations and cyclical components, use local time for behavioural features, and shift any rolling window by one period so it cannot include the value you are predicting.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-timestamp-is-not-a-feature","runs":[{"text":"A timestamp is not a feature"}]},{"t":"p","runs":[{"text":"Feed a raw timestamp to a model and it becomes a large integer that increases forever. A tree will split it into “before 12 March” and “after”, which is memorising your training period rather than learning anything, and every future date falls beyond the largest value it has ever seen."}]},{"t":"p","runs":[{"text":"Timestamps have to be decomposed. The standard set:"}]},{"t":"code","lang":"python","label":"Python","text":"d = df[\"created_at\"]\ndf[\"hour\"] = d.dt.hour\ndf[\"dayofweek\"] = d.dt.dayofweek\ndf[\"day\"] = d.dt.day\ndf[\"month\"] = d.dt.month\ndf[\"is_weekend\"] = d.dt.dayofweek >= 5\ndf[\"days_since_signup\"] = (d - df[\"signup_at\"]).dt.days"},{"t":"p","runs":[{"text":"The last one is usually the most valuable, and it is the one people forget. "},{"text":"Durations beat dates.","b":true},{"text":" “Days since last purchase”, “account age in days”, “hours until the deadline” are meaningful for any row at any time. “March 2024” is meaningful only inside your training window."}]},{"t":"h2","id":"the-midnight-problem","runs":[{"text":"The midnight problem"}]},{"t":"p","runs":[{"text":"Encode hour as 0 to 23 and you have told the model that 23:00 and 00:00 are 23 units apart (the largest possible distance) when they are one hour apart. The same defect appears at December to January, and at Sunday to Monday."}]},{"t":"p","runs":[{"text":"The fix is to place the value on a circle:"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\ndf[\"hour_sin\"] = np.sin(2 * np.pi * df[\"hour\"] / 24)\ndf[\"hour_cos\"] = np.cos(2 * np.pi * df[\"hour\"] / 24)"},{"t":"p","runs":[{"text":"Two columns instead of one, and now 23:00 and 00:00 sit adjacent in the plane. Both columns are needed: sine alone maps 06:00 and 18:00 to the same value, and the cosine is what distinguishes them."}]},{"t":"p","runs":[{"text":"A caveat that is rarely stated. This matters for linear models, distance-based models and neural networks, where continuity is what the model can use. "},{"text":"Trees do not need it","b":true},{"text":": a tree can split hour at 22 and again at 5 and reconstruct “night” perfectly well. If you are using boosted trees, plain integer hour is fine and the sine-cosine pair is slightly harmful, because it makes a sharp boundary the tree could have found in one split into a curve it must approximate with several."}]},{"t":"h2","id":"time-zones-and-the-bug-that-survives-review","runs":[{"text":"Time zones, and the bug that survives review"}]},{"t":"p","runs":[{"text":"Store everything in UTC and convert once, deliberately, when you need local behaviour."}]},{"t":"p","runs":[{"text":"But note what “9am” means for the model. If your users are spread across time zones and you use UTC hour, then 09:00 UTC is early morning in London, mid-afternoon in Delhi and the middle of the night in California. The behavioural pattern you are trying to capture (people shop in the evening) is smeared across the whole day and the feature stops working."}]},{"t":"p","runs":[{"text":"Use "},{"text":"local time for behavioural features","b":true},{"text":" and UTC for ordering, durations and joins. This is a two-line fix that regularly improves a model more than an afternoon of tuning, and the bug is invisible because both versions run without error."}]},{"t":"h2","id":"the-calendar-is-not-uniform","runs":[{"text":"The calendar is not uniform"}]},{"t":"p","runs":[{"text":"Real series are shaped by human calendars, and the calendar you need is regional."}]},{"t":"ul","items":[[{"text":"Public holidays, which differ by country and by state. India’s differ by state; the UK’s differ between England, Scotland and Northern Ireland."}],[{"text":"Paydays. In many markets demand spikes at month end and the first of the month."}],[{"text":"Festivals that move. Diwali, Eid and Easter shift by weeks against the Gregorian calendar, so “October” does not consistently mean “Diwali month”. A feature like "},{"text":"days_to_diwali","code":true},{"text":" works where "},{"text":"month","code":true},{"text":" cannot."}],[{"text":"School terms, financial year ends, tax deadlines."}]]},{"t":"p","runs":[{"text":"The free "},{"text":"holidays","code":true},{"text":" package covers over 100 countries and takes one line. Building a "},{"text":"days_until_next_holiday","code":true},{"text":" and "},{"text":"days_since_last_holiday","code":true},{"text":" pair is often worth more than every other date feature combined for retail and logistics."}]},{"t":"h2","id":"lags-and-rolling-windows","runs":[{"text":"Lags and rolling windows"}]},{"t":"p","runs":[{"text":"For anything with a time axis, the past values of the target are usually your strongest features:"}]},{"t":"code","lang":"python","label":"Python","text":"df[\"sales_lag_7\"] = df.groupby(\"store\")[\"sales\"].shift(7)\ndf[\"sales_roll_28\"] = (df.groupby(\"store\")[\"sales\"]\n                         .shift(1).rolling(28).mean())"},{"t":"p","runs":[{"text":"Notice the "},{"text":".shift(1)","code":true},{"text":" before "},{"text":".rolling(28)","code":true},{"text":". Without it, the window includes today’s sales, which you do not know when you are predicting today’s sales. This single missing shift is the most common leakage bug in time series work, it produces a spectacular validation score, and it fails completely in production. If you take one line from this lesson, take that one."}]},{"t":"h2","id":"and-split-by-time","runs":[{"text":"And split by time"}]},{"t":"p","runs":[{"text":"Any model with date features must be validated on a period "},{"text":"after","i":true},{"text":" the training period, never on a random sample of rows. Module 4 covers the mechanics; the reason belongs here, because it is a property of the data rather than of the validation technique: a random split lets the model see Tuesday and Thursday while predicting Wednesday, which is not a situation that will ever occur again."}]}],"check":{"question":[{"text":"A demand model uses a 28-day rolling mean of sales, computed as "},{"text":"rolling(28).mean()","code":true},{"text":" without shifting. Validation MAPE is 4%; in production it is 19%. What is the mechanism?"}],"options":[{"text":[{"text":"The rolling window is too short to capture seasonality, so production performance degrades as the season changes."}],"correct":false,"tempting":[{"text":"Window length affects how much seasonal signal is captured, but it would show up equally in validation rather than producing a five-fold gap."}]},{"text":[{"text":"Production data has more missing days, so the rolling mean is computed over fewer observations."}],"correct":false,"tempting":[{"text":"Missing data would add noise to the rolling feature, but not create a systematic advantage during validation that disappears afterwards."}]},{"text":[{"text":"The unshifted window includes the current day’s sales, so during validation the feature partly contains the answer, which is unavailable at prediction time."}],"correct":true,"tempting":null},{"text":[{"text":"The model was validated on a random split rather than a time-based one, so it saw future periods during training."}],"correct":false,"tempting":[{"text":"A real and related error, but the described defect is inside the feature itself and would inflate the score even under a correct time-based split."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"outliers-and-what-to-do-with-them","title":"Outliers: error, extreme, or the thing you are looking for","minutes":8},"next":{"slug":"text-as-numbers","title":"Text as columns: counting words before you embed them","minutes":9}}}