{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"the-whole-loop","title":"The whole loop, once, before we slow down","url":"https://addaly.com/learn/machine-learning-foundations/the-whole-loop","position":8,"of":92,"minutes":10,"answer":"The whole discipline is nine steps with one irreversible arrow: the test set is touched once, at the end, after every decision has been made without it.","keyIdea":"The whole discipline is nine steps with one irreversible arrow: the test set is touched once, at the end, after every decision has been made without it."},"body":{"format":"markdown","text":"## A map you can hold in your head\n\nEverything in the next eight modules is one of nine steps. Seeing them together now means each later lesson lands somewhere rather than floating.\n\n1. **Frame.** What decision, what row, what target, what moment. From the previous lessons.\n2. **Collect and split.** Get the data, and put a portion away before you look at any of it.\n3. **Explore.** Distributions, missingness, obvious errors, the target's base rate.\n4. **Engineer features.** The bulk of the work. Module 3.\n5. **Choose a model family and fit.** Modules 2, 6, 8.\n6. **Tune against validation.** Module 4.\n7. **Evaluate on test, once.** Module 5.\n8. **Deploy.** Module 9.\n9. **Monitor, and go back to step 1.** Also module 9.\n\nThe arrows go both ways between 3 and 6 many times. They go one way into 7, and that is the discipline the whole field is built on.\n\n> Figure: The loop, with its one irreversible arrow. Steps: 1. Frame (Decision, row, target, moment); 2. Split (Put the test rows away before looking); 3. Explore and engineer (Distributions, missingness, features: most of the work); 4. Fit and tune (Model family and hyperparameters, on validation only); 5. Test, once (The single-use measurement); 6. Deploy and monitor (Then back to the framing). Exploring, engineering and tuning cycle many times, all against validation data. The arrow into the test set goes one way and is followed once, after every decision has been made without it.\n\n## The same loop in thirty lines\n\nThis runs on a free Colab or Kaggle notebook, on a laptop with 4 GB of RAM, and it contains a version of every step above.\n\n```python\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split, cross_val_score\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.preprocessing import StandardScaler, OneHotEncoder\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.metrics import roc_auc_score\n\ndf = pd.read_csv(\"applications.csv\")\ny = df.pop(\"defaulted_90d\")\n\n# 2. split first, look second\nX_tr, X_te, y_tr, y_te = train_test_split(\n    df, y, test_size=0.2, stratify=y, random_state=0)\n\nnum = X_tr.select_dtypes(\"number\").columns\ncat = X_tr.select_dtypes(\"object\").columns\n\nprep = ColumnTransformer([\n    (\"num\", Pipeline([(\"imp\", SimpleImputer(strategy=\"median\")),\n                      (\"sc\", StandardScaler())]), num),\n    (\"cat\", OneHotEncoder(handle_unknown=\"ignore\"), cat),\n])\n\npipe = Pipeline([(\"prep\", prep),\n                 (\"model\", HistGradientBoostingClassifier(random_state=0))])\n\n# 5-6. fit and tune using cross-validation on TRAIN only\nprint(\"cv auc\", cross_val_score(pipe, X_tr, y_tr, cv=5, scoring=\"roc_auc\").mean())\n\n# baseline for comparison\nbase = DummyClassifier(strategy=\"prior\").fit(X_tr, y_tr)\nprint(\"baseline auc\", roc_auc_score(y_te, base.predict_proba(X_te)[:, 1]))\n\n# 7. test set, once, at the end\npipe.fit(X_tr, y_tr)\nprint(\"test auc\", roc_auc_score(y_te, pipe.predict_proba(X_te)[:, 1]))\n```\n\nRead it for the shape, not the details. Four things in there are load-bearing and worth naming now.\n\n**The split happens on line 12, before any exploration.** Every transformation after it is fitted inside a `Pipeline`, which means the median used for imputation and the mean used for scaling are computed on training rows only. Do that by hand instead and you will one day compute the median on the whole dataset, leak test information into training, and get a score you cannot reproduce in production.\n\n**`handle_unknown=\"ignore\"`.** A category appears in production that was not in training. Without this line, the pipeline raises and your service returns a 500. With it, that row gets all-zeros for the category block and a prediction still comes out. Neither is obviously right; the point is that you chose.\n\n**Cross-validation runs on training data only.** The test set is not involved in any decision.\n\n**The test line appears once, at the end, after everything else is settled.**\n\n## Where the time actually goes\n\nIf you have read that machine learning is 80% data cleaning, the number is folklore but the direction is right. A more useful breakdown, from projects that shipped:\n\n- Deciding the question, the row and the target: more than people expect, and it is the highest-leverage time in the project.\n- Getting the data and making it trustworthy: the largest single block, and the least visible.\n- Feature engineering: the second largest, and where most accuracy is won on tabular problems.\n- Model selection and tuning: usually far smaller than the excitement about it suggests. `HistGradientBoostingClassifier` with default settings is a strong model; tuning typically buys single-digit percentage improvements over a sensible default.\n- Evaluation, deployment and monitoring: the part that decides whether the work survives a year.\n\nThe first three of those five are covered in modules 1, 3 and 5. That is deliberate.\n\n## What to do with this lesson\n\nRun the code on any dataset you can find: the Titanic set, a CSV of your own expenses, anything with a column you could try to predict. It will fail in some interesting way. Getting it to run once, end to end, on your own machine or a free notebook, is worth more than the next three lessons read passively, and it gives every later lesson something concrete to attach to."},"check":{"question":"In the pipeline above, why does putting the imputer and scaler inside `Pipeline` rather than applying them to the full dataframe first actually change the resulting numbers?","options":[{"text":"It ensures the transformations run in the correct order, which manual application can get wrong.","correct":false,"tempting":"Ordering is a real convenience of pipelines, but applying the same steps manually in the same order would still produce different numbers, so ordering is not the mechanism."},{"text":"It makes the code shorter and reusable, so there is less chance of a typo between the training and prediction paths.","correct":false,"tempting":"Describes an engineering benefit rather than a statistical one; the numbers would differ even in typo-free manual code."},{"text":"It allows the same object to be serialised for deployment, so training and serving stay in sync.","correct":false,"tempting":"True and important for module 9, but this benefit is about production drift rather than about the validation score being honest."},{"text":"Inside a pipeline, the median and the mean are computed from training rows only, so no information from the held-out rows reaches the model.","correct":true}]},"prev":{"slug":"when-not-to-use-machine-learning","title":"The problems where the right answer is a rule","url":"https://addaly.com/learn/machine-learning-foundations/when-not-to-use-machine-learning"},"next":{"slug":"a-toolkit-that-costs-nothing","title":"A working setup that costs nothing","url":"https://addaly.com/learn/machine-learning-foundations/a-toolkit-that-costs-nothing"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The whole loop, once, before we slow down', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/the-whole-loop (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The whole loop, once, before we slow down","courseTitle":"Machine Learning, Foundations","module":{"index":1,"slug":"what-learning-is","title":"What learning is, and what it is not"},"answer":null,"keyIdea":"The whole discipline is nine steps with one irreversible arrow: the test set is touched once, at the end, after every decision has been made without it.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Frame","detail":"Decision, row, target, moment"},{"label":"Split","detail":"Put the test rows away before looking"},{"label":"Explore and engineer","detail":"Distributions, missingness, features: most of the work"},{"label":"Fit and tune","detail":"Model family and hyperparameters, on validation only"},{"label":"Test, once","detail":"The single-use measurement"},{"label":"Deploy and monitor","detail":"Then back to the framing"}],"title":"The loop, with its one irreversible arrow","caption":"Exploring, engineering and tuning cycle many times, all against validation data. The arrow into the test set goes one way and is followed once, after every decision has been made without it."},"alt":"Figure: The loop, with its one irreversible arrow. Steps: 1. Frame (Decision, row, target, moment); 2. Split (Put the test rows away before looking); 3. Explore and engineer (Distributions, missingness, features: most of the work); 4. Fit and tune (Model family and hyperparameters, on validation only); 5. Test, once (The single-use measurement); 6. Deploy and monitor (Then back to the framing). Exploring, engineering and tuning cycle many times, all against validation data. The arrow into the test set goes one way and is followed once, after every decision has been made without it."}},"blocks":[{"t":"h2","id":"a-map-you-can-hold-in-your-head","runs":[{"text":"A map you can hold in your head"}]},{"t":"p","runs":[{"text":"Everything in the next eight modules is one of nine steps. Seeing them together now means each later lesson lands somewhere rather than floating."}]},{"t":"ol","start":1,"items":[[{"text":"Frame.","b":true},{"text":" What decision, what row, what target, what moment. From the previous lessons."}],[{"text":"Collect and split.","b":true},{"text":" Get the data, and put a portion away before you look at any of it."}],[{"text":"Explore.","b":true},{"text":" Distributions, missingness, obvious errors, the target’s base rate."}],[{"text":"Engineer features.","b":true},{"text":" The bulk of the work. Module 3."}],[{"text":"Choose a model family and fit.","b":true},{"text":" Modules 2, 6, 8."}],[{"text":"Tune against validation.","b":true},{"text":" Module 4."}],[{"text":"Evaluate on test, once.","b":true},{"text":" Module 5."}],[{"text":"Deploy.","b":true},{"text":" Module 9."}],[{"text":"Monitor, and go back to step 1.","b":true},{"text":" Also module 9."}]]},{"t":"p","runs":[{"text":"The arrows go both ways between 3 and 6 many times. They go one way into 7, and that is the discipline the whole field is built on."}]},{"t":"h2","id":"the-same-loop-in-thirty-lines","runs":[{"text":"The same loop in thirty lines"}]},{"t":"p","runs":[{"text":"This runs on a free Colab or Kaggle notebook, on a laptop with 4 GB of RAM, and it contains a version of every step above."}]},{"t":"code","lang":"python","label":"Python","text":"import pandas as pd\nfrom sklearn.model_selection import train_test_split, cross_val_score\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.preprocessing import StandardScaler, OneHotEncoder\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.ensemble import HistGradientBoostingClassifier\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.metrics import roc_auc_score\n\ndf = pd.read_csv(\"applications.csv\")\ny = df.pop(\"defaulted_90d\")\n\n# 2. split first, look second\nX_tr, X_te, y_tr, y_te = train_test_split(\n    df, y, test_size=0.2, stratify=y, random_state=0)\n\nnum = X_tr.select_dtypes(\"number\").columns\ncat = X_tr.select_dtypes(\"object\").columns\n\nprep = ColumnTransformer([\n    (\"num\", Pipeline([(\"imp\", SimpleImputer(strategy=\"median\")),\n                      (\"sc\", StandardScaler())]), num),\n    (\"cat\", OneHotEncoder(handle_unknown=\"ignore\"), cat),\n])\n\npipe = Pipeline([(\"prep\", prep),\n                 (\"model\", HistGradientBoostingClassifier(random_state=0))])\n\n# 5-6. fit and tune using cross-validation on TRAIN only\nprint(\"cv auc\", cross_val_score(pipe, X_tr, y_tr, cv=5, scoring=\"roc_auc\").mean())\n\n# baseline for comparison\nbase = DummyClassifier(strategy=\"prior\").fit(X_tr, y_tr)\nprint(\"baseline auc\", roc_auc_score(y_te, base.predict_proba(X_te)[:, 1]))\n\n# 7. test set, once, at the end\npipe.fit(X_tr, y_tr)\nprint(\"test auc\", roc_auc_score(y_te, pipe.predict_proba(X_te)[:, 1]))"},{"t":"p","runs":[{"text":"Read it for the shape, not the details. Four things in there are load-bearing and worth naming now."}]},{"t":"p","runs":[{"text":"The split happens on line 12, before any exploration.","b":true},{"text":" Every transformation after it is fitted inside a "},{"text":"Pipeline","code":true},{"text":", which means the median used for imputation and the mean used for scaling are computed on training rows only. Do that by hand instead and you will one day compute the median on the whole dataset, leak test information into training, and get a score you cannot reproduce in production."}]},{"t":"p","runs":[{"text":"handle_unknown=\"ignore\"","b":true,"code":true},{"text":".","b":true},{"text":" A category appears in production that was not in training. Without this line, the pipeline raises and your service returns a 500. With it, that row gets all-zeros for the category block and a prediction still comes out. Neither is obviously right; the point is that you chose."}]},{"t":"p","runs":[{"text":"Cross-validation runs on training data only.","b":true},{"text":" The test set is not involved in any decision."}]},{"t":"p","runs":[{"text":"The test line appears once, at the end, after everything else is settled.","b":true}]},{"t":"h2","id":"where-the-time-actually-goes","runs":[{"text":"Where the time actually goes"}]},{"t":"p","runs":[{"text":"If you have read that machine learning is 80% data cleaning, the number is folklore but the direction is right. A more useful breakdown, from projects that shipped:"}]},{"t":"ul","items":[[{"text":"Deciding the question, the row and the target: more than people expect, and it is the highest-leverage time in the project."}],[{"text":"Getting the data and making it trustworthy: the largest single block, and the least visible."}],[{"text":"Feature engineering: the second largest, and where most accuracy is won on tabular problems."}],[{"text":"Model selection and tuning: usually far smaller than the excitement about it suggests. "},{"text":"HistGradientBoostingClassifier","code":true},{"text":" with default settings is a strong model; tuning typically buys single-digit percentage improvements over a sensible default."}],[{"text":"Evaluation, deployment and monitoring: the part that decides whether the work survives a year."}]]},{"t":"p","runs":[{"text":"The first three of those five are covered in modules 1, 3 and 5. That is deliberate."}]},{"t":"h2","id":"what-to-do-with-this-lesson","runs":[{"text":"What to do with this lesson"}]},{"t":"p","runs":[{"text":"Run the code on any dataset you can find: the Titanic set, a CSV of your own expenses, anything with a column you could try to predict. It will fail in some interesting way. Getting it to run once, end to end, on your own machine or a free notebook, is worth more than the next three lessons read passively, and it gives every later lesson something concrete to attach to."}]}],"check":{"question":[{"text":"In the pipeline above, why does putting the imputer and scaler inside "},{"text":"Pipeline","code":true},{"text":" rather than applying them to the full dataframe first actually change the resulting numbers?"}],"options":[{"text":[{"text":"It ensures the transformations run in the correct order, which manual application can get wrong."}],"correct":false,"tempting":[{"text":"Ordering is a real convenience of pipelines, but applying the same steps manually in the same order would still produce different numbers, so ordering is not the mechanism."}]},{"text":[{"text":"It makes the code shorter and reusable, so there is less chance of a typo between the training and prediction paths."}],"correct":false,"tempting":[{"text":"Describes an engineering benefit rather than a statistical one; the numbers would differ even in typo-free manual code."}]},{"text":[{"text":"It allows the same object to be serialised for deployment, so training and serving stay in sync."}],"correct":false,"tempting":[{"text":"True and important for module 9, but this benefit is about production drift rather than about the validation score being honest."}]},{"text":[{"text":"Inside a pipeline, the median and the mean are computed from training rows only, so no information from the held-out rows reaches the model."}],"correct":true,"tempting":null}],"order":[2,0,3,1],"why":null},"prev":{"slug":"when-not-to-use-machine-learning","title":"The problems where the right answer is a rule","minutes":8},"next":{"slug":"a-toolkit-that-costs-nothing","title":"A working setup that costs nothing","minutes":8}}}