{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"imbalanced-classes","title":"When one class is 0.3% of the data","url":"https://addaly.com/learn/machine-learning-foundations/imbalanced-classes","position":31,"of":92,"minutes":9,"answer":"Imbalance usually breaks the decision threshold rather than the model, so move the threshold and change the metric before resampling, and resample only inside the cross-validation fold, never before the split.","keyIdea":"Imbalance usually breaks the decision threshold rather than the model, so move the threshold and change the metric before resampling, and resample only inside the cross-validation fold, never before the split."},"body":{"format":"markdown","text":"## What imbalance actually breaks\n\nFraud at 0.3%. Rare disease at 0.1%. Equipment failure at 2%. The intuition is that the model will \"ignore\" the minority class, and that is roughly right, but it is worth being precise about the mechanism, because the precision determines the fix.\n\nA model minimising log loss on a 0.3% positive rate can achieve a very low loss by predicting approximately 0.003 for everything. It is not broken; it is correct about the base rate and has found that the features do not move it much for most rows. The problem is that the *decision rule* you then apply, predict positive if p > 0.5, never fires.\n\nThat distinction matters, because it means **the model is often fine and the threshold is the problem.** Try that before anything else.\n\n> Figure: The order to try things at 0.3 per cent positives. Steps: 1. Move the threshold (Free, immediate, fixes most cases); 2. Change the metric (Average precision, not accuracy); 3. Class weights (One argument; distorts probabilities); 4. Resample (Only inside the fold; SMOTE often adds nothing); 5. Collect more positives (The largest win, when available). The model is usually fine; the 0.5 decision rule is what never fires. Try the cheap fixes in this order and stop when the number moves. Resampling comes last because it usually adds little over weights and a moved threshold, and it leaks if it happens before the split.\n\n## The order to try things\n\n**1. Change the threshold.** Free, immediate, and it fixes the majority of cases. Get probabilities with `predict_proba`, then choose the cut point using precision and recall on validation data. Module 5 does this properly.\n\n**2. Change the metric.** Accuracy is useless here: as the baseline lesson showed, predicting \"never\" scores 99.7%. Use precision, recall, F1, or better, average precision (the area under the precision-recall curve), which is far more informative than ROC AUC at extreme imbalance because it does not reward performance on the vast negative class.\n\n**3. Class weights.** One argument, no new rows, and it changes the loss so minority errors cost more:\n\n```python\nLogisticRegression(class_weight=\"balanced\")\nRandomForestClassifier(class_weight=\"balanced_subsample\")\n```\n\n`\"balanced\"` sets each class's weight inversely proportional to its frequency. For XGBoost the equivalent is `scale_pos_weight`, set to the ratio of negatives to positives.\n\nBe aware of the side effect: weighting distorts the predicted probabilities. A weighted model's 0.6 is not a 60% chance of the event. If you need calibrated probabilities (for expected-cost decisions, for pricing) either leave the weights out and move the threshold instead, or recalibrate afterwards.\n\n**4. Resampling.** Only after the above.\n\n## Resampling, and why it disappoints\n\n**Random undersampling** discards majority rows until the classes balance. Fast, and it throws away most of your data, which is a real loss when the majority class contains the variety the model needs to learn the boundary.\n\n**Random oversampling** duplicates minority rows. No new information, and the model can memorise the duplicated rows, which shows up as an inflated validation score if the duplication happened before the split.\n\n**SMOTE** creates synthetic minority rows by interpolating between a minority point and one of its k nearest minority neighbours. It is the most cited method in this area and its reputation exceeds its results.\n\nThe honest summary of the evidence: on tabular problems with a strong model such as gradient boosting, SMOTE often gives little or no improvement over adjusting class weights and the threshold, and several careful comparisons have found it makes calibration worse. It also assumes that the space between two minority points is itself minority territory, which need not hold: in fraud, the region between two different fraud patterns may be entirely legitimate behaviour. And it interpolates numerically, so it does not know what to do with categorical columns; SMOTE-NC exists for that and is more delicate than it looks.\n\nIf you use it: `imbalanced-learn` is the free library, and its `Pipeline` (not scikit-learn's) is the one that resamples inside each fold. Resampling before cross-validation is the leakage from the previous lesson, and it is the most common way SMOTE produces a wonderful number that means nothing.\n\n## Collect more of the rare class, if you can\n\nUnglamorous and usually the largest win available. Two hundred more genuine fraud examples are worth more than any resampling scheme, because they contain information that no interpolation can invent. If labelling is expensive, spend it on the minority class: label all the positives you can find and a random sample of negatives, then weight the negatives up to correct for the sampling. This is standard practice in survey statistics and underused here.\n\n## When it is not a classification problem at all\n\nAt extreme imbalance, under about 0.1%, or when the positive class is heterogeneous and the negatives are uniform, reframe.\n\n**Anomaly detection** models what normal looks like and flags departures, using only the majority class. Useful when the positives are too few and too varied to learn from, and when tomorrow's positives will not resemble today's. Isolation Forest and One-Class SVM are the standard tools; module 7 covers them.\n\n**Ranking** drops the classification framing entirely. A fraud team can investigate 200 cases a day, so the real question is not \"which are fraud\" but \"which 200 are most worth looking at\". Optimise precision at 200, and the imbalance stops being a problem to solve and becomes a property of the situation.\n\nThat reframing is often the most valuable thing you can bring to an imbalanced problem, and it usually comes from asking what the person receiving the output is actually going to do with it."},"check":{"question":"A team applies SMOTE to a 1% fraud dataset and reports cross-validated F1 rising from 0.31 to 0.68. Which check would most likely explain the gain?","options":[{"text":"Whether the model's probabilities are calibrated after resampling, since SMOTE distorts the predicted base rate.","correct":false,"tempting":"Calibration does degrade after resampling and is worth checking, but miscalibration would not by itself inflate a cross-validated F1 score."},{"text":"Whether F1 was computed with macro rather than binary averaging, which changes the score substantially at 1% prevalence.","correct":false,"tempting":"Averaging choice matters a great deal for reported numbers, but it would affect both the before and after figures equally."},{"text":"Whether SMOTE was applied before the cross-validation split, so synthetic points interpolated from a fold's training rows also appear in its validation portion.","correct":true},{"text":"Whether the k-neighbours parameter of SMOTE was tuned, since the default of 5 is often too small for sparse minority classes.","correct":false,"tempting":"Tuning k changes how synthetic points are placed, but no setting of k produces a doubling of F1 through legitimate means."}]},"prev":{"slug":"where-the-rows-came-from","title":"Where the rows came from, and who is missing","url":"https://addaly.com/learn/machine-learning-foundations/where-the-rows-came-from"},"next":{"slug":"not-looking","title":"Train, validation, test, and the discipline of not looking","url":"https://addaly.com/learn/machine-learning-foundations/not-looking"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When one class is 0.3% of the data', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/imbalanced-classes (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When one class is 0.3% of the data","courseTitle":"Machine Learning, Foundations","module":{"index":3,"slug":"the-data-is-the-job","title":"The data is the job"},"answer":null,"keyIdea":"Imbalance usually breaks the decision threshold rather than the model, so move the threshold and change the metric before resampling, and resample only inside the cross-validation fold, never before the split.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Move the threshold","detail":"Free, immediate, fixes most cases"},{"label":"Change the metric","detail":"Average precision, not accuracy"},{"label":"Class weights","detail":"One argument; distorts probabilities"},{"label":"Resample","detail":"Only inside the fold; SMOTE often adds nothing"},{"label":"Collect more positives","detail":"The largest win, when available"}],"title":"The order to try things at 0.3 per cent positives","caption":"The model is usually fine; the 0.5 decision rule is what never fires. Try the cheap fixes in this order and stop when the number moves. Resampling comes last because it usually adds little over weights and a moved threshold, and it leaks if it happens before the split."},"alt":"Figure: The order to try things at 0.3 per cent positives. Steps: 1. Move the threshold (Free, immediate, fixes most cases); 2. Change the metric (Average precision, not accuracy); 3. Class weights (One argument; distorts probabilities); 4. Resample (Only inside the fold; SMOTE often adds nothing); 5. Collect more positives (The largest win, when available). The model is usually fine; the 0.5 decision rule is what never fires. Try the cheap fixes in this order and stop when the number moves. Resampling comes last because it usually adds little over weights and a moved threshold, and it leaks if it happens before the split."}},"blocks":[{"t":"h2","id":"what-imbalance-actually-breaks","runs":[{"text":"What imbalance actually breaks"}]},{"t":"p","runs":[{"text":"Fraud at 0.3%. Rare disease at 0.1%. Equipment failure at 2%. The intuition is that the model will “ignore” the minority class, and that is roughly right, but it is worth being precise about the mechanism, because the precision determines the fix."}]},{"t":"p","runs":[{"text":"A model minimising log loss on a 0.3% positive rate can achieve a very low loss by predicting approximately 0.003 for everything. It is not broken; it is correct about the base rate and has found that the features do not move it much for most rows. The problem is that the "},{"text":"decision rule","i":true},{"text":" you then apply, predict positive if p > 0.5, never fires."}]},{"t":"p","runs":[{"text":"That distinction matters, because it means "},{"text":"the model is often fine and the threshold is the problem.","b":true},{"text":" Try that before anything else."}]},{"t":"h2","id":"the-order-to-try-things","runs":[{"text":"The order to try things"}]},{"t":"p","runs":[{"text":"1. Change the threshold.","b":true},{"text":" Free, immediate, and it fixes the majority of cases. Get probabilities with "},{"text":"predict_proba","code":true},{"text":", then choose the cut point using precision and recall on validation data. Module 5 does this properly."}]},{"t":"p","runs":[{"text":"2. Change the metric.","b":true},{"text":" Accuracy is useless here: as the baseline lesson showed, predicting “never” scores 99.7%. Use precision, recall, F1, or better, average precision (the area under the precision-recall curve), which is far more informative than ROC AUC at extreme imbalance because it does not reward performance on the vast negative class."}]},{"t":"p","runs":[{"text":"3. Class weights.","b":true},{"text":" One argument, no new rows, and it changes the loss so minority errors cost more:"}]},{"t":"code","lang":"python","label":"Python","text":"LogisticRegression(class_weight=\"balanced\")\nRandomForestClassifier(class_weight=\"balanced_subsample\")"},{"t":"p","runs":[{"text":"\"balanced\"","code":true},{"text":" sets each class’s weight inversely proportional to its frequency. For XGBoost the equivalent is "},{"text":"scale_pos_weight","code":true},{"text":", set to the ratio of negatives to positives."}]},{"t":"p","runs":[{"text":"Be aware of the side effect: weighting distorts the predicted probabilities. A weighted model’s 0.6 is not a 60% chance of the event. If you need calibrated probabilities (for expected-cost decisions, for pricing) either leave the weights out and move the threshold instead, or recalibrate afterwards."}]},{"t":"p","runs":[{"text":"4. Resampling.","b":true},{"text":" Only after the above."}]},{"t":"h2","id":"resampling-and-why-it-disappoints","runs":[{"text":"Resampling, and why it disappoints"}]},{"t":"p","runs":[{"text":"Random undersampling","b":true},{"text":" discards majority rows until the classes balance. Fast, and it throws away most of your data, which is a real loss when the majority class contains the variety the model needs to learn the boundary."}]},{"t":"p","runs":[{"text":"Random oversampling","b":true},{"text":" duplicates minority rows. No new information, and the model can memorise the duplicated rows, which shows up as an inflated validation score if the duplication happened before the split."}]},{"t":"p","runs":[{"text":"SMOTE","b":true},{"text":" creates synthetic minority rows by interpolating between a minority point and one of its k nearest minority neighbours. It is the most cited method in this area and its reputation exceeds its results."}]},{"t":"p","runs":[{"text":"The honest summary of the evidence: on tabular problems with a strong model such as gradient boosting, SMOTE often gives little or no improvement over adjusting class weights and the threshold, and several careful comparisons have found it makes calibration worse. It also assumes that the space between two minority points is itself minority territory, which need not hold: in fraud, the region between two different fraud patterns may be entirely legitimate behaviour. And it interpolates numerically, so it does not know what to do with categorical columns; SMOTE-NC exists for that and is more delicate than it looks."}]},{"t":"p","runs":[{"text":"If you use it: "},{"text":"imbalanced-learn","code":true},{"text":" is the free library, and its "},{"text":"Pipeline","code":true},{"text":" (not scikit-learn’s) is the one that resamples inside each fold. Resampling before cross-validation is the leakage from the previous lesson, and it is the most common way SMOTE produces a wonderful number that means nothing."}]},{"t":"h2","id":"collect-more-of-the-rare-class-if-you-can","runs":[{"text":"Collect more of the rare class, if you can"}]},{"t":"p","runs":[{"text":"Unglamorous and usually the largest win available. Two hundred more genuine fraud examples are worth more than any resampling scheme, because they contain information that no interpolation can invent. If labelling is expensive, spend it on the minority class: label all the positives you can find and a random sample of negatives, then weight the negatives up to correct for the sampling. This is standard practice in survey statistics and underused here."}]},{"t":"h2","id":"when-it-is-not-a-classification-problem-at-all","runs":[{"text":"When it is not a classification problem at all"}]},{"t":"p","runs":[{"text":"At extreme imbalance, under about 0.1%, or when the positive class is heterogeneous and the negatives are uniform, reframe."}]},{"t":"p","runs":[{"text":"Anomaly detection","b":true},{"text":" models what normal looks like and flags departures, using only the majority class. Useful when the positives are too few and too varied to learn from, and when tomorrow’s positives will not resemble today’s. Isolation Forest and One-Class SVM are the standard tools; module 7 covers them."}]},{"t":"p","runs":[{"text":"Ranking","b":true},{"text":" drops the classification framing entirely. A fraud team can investigate 200 cases a day, so the real question is not “which are fraud” but “which 200 are most worth looking at”. Optimise precision at 200, and the imbalance stops being a problem to solve and becomes a property of the situation."}]},{"t":"p","runs":[{"text":"That reframing is often the most valuable thing you can bring to an imbalanced problem, and it usually comes from asking what the person receiving the output is actually going to do with it."}]}],"check":{"question":[{"text":"A team applies SMOTE to a 1% fraud dataset and reports cross-validated F1 rising from 0.31 to 0.68. Which check would most likely explain the gain?"}],"options":[{"text":[{"text":"Whether the model’s probabilities are calibrated after resampling, since SMOTE distorts the predicted base rate."}],"correct":false,"tempting":[{"text":"Calibration does degrade after resampling and is worth checking, but miscalibration would not by itself inflate a cross-validated F1 score."}]},{"text":[{"text":"Whether F1 was computed with macro rather than binary averaging, which changes the score substantially at 1% prevalence."}],"correct":false,"tempting":[{"text":"Averaging choice matters a great deal for reported numbers, but it would affect both the before and after figures equally."}]},{"text":[{"text":"Whether SMOTE was applied before the cross-validation split, so synthetic points interpolated from a fold’s training rows also appear in its validation portion."}],"correct":true,"tempting":null},{"text":[{"text":"Whether the k-neighbours parameter of SMOTE was tuned, since the default of 5 is often too small for sparse minority classes."}],"correct":false,"tempting":[{"text":"Tuning k changes how synthetic points are placed, but no setting of k produces a doubling of F1 through legitimate means."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"where-the-rows-came-from","title":"Where the rows came from, and who is missing","minutes":9},"next":{"slug":"not-looking","title":"Train, validation, test, and the discipline of not looking","minutes":9}}}