When one class is 0.3% of the data

What imbalance actually breaks

Fraud at 0.3%. Rare disease at 0.1%. Equipment failure at 2%. The intuition is that the model will “ignore” the minority class, and that is roughly right, but it is worth being precise about the mechanism, because the precision determines the fix.

A model minimising log loss on a 0.3% positive rate can achieve a very low loss by predicting approximately 0.003 for everything. It is not broken; it is correct about the base rate and has found that the features do not move it much for most rows. The problem is that the decision rule you then apply, predict positive if p > 0.5, never fires.

That distinction matters, because it means the model is often fine and the threshold is the problem. Try that before anything else.

The order to try things

1. Change the threshold. Free, immediate, and it fixes the majority of cases. Get probabilities with predict_proba, then choose the cut point using precision and recall on validation data. Module 5 does this properly.

2. Change the metric. Accuracy is useless here: as the baseline lesson showed, predicting “never” scores 99.7%. Use precision, recall, F1, or better, average precision (the area under the precision-recall curve), which is far more informative than ROC AUC at extreme imbalance because it does not reward performance on the vast negative class.

3. Class weights. One argument, no new rows, and it changes the loss so minority errors cost more:

LogisticRegression(class_weight="balanced")
RandomForestClassifier(class_weight="balanced_subsample")

"balanced" sets each class’s weight inversely proportional to its frequency. For XGBoost the equivalent is scale_pos_weight, set to the ratio of negatives to positives.

Be aware of the side effect: weighting distorts the predicted probabilities. A weighted model’s 0.6 is not a 60% chance of the event. If you need calibrated probabilities (for expected-cost decisions, for pricing) either leave the weights out and move the threshold instead, or recalibrate afterwards.

4. Resampling. Only after the above.

Resampling, and why it disappoints

Random undersampling discards majority rows until the classes balance. Fast, and it throws away most of your data, which is a real loss when the majority class contains the variety the model needs to learn the boundary.

Random oversampling duplicates minority rows. No new information, and the model can memorise the duplicated rows, which shows up as an inflated validation score if the duplication happened before the split.

SMOTE creates synthetic minority rows by interpolating between a minority point and one of its k nearest minority neighbours. It is the most cited method in this area and its reputation exceeds its results.

The honest summary of the evidence: on tabular problems with a strong model such as gradient boosting, SMOTE often gives little or no improvement over adjusting class weights and the threshold, and several careful comparisons have found it makes calibration worse. It also assumes that the space between two minority points is itself minority territory, which need not hold: in fraud, the region between two different fraud patterns may be entirely legitimate behaviour. And it interpolates numerically, so it does not know what to do with categorical columns; SMOTE-NC exists for that and is more delicate than it looks.

If you use it: imbalanced-learn is the free library, and its Pipeline (not scikit-learn’s) is the one that resamples inside each fold. Resampling before cross-validation is the leakage from the previous lesson, and it is the most common way SMOTE produces a wonderful number that means nothing.

Collect more of the rare class, if you can

Unglamorous and usually the largest win available. Two hundred more genuine fraud examples are worth more than any resampling scheme, because they contain information that no interpolation can invent. If labelling is expensive, spend it on the minority class: label all the positives you can find and a random sample of negatives, then weight the negatives up to correct for the sampling. This is standard practice in survey statistics and underused here.

When it is not a classification problem at all

At extreme imbalance, under about 0.1%, or when the positive class is heterogeneous and the negatives are uniform, reframe.

Anomaly detection models what normal looks like and flags departures, using only the majority class. Useful when the positives are too few and too varied to learn from, and when tomorrow’s positives will not resemble today’s. Isolation Forest and One-Class SVM are the standard tools; module 7 covers them.

Ranking drops the classification framing entirely. A fraud team can investigate 200 cases a day, so the real question is not “which are fraud” but “which 200 are most worth looking at”. Optimise precision at 200, and the imbalance stops being a problem to solve and becomes a property of the situation.

That reframing is often the most valuable thing you can bring to an imbalanced problem, and it usually comes from asking what the person receiving the output is actually going to do with it.