Naive Bayes: wrong assumptions, useful results

Turning the problem around

Most classifiers model P(class | features) directly. Naive Bayes models P(features | class) and flips it with Bayes’ rule:

P(spam | words) is proportional to P(words | spam) * P(spam)

The first term is estimated by counting how often each word appears in spam. The second is just the fraction of messages that are spam. Nothing is fitted iteratively; the whole model is a table of counts.

The naive part

P(words | spam) for a 50-word message is a joint probability over 50 variables, and you would need an impossible amount of data to estimate it.

So assume the words are independent given the class, and multiply:

P(words | spam) = P(word1 | spam) * P(word2 | spam) * ...

This assumption is false. “Free” and “money” co-occur far more than independence predicts. “New” and “York” are not independent in any corpus. The model does not care and works anyway, which is the interesting part.

Why a false assumption still classifies well

The explanation is worth having, because it generalises.

When features are correlated, the model counts the same evidence repeatedly. Seeing “free”, “money” and “cash” in one message is treated as three independent pieces of evidence when it is closer to one, so the model’s confidence is inflated, often to 0.9999.

But classification only requires the ranking of the class scores to be right, not their magnitudes. Over-counting evidence tends to push scores toward the extremes without changing which class comes out on top. The probabilities become nonsense; the decision remains correct.

This is exactly the pattern from the calibration lesson: excellent discrimination, terrible calibration. If you need Naive Bayes probabilities to mean something, calibrate them afterwards. If you only need the label, do not bother.

The three variants

MultinomialNB: for counts. Word counts, TF-IDF. The standard choice for text.

BernoulliNB: for binary presence or absence. Better than Multinomial for short documents, where whether a word appears matters more than how often.

GaussianNB: for continuous features, assuming each is normally distributed within each class. Two assumptions stacked (independence and normality) and it is usually the weakest of the three on real data.

from sklearn.naive_bayes import MultinomialNB
MultinomialNB(alpha=1.0).fit(X_counts, y)

Smoothing, and why it is mandatory

A word that never appeared in spam during training gets P(word | spam) = 0. Multiply by zero and the whole message’s spam probability is zero, no matter what the other 49 words say. One unseen word vetoes all the evidence.

Laplace smoothing adds a small count to everything:

P(word | spam) = (count + alpha) / (total + alpha * vocabulary_size)

alpha=1.0 is the default and it is the difference between a working model and a broken one. It is worth understanding rather than accepting, because the same zero-probability problem appears in many count-based methods.

A related implementation detail: real implementations sum logs rather than multiplying probabilities, because multiplying fifty numbers below 0.1 underflows to zero in floating point. If you ever implement this, work in log space.

When it earns its place

Very fast. Training is one pass of counting; prediction is a sum of logs. On a million documents this trains in seconds where a boosted model takes an hour.

Works with tiny data. Because it estimates each feature’s statistics independently, it needs far fewer examples than a model estimating interactions. With 200 labelled documents it is often the best thing available.

Streams naturally. partial_fit lets you update counts as data arrives, with no retraining.

Genuinely competitive on text, especially short text and especially with few labels. It is a legitimate baseline, not a teaching toy: the classic Naive Bayes plus TF-IDF spam filter was the state of the art for years and is still deployed.

Where it fails

When feature interactions carry the signal, it cannot see them, by construction. When you need calibrated probabilities, it is unusable without post-processing. And on tabular data with correlated continuous features it is usually beaten comfortably by anything else in this module.

The reason it stays in the curriculum is that it is the cleanest example of a principle worth internalising: a model whose assumptions are plainly false can still be the right tool, if the way it is wrong does not affect the decision you are making. Asking how a model is wrong, rather than whether, is most of applied judgement.