Text as columns: counting words before you embed them

The simplest thing that works

Before embeddings and transformers, there is a technique from the 1960s that is still a strong baseline on many text tasks, trains in seconds on a laptop, and is fully interpretable.

Count the words.

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(min_df=5, max_df=0.8, ngram_range=(1, 2))
X = vec.fit_transform(docs)     # sparse: docs x vocabulary

One column per word in the vocabulary; each cell is how many times that word appears in that document. This is the bag of words, so called because it discards order entirely: “the dog bit the man” and “the man bit the dog” produce identical rows.

That sounds fatal and mostly is not, for classification. Whether a review is negative, whether an email is spam, whether a ticket is about billing: these are largely determined by which words are present, not their arrangement. Where order matters, ngram_range=(1, 2) adds two-word phrases, which recovers “not good” as distinct from “good” and is often the single most useful setting in this vectoriser.

TF-IDF: weighting by how unusual a word is

Raw counts overweight common words. “The” appears everywhere and separates nothing.

TF-IDF multiplies each count by the inverse document frequency: a term that grows as a word appears in fewer documents. Rare words get amplified, ubiquitous ones get crushed towards zero.

from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer(min_df=5, sublinear_tf=True, ngram_range=(1, 2))

sublinear_tf=True replaces the raw count with 1 + log(count), on the reasoning that a word appearing 40 times is not 40 times as indicative as one appearing once. It usually helps a little, costs nothing, and hardly anyone turns it on.

TF-IDF plus LogisticRegression is the classic text pipeline. On a two-class problem with a few thousand labelled documents it is frequently within a couple of points of a fine-tuned transformer, trains in under a second on a CPU, and gives you a coefficient per word so you can read exactly what it learned. Start here, always. Then measure whether the heavier option is worth its cost.

Character n-grams, which people forget

TfidfVectorizer(analyzer="char_wb", ngram_range=(3, 5))

This counts sequences of 3 to 5 characters instead of words, and it is the right tool for several situations word-level features handle badly: misspellings, product codes, URLs, names, languages without spaces between words, and code-mixed text like Hinglish where the same word appears in two scripts. It is also robust to the deliberate character substitutions used to evade spam filters, because “v1agra” and “viagra” share most of their character trigrams while sharing no words at all.

For Indian-language text on a Latin keyboard, where spelling is unstandardised and the same word appears five ways, character n-grams frequently beat word features outright.

What preprocessing actually helps

The traditional list is lowercasing, removing punctuation, removing stop words, and stemming. Some of it helps and some is folklore.

• Lowercasing: usually helps, halves the vocabulary. Hurts when case carries meaning, as in named entities or angry text.
• Stop word removal: much less useful than its reputation. TF-IDF already downweights common words, and max_df=0.8 removes anything appearing in over 80% of documents. Removing a fixed English stop list can also delete genuine signal: “not” is on most stop lists and is the difference between a positive and negative review.
• Stemming and lemmatisation: merges word forms so “running” and “ran” become one. Helps on small datasets where each form is rare; contributes little on large ones. Stemming is crude and fast, lemmatisation is slower and correct.
• Removing rare words with min_df=5: important. A word appearing in three documents cannot support a reliable coefficient and inflates the vocabulary enormously.

Test, do not assume. Each of these is one argument, and comparing four pipelines takes ten minutes.

Where this stops working

Be clear about the ceiling. Bag of words cannot tell “the drug caused the reaction” from “the reaction caused the drug”. It has no notion that “doctor” and “physician” are related: they are two independent columns as different from each other as either is from “aubergine”. And the vocabulary is fixed at training time, so a word that appears only in production is invisible.

Embeddings solve exactly these three problems, and module 7 explains how. But the order matters: build the count-based baseline first, get its number, and then you can say what the embedding bought you. Teams that skip the baseline routinely deploy an expensive model that beat nothing in particular.