{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"text-as-numbers","title":"Text as columns: counting words before you embed them","url":"https://addaly.com/learn/machine-learning-foundations/text-as-numbers","position":27,"of":92,"minutes":9,"answer":"TF-IDF over word and bigram counts with a linear model is a strong, fast, fully interpretable text baseline; its ceiling is that it cannot represent word order or word similarity.","keyIdea":"TF-IDF over word and bigram counts with a linear model is a strong, fast, fully interpretable text baseline; its ceiling is that it cannot represent word order or word similarity, which is precisely what embeddings add."},"body":{"format":"markdown","text":"## The simplest thing that works\n\nBefore embeddings and transformers, there is a technique from the 1960s that is still a strong baseline on many text tasks, trains in seconds on a laptop, and is fully interpretable.\n\nCount the words.\n\n```python\nfrom sklearn.feature_extraction.text import CountVectorizer\nvec = CountVectorizer(min_df=5, max_df=0.8, ngram_range=(1, 2))\nX = vec.fit_transform(docs)     # sparse: docs x vocabulary\n```\n\nOne column per word in the vocabulary; each cell is how many times that word appears in that document. This is the **bag of words**, so called because it discards order entirely: \"the dog bit the man\" and \"the man bit the dog\" produce identical rows.\n\nThat sounds fatal and mostly is not, for classification. Whether a review is negative, whether an email is spam, whether a ticket is about billing: these are largely determined by which words are present, not their arrangement. Where order matters, `ngram_range=(1, 2)` adds two-word phrases, which recovers \"not good\" as distinct from \"good\" and is often the single most useful setting in this vectoriser.\n\n## TF-IDF: weighting by how unusual a word is\n\nRaw counts overweight common words. \"The\" appears everywhere and separates nothing.\n\nTF-IDF multiplies each count by the inverse document frequency: a term that grows as a word appears in fewer documents. Rare words get amplified, ubiquitous ones get crushed towards zero.\n\n```python\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nvec = TfidfVectorizer(min_df=5, sublinear_tf=True, ngram_range=(1, 2))\n```\n\n`sublinear_tf=True` replaces the raw count with `1 + log(count)`, on the reasoning that a word appearing 40 times is not 40 times as indicative as one appearing once. It usually helps a little, costs nothing, and hardly anyone turns it on.\n\nTF-IDF plus `LogisticRegression` is the classic text pipeline. On a two-class problem with a few thousand labelled documents it is frequently within a couple of points of a fine-tuned transformer, trains in under a second on a CPU, and gives you a coefficient per word so you can read exactly what it learned. Start here, always. Then measure whether the heavier option is worth its cost.\n\n## Character n-grams, which people forget\n\n```python\nTfidfVectorizer(analyzer=\"char_wb\", ngram_range=(3, 5))\n```\n\nThis counts sequences of 3 to 5 characters instead of words, and it is the right tool for several situations word-level features handle badly: misspellings, product codes, URLs, names, languages without spaces between words, and code-mixed text like Hinglish where the same word appears in two scripts. It is also robust to the deliberate character substitutions used to evade spam filters, because \"v1agra\" and \"viagra\" share most of their character trigrams while sharing no words at all.\n\nFor Indian-language text on a Latin keyboard, where spelling is unstandardised and the same word appears five ways, character n-grams frequently beat word features outright.\n\n## What preprocessing actually helps\n\nThe traditional list is lowercasing, removing punctuation, removing stop words, and stemming. Some of it helps and some is folklore.\n\n- **Lowercasing**: usually helps, halves the vocabulary. Hurts when case carries meaning, as in named entities or angry text.\n- **Stop word removal**: much less useful than its reputation. TF-IDF already downweights common words, and `max_df=0.8` removes anything appearing in over 80% of documents. Removing a fixed English stop list can also delete genuine signal: \"not\" is on most stop lists and is the difference between a positive and negative review.\n- **Stemming and lemmatisation**: merges word forms so \"running\" and \"ran\" become one. Helps on small datasets where each form is rare; contributes little on large ones. Stemming is crude and fast, lemmatisation is slower and correct.\n- **Removing rare words** with `min_df=5`: important. A word appearing in three documents cannot support a reliable coefficient and inflates the vocabulary enormously.\n\nTest, do not assume. Each of these is one argument, and comparing four pipelines takes ten minutes.\n\n## Where this stops working\n\nBe clear about the ceiling. Bag of words cannot tell \"the drug caused the reaction\" from \"the reaction caused the drug\". It has no notion that \"doctor\" and \"physician\" are related: they are two independent columns as different from each other as either is from \"aubergine\". And the vocabulary is fixed at training time, so a word that appears only in production is invisible.\n\nEmbeddings solve exactly these three problems, and module 7 explains how. But the order matters: build the count-based baseline first, get its number, and then you can say what the embedding bought you. Teams that skip the baseline routinely deploy an expensive model that beat nothing in particular."},"check":{"question":"A sentiment classifier on product reviews uses TF-IDF with a standard English stop-word list and performs poorly on reviews containing complaints. What is the most likely cause?","options":[{"text":"TF-IDF underweights rare complaint vocabulary, so negative reviews have weak feature values.","correct":false,"tempting":"Inverts the mechanism: inverse document frequency amplifies rare terms rather than suppressing them."},{"text":"Negation words like 'not' and 'never' are on standard stop-word lists, so 'not good' and 'good' become indistinguishable.","correct":true},{"text":"Complaints are longer, and TF-IDF does not normalise for document length.","correct":false,"tempting":"TfidfVectorizer applies L2 normalisation per document by default, so length is largely accounted for already."},{"text":"The vectoriser lowercases text, discarding the emphatic capitalisation common in complaints.","correct":false,"tempting":"Case can carry emphasis, but losing it is a mild effect compared with deleting the words that reverse a sentence's meaning."}]},"prev":{"slug":"dates-and-cycles","title":"Dates, durations, and the trouble with midnight","url":"https://addaly.com/learn/machine-learning-foundations/dates-and-cycles"},"next":{"slug":"leakage-in-detail","title":"Leakage: a catalogue of the ways the answer gets in","url":"https://addaly.com/learn/machine-learning-foundations/leakage-in-detail"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Text as columns: counting words before you embed them', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/text-as-numbers (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Text as columns: counting words before you embed them","courseTitle":"Machine Learning, Foundations","module":{"index":3,"slug":"the-data-is-the-job","title":"The data is the job"},"answer":null,"keyIdea":"TF-IDF over word and bigram counts with a linear model is a strong, fast, fully interpretable text baseline; its ceiling is that it cannot represent word order or word similarity, which is precisely what embeddings add.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-simplest-thing-that-works","runs":[{"text":"The simplest thing that works"}]},{"t":"p","runs":[{"text":"Before embeddings and transformers, there is a technique from the 1960s that is still a strong baseline on many text tasks, trains in seconds on a laptop, and is fully interpretable."}]},{"t":"p","runs":[{"text":"Count the words."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.feature_extraction.text import CountVectorizer\nvec = CountVectorizer(min_df=5, max_df=0.8, ngram_range=(1, 2))\nX = vec.fit_transform(docs)     # sparse: docs x vocabulary"},{"t":"p","runs":[{"text":"One column per word in the vocabulary; each cell is how many times that word appears in that document. This is the "},{"text":"bag of words","b":true},{"text":", so called because it discards order entirely: “the dog bit the man” and “the man bit the dog” produce identical rows."}]},{"t":"p","runs":[{"text":"That sounds fatal and mostly is not, for classification. Whether a review is negative, whether an email is spam, whether a ticket is about billing: these are largely determined by which words are present, not their arrangement. Where order matters, "},{"text":"ngram_range=(1, 2)","code":true},{"text":" adds two-word phrases, which recovers “not good” as distinct from “good” and is often the single most useful setting in this vectoriser."}]},{"t":"h2","id":"tf-idf-weighting-by-how-unusual-a-word-is","runs":[{"text":"TF-IDF: weighting by how unusual a word is"}]},{"t":"p","runs":[{"text":"Raw counts overweight common words. “The” appears everywhere and separates nothing."}]},{"t":"p","runs":[{"text":"TF-IDF multiplies each count by the inverse document frequency: a term that grows as a word appears in fewer documents. Rare words get amplified, ubiquitous ones get crushed towards zero."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.feature_extraction.text import TfidfVectorizer\nvec = TfidfVectorizer(min_df=5, sublinear_tf=True, ngram_range=(1, 2))"},{"t":"p","runs":[{"text":"sublinear_tf=True","code":true},{"text":" replaces the raw count with "},{"text":"1 + log(count)","code":true},{"text":", on the reasoning that a word appearing 40 times is not 40 times as indicative as one appearing once. It usually helps a little, costs nothing, and hardly anyone turns it on."}]},{"t":"p","runs":[{"text":"TF-IDF plus "},{"text":"LogisticRegression","code":true},{"text":" is the classic text pipeline. On a two-class problem with a few thousand labelled documents it is frequently within a couple of points of a fine-tuned transformer, trains in under a second on a CPU, and gives you a coefficient per word so you can read exactly what it learned. Start here, always. Then measure whether the heavier option is worth its cost."}]},{"t":"h2","id":"character-n-grams-which-people-forget","runs":[{"text":"Character n-grams, which people forget"}]},{"t":"code","lang":"python","label":"Python","text":"TfidfVectorizer(analyzer=\"char_wb\", ngram_range=(3, 5))"},{"t":"p","runs":[{"text":"This counts sequences of 3 to 5 characters instead of words, and it is the right tool for several situations word-level features handle badly: misspellings, product codes, URLs, names, languages without spaces between words, and code-mixed text like Hinglish where the same word appears in two scripts. It is also robust to the deliberate character substitutions used to evade spam filters, because “v1agra” and “viagra” share most of their character trigrams while sharing no words at all."}]},{"t":"p","runs":[{"text":"For Indian-language text on a Latin keyboard, where spelling is unstandardised and the same word appears five ways, character n-grams frequently beat word features outright."}]},{"t":"h2","id":"what-preprocessing-actually-helps","runs":[{"text":"What preprocessing actually helps"}]},{"t":"p","runs":[{"text":"The traditional list is lowercasing, removing punctuation, removing stop words, and stemming. Some of it helps and some is folklore."}]},{"t":"ul","items":[[{"text":"Lowercasing","b":true},{"text":": usually helps, halves the vocabulary. Hurts when case carries meaning, as in named entities or angry text."}],[{"text":"Stop word removal","b":true},{"text":": much less useful than its reputation. TF-IDF already downweights common words, and "},{"text":"max_df=0.8","code":true},{"text":" removes anything appearing in over 80% of documents. Removing a fixed English stop list can also delete genuine signal: “not” is on most stop lists and is the difference between a positive and negative review."}],[{"text":"Stemming and lemmatisation","b":true},{"text":": merges word forms so “running” and “ran” become one. Helps on small datasets where each form is rare; contributes little on large ones. Stemming is crude and fast, lemmatisation is slower and correct."}],[{"text":"Removing rare words","b":true},{"text":" with "},{"text":"min_df=5","code":true},{"text":": important. A word appearing in three documents cannot support a reliable coefficient and inflates the vocabulary enormously."}]]},{"t":"p","runs":[{"text":"Test, do not assume. Each of these is one argument, and comparing four pipelines takes ten minutes."}]},{"t":"h2","id":"where-this-stops-working","runs":[{"text":"Where this stops working"}]},{"t":"p","runs":[{"text":"Be clear about the ceiling. Bag of words cannot tell “the drug caused the reaction” from “the reaction caused the drug”. It has no notion that “doctor” and “physician” are related: they are two independent columns as different from each other as either is from “aubergine”. And the vocabulary is fixed at training time, so a word that appears only in production is invisible."}]},{"t":"p","runs":[{"text":"Embeddings solve exactly these three problems, and module 7 explains how. But the order matters: build the count-based baseline first, get its number, and then you can say what the embedding bought you. Teams that skip the baseline routinely deploy an expensive model that beat nothing in particular."}]}],"check":{"question":[{"text":"A sentiment classifier on product reviews uses TF-IDF with a standard English stop-word list and performs poorly on reviews containing complaints. What is the most likely cause?"}],"options":[{"text":[{"text":"TF-IDF underweights rare complaint vocabulary, so negative reviews have weak feature values."}],"correct":false,"tempting":[{"text":"Inverts the mechanism: inverse document frequency amplifies rare terms rather than suppressing them."}]},{"text":[{"text":"Negation words like ‘not’ and ‘never’ are on standard stop-word lists, so ‘not good’ and ‘good’ become indistinguishable."}],"correct":true,"tempting":null},{"text":[{"text":"Complaints are longer, and TF-IDF does not normalise for document length."}],"correct":false,"tempting":[{"text":"TfidfVectorizer applies L2 normalisation per document by default, so length is largely accounted for already."}]},{"text":[{"text":"The vectoriser lowercases text, discarding the emphatic capitalisation common in complaints."}],"correct":false,"tempting":[{"text":"Case can carry emphasis, but losing it is a mild effect compared with deleting the words that reverse a sentence’s meaning."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"dates-and-cycles","title":"Dates, durations, and the trouble with midnight","minutes":8},"next":{"slug":"leakage-in-detail","title":"Leakage: a catalogue of the ways the answer gets in","minutes":10}}}