{"course":{"slug":"machine-learning-foundations","title":"Machine Learning, Foundations","url":"https://addaly.com/learn/machine-learning-foundations","level":"intermediate","track":"start"},"lesson":{"slug":"more-than-two-classes","title":"More than two classes, and what changes","url":"https://addaly.com/learn/machine-learning-foundations/more-than-two-classes","position":19,"of":92,"minutes":8,"answer":"Softmax makes classes compete for a fixed probability budget, which is correct when exactly one label is true and wrong for multi-label problems, where independent sigmoids with their own thresholds are needed.","keyIdea":"Softmax makes classes compete for a fixed probability budget, which is correct when exactly one label is true and wrong for multi-label problems, where independent sigmoids with their own thresholds are needed."},"body":{"format":"markdown","text":"## Two strategies\n\nLogistic regression is built for two classes. Real problems have four, or fifty, or three thousand. There are two ways to get there and they behave differently.\n\n**One-vs-rest.** Train one binary classifier per class: \"is it A or not\", \"is it B or not\", and so on. To predict, run all of them and take the highest score. With 50 classes you train 50 models.\n\nSimple and it works with any binary classifier. Two weaknesses: each classifier sees a badly imbalanced problem (one class against 49), and the scores come from independently fitted models, so they are on different scales and do not sum to 1. Comparing them is a bit of a fudge.\n\n**Multinomial (softmax).** One model, with one weight vector per class, all fitted together against a single loss. For each row, compute a score per class and convert the whole vector to probabilities at once:\n\n```text\np_k = exp(s_k) / sum over all j of exp(s_j)\n```\n\nThat is softmax. It exponentiates to make everything positive, then divides by the total so the probabilities sum to exactly 1. The classes compete: raising one score lowers every other probability, which is the correct behaviour when exactly one class is true.\n\nModern scikit-learn uses multinomial by default for `LogisticRegression`, and it is the right default. Neural network classifiers always use softmax on the output layer with cross-entropy loss, which is the same construction.\n\n## Two numerical details that matter in practice\n\n**Softmax overflows if you write it naively.** `exp(1000)` is infinity in floating point. Every real implementation subtracts the maximum score before exponentiating, which does not change the result mathematically (the constant cancels between numerator and denominator) but keeps every exponent at or below zero.\n\n```python\ndef softmax(s):\n    s = s - s.max()\n    e = np.exp(s)\n    return e / e.sum()\n```\n\nIf you ever write this yourself, that one line is the difference between working code and `NaN`.\n\n**Softmax and sigmoid are the same thing at two classes.** Softmax with two classes reduces exactly to the sigmoid. Nothing new was introduced; the binary case was the special case all along.\n\n## Multi-class is not multi-label\n\nA distinction worth being firm about, because mixing them up produces a model that cannot express the truth.\n\n**Multi-class**: exactly one label is correct. A photo is a cat, or a dog, or a horse. Softmax, and the probabilities sum to 1.\n\n**Multi-label**: any number of labels can be correct at once. An article is about politics *and* economics *and* India. Here softmax is wrong: it forces the labels to compete for a fixed budget of probability, so a three-topic article has to divide its confidence three ways. Use independent sigmoids, one per label, each with its own binary cross-entropy and its own threshold. In scikit-learn that is `MultiOutputClassifier` or `OneVsRestClassifier`.\n\nAsk of any labelling task: can two labels be true at once? The answer determines the output layer, and getting it wrong caps your accuracy in a way no amount of tuning recovers.\n\n## Metrics need an averaging decision\n\nPrecision and recall are defined per class, so reporting one number for a multi-class problem requires choosing how to combine them, and the choices disagree sharply.\n\n- **Macro**: compute the metric per class, then take the plain mean. Every class counts equally, so a rare class with 12 examples influences the score as much as a common one with 40,000.\n- **Weighted**: the mean weighted by class frequency. Dominated by the common classes.\n- **Micro**: pool all the decisions and compute once. For single-label multi-class this equals accuracy.\n\nThese can differ by 30 points on an imbalanced problem, and quoting one without saying which is a common way to mislead without lying. If the rare classes are the point (rare diseases, rare failure modes) use macro and say so.\n\n```python\nfrom sklearn.metrics import classification_report\nprint(classification_report(y_te, pred, digits=3))\n```\n\nThat report gives per-class precision, recall and support, plus all three averages. Read the per-class rows before the summary line. A model with 0.91 weighted F1 that scores 0.0 on your two rarest classes is a specific, common and completely invisible failure if you only look at the average.\n\n## When there are thousands of classes\n\nAt very high class counts (product catalogues, word vocabularies) softmax gets expensive, because every prediction computes a score for every class. The workarounds are hierarchical softmax and sampled softmax, and in retrieval settings the problem is usually reframed as nearest-neighbour search over embeddings, which module 7 covers. Worth knowing the shape of the problem exists; you will not meet it on tabular data."},"check":{"question":"A news classifier tags articles with any of eight topics, and articles frequently belong to two or three. The team trains a softmax model over the eight topics and finds it never confidently assigns more than one topic. What is the cause?","options":[{"text":"The training data is imbalanced across topics, so the model has collapsed onto the most common one.","correct":false,"tempting":"Imbalance would skew which topic is chosen, but not create a structural inability to assign several topics at once."},{"text":"Eight classes is too many for one softmax layer, and the model needs a hierarchical output.","correct":false,"tempting":"Hierarchical softmax addresses computational cost at thousands of classes, not the number of labels a model can assign per row."},{"text":"Softmax forces the eight probabilities to sum to 1, so confidence in one topic mechanically reduces confidence in the others; multi-label needs independent sigmoids.","correct":true},{"text":"The decision threshold is fixed at the argmax, and lowering it to 0.3 would allow several topics through.","correct":false,"tempting":"Thresholding softmax outputs can surface secondary topics, but the normalisation still caps total confidence at 1, so a three-topic article cannot be confident about all three."}]},"prev":{"slug":"making-a-linear-model-curve","title":"Making a straight-line model bend","url":"https://addaly.com/learn/machine-learning-foundations/making-a-linear-model-curve"},"next":{"slug":"features-beat-models","title":"Features beat models","url":"https://addaly.com/learn/machine-learning-foundations/features-beat-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'More than two classes, and what changes', in Machine Learning, Foundations. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/machine-learning-foundations/more-than-two-classes (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"More than two classes, and what changes","courseTitle":"Machine Learning, Foundations","module":{"index":2,"slug":"linear-models-and-fitting","title":"Linear models and the machinery of fitting"},"answer":null,"keyIdea":"Softmax makes classes compete for a fixed probability budget, which is correct when exactly one label is true and wrong for multi-label problems, where independent sigmoids with their own thresholds are needed.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-strategies","runs":[{"text":"Two strategies"}]},{"t":"p","runs":[{"text":"Logistic regression is built for two classes. Real problems have four, or fifty, or three thousand. There are two ways to get there and they behave differently."}]},{"t":"p","runs":[{"text":"One-vs-rest.","b":true},{"text":" Train one binary classifier per class: “is it A or not”, “is it B or not”, and so on. To predict, run all of them and take the highest score. With 50 classes you train 50 models."}]},{"t":"p","runs":[{"text":"Simple and it works with any binary classifier. Two weaknesses: each classifier sees a badly imbalanced problem (one class against 49), and the scores come from independently fitted models, so they are on different scales and do not sum to 1. Comparing them is a bit of a fudge."}]},{"t":"p","runs":[{"text":"Multinomial (softmax).","b":true},{"text":" One model, with one weight vector per class, all fitted together against a single loss. For each row, compute a score per class and convert the whole vector to probabilities at once:"}]},{"t":"code","lang":"text","label":"Text","text":"p_k = exp(s_k) / sum over all j of exp(s_j)"},{"t":"p","runs":[{"text":"That is softmax. It exponentiates to make everything positive, then divides by the total so the probabilities sum to exactly 1. The classes compete: raising one score lowers every other probability, which is the correct behaviour when exactly one class is true."}]},{"t":"p","runs":[{"text":"Modern scikit-learn uses multinomial by default for "},{"text":"LogisticRegression","code":true},{"text":", and it is the right default. Neural network classifiers always use softmax on the output layer with cross-entropy loss, which is the same construction."}]},{"t":"h2","id":"two-numerical-details-that-matter-in-practice","runs":[{"text":"Two numerical details that matter in practice"}]},{"t":"p","runs":[{"text":"Softmax overflows if you write it naively.","b":true},{"text":" "},{"text":"exp(1000)","code":true},{"text":" is infinity in floating point. Every real implementation subtracts the maximum score before exponentiating, which does not change the result mathematically (the constant cancels between numerator and denominator) but keeps every exponent at or below zero."}]},{"t":"code","lang":"python","label":"Python","text":"def softmax(s):\n    s = s - s.max()\n    e = np.exp(s)\n    return e / e.sum()"},{"t":"p","runs":[{"text":"If you ever write this yourself, that one line is the difference between working code and "},{"text":"NaN","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Softmax and sigmoid are the same thing at two classes.","b":true},{"text":" Softmax with two classes reduces exactly to the sigmoid. Nothing new was introduced; the binary case was the special case all along."}]},{"t":"h2","id":"multi-class-is-not-multi-label","runs":[{"text":"Multi-class is not multi-label"}]},{"t":"p","runs":[{"text":"A distinction worth being firm about, because mixing them up produces a model that cannot express the truth."}]},{"t":"p","runs":[{"text":"Multi-class","b":true},{"text":": exactly one label is correct. A photo is a cat, or a dog, or a horse. Softmax, and the probabilities sum to 1."}]},{"t":"p","runs":[{"text":"Multi-label","b":true},{"text":": any number of labels can be correct at once. An article is about politics "},{"text":"and","i":true},{"text":" economics "},{"text":"and","i":true},{"text":" India. Here softmax is wrong: it forces the labels to compete for a fixed budget of probability, so a three-topic article has to divide its confidence three ways. Use independent sigmoids, one per label, each with its own binary cross-entropy and its own threshold. In scikit-learn that is "},{"text":"MultiOutputClassifier","code":true},{"text":" or "},{"text":"OneVsRestClassifier","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Ask of any labelling task: can two labels be true at once? The answer determines the output layer, and getting it wrong caps your accuracy in a way no amount of tuning recovers."}]},{"t":"h2","id":"metrics-need-an-averaging-decision","runs":[{"text":"Metrics need an averaging decision"}]},{"t":"p","runs":[{"text":"Precision and recall are defined per class, so reporting one number for a multi-class problem requires choosing how to combine them, and the choices disagree sharply."}]},{"t":"ul","items":[[{"text":"Macro","b":true},{"text":": compute the metric per class, then take the plain mean. Every class counts equally, so a rare class with 12 examples influences the score as much as a common one with 40,000."}],[{"text":"Weighted","b":true},{"text":": the mean weighted by class frequency. Dominated by the common classes."}],[{"text":"Micro","b":true},{"text":": pool all the decisions and compute once. For single-label multi-class this equals accuracy."}]]},{"t":"p","runs":[{"text":"These can differ by 30 points on an imbalanced problem, and quoting one without saying which is a common way to mislead without lying. If the rare classes are the point (rare diseases, rare failure modes) use macro and say so."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import classification_report\nprint(classification_report(y_te, pred, digits=3))"},{"t":"p","runs":[{"text":"That report gives per-class precision, recall and support, plus all three averages. Read the per-class rows before the summary line. A model with 0.91 weighted F1 that scores 0.0 on your two rarest classes is a specific, common and completely invisible failure if you only look at the average."}]},{"t":"h2","id":"when-there-are-thousands-of-classes","runs":[{"text":"When there are thousands of classes"}]},{"t":"p","runs":[{"text":"At very high class counts (product catalogues, word vocabularies) softmax gets expensive, because every prediction computes a score for every class. The workarounds are hierarchical softmax and sampled softmax, and in retrieval settings the problem is usually reframed as nearest-neighbour search over embeddings, which module 7 covers. Worth knowing the shape of the problem exists; you will not meet it on tabular data."}]}],"check":{"question":[{"text":"A news classifier tags articles with any of eight topics, and articles frequently belong to two or three. The team trains a softmax model over the eight topics and finds it never confidently assigns more than one topic. What is the cause?"}],"options":[{"text":[{"text":"The training data is imbalanced across topics, so the model has collapsed onto the most common one."}],"correct":false,"tempting":[{"text":"Imbalance would skew which topic is chosen, but not create a structural inability to assign several topics at once."}]},{"text":[{"text":"Eight classes is too many for one softmax layer, and the model needs a hierarchical output."}],"correct":false,"tempting":[{"text":"Hierarchical softmax addresses computational cost at thousands of classes, not the number of labels a model can assign per row."}]},{"text":[{"text":"Softmax forces the eight probabilities to sum to 1, so confidence in one topic mechanically reduces confidence in the others; multi-label needs independent sigmoids."}],"correct":true,"tempting":null},{"text":[{"text":"The decision threshold is fixed at the argmax, and lowering it to 0.3 would allow several topics through."}],"correct":false,"tempting":[{"text":"Thresholding softmax outputs can surface secondary topics, but the normalisation still caps total confidence at 1, so a three-topic article cannot be confident about all three."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"making-a-linear-model-curve","title":"Making a straight-line model bend","minutes":8},"next":{"slug":"features-beat-models","title":"Features beat models","minutes":9}}}