{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"multi-label-and-partial-credit","title":"When an item has several right answers","url":"https://addaly.com/learn/evaluating-ai/multi-label-and-partial-credit","position":13,"of":76,"minutes":8,"answer":"Exact-match scoring on multi-label tasks collapses under its own arithmetic, so score at the level the product acts on and audit predicted labels by hand, because the gold set is rarely complete.","keyIdea":"Exact-match scoring on multi-label tasks collapses under its own arithmetic, so score at the level the product acts on and audit predicted labels by hand, because the gold set is rarely complete."},"body":{"format":"markdown","text":"## One ticket, three labels\n\nReal classification tasks are rarely one label per item. A support message says the payment failed, asks for a refund, and threatens to leave. A post is both political and abusive. A product belongs to three categories. The moment more than one label can be true at once, most of the arithmetic from the confusion-matrix lesson needs a decision before you can apply it.\n\nStart with the strictest possible rule, **exact match**: the prediction is correct only if the predicted label set equals the true label set. It is easy to compute and it punishes you brutally, for a reason worth seeing in numbers. If items carry five labels and each is predicted correctly 90% of the time, independently:\n\n```\nexact match = 0.90 ** 5 = 0.59\n```\n\nA system that is right about every individual label nine times in ten looks like a 59% system. Nothing is broken; the metric is multiplying. Report exact match only where the product needs the whole set right: populating a form that a human will not review, for instance.\n\n## The three usable alternatives\n\n**Per-label precision and recall.** Treat each label as its own binary problem. This is the diagnostic view and it is where you should always start, because it shows that the failure is concentrated in two labels out of forty.\n\n**Sample-averaged F1.** For each item, compute the overlap between the predicted and true sets, then average across items. For an item where you predicted `{billing, refund}` and the truth was `{billing, refund, churn-risk}`: precision 2/2, recall 2/3, F1 0.8. This answers \"how good is a typical prediction\", which is closest to what a user experiences.\n\n**Hamming loss.** The fraction of individual label decisions that are wrong across the whole grid. It is stable and it is nearly useless on its own, because with forty labels and two true ones, always predicting nothing gives a Hamming loss under 5%.\n\n```python\nfrom sklearn.metrics import classification_report, f1_score\nprint(classification_report(Y_true, Y_pred, target_names=labels))   # per label\nf1_score(Y_true, Y_pred, average=\"samples\")                          # per item\n```\n\n## Thresholds are per label, not global\n\nMulti-label models usually emit an independent score per label. A single global cutoff of 0.5 will be far too high for rare labels and too low for common ones, because the model's score distribution differs per label with the training frequency.\n\nTune one threshold per label on the development set, maximising whatever that label needs: recall for the safety label, precision for the label that triggers a refund. It is a loop of ten lines and it routinely gains more than a model change. Record the thresholds with the model version; a threshold table that lives in somebody's notebook is a production incident waiting to happen.\n\n## Taxonomies: wrong, and wrong by how much\n\nWhen labels form a hierarchy (`refund` above `refund-partial` and `refund-full`) flat scoring treats a `refund-partial` prediction on a `refund-full` item exactly like predicting `billing`. That is wrong in a way that matters, because the two errors have different consequences.\n\nThe fix is not a clever metric. It is to score at the level the product acts on:\n\n- If the label routes the ticket to a queue and both children go to the same queue, evaluate at the parent level, where the system is right.\n- If the label sets the refund amount, evaluate at the leaf level, where the system is wrong.\n\nCompute both and report them as two lines: \"parent-level accuracy 91%, leaf-level 68%\". This says precisely what works and what does not, and it stops the argument about whether the model is good.\n\n## The gold set is probably incomplete\n\nHere is the failure specific to multi-label work, and it catches good teams. Annotators tag the labels they notice. On a long message they tag two and miss the third. Your model predicts the third correctly, and your evaluation counts it as a false positive.\n\nThe consequence is systematic: measured precision is biased downwards, and the bias grows with how good the model is. A model that finds labels humans overlook scores worse.\n\nThe way out is a targeted audit. Take 50 predicted labels that the gold set does not contain and have a person judge each one: not \"did the annotator write this\", but \"is this true of this item\". You get a corrected precision and, usually, a list of annotation-guide holes. Run this before you conclude that precision is falling; roughly a third of the time the model is right and the file is out of date."},"check":{"question":"A multi-label classifier is right about each individual label 92% of the time, and items carry four labels on average. Exact-match accuracy comes out near 72%. What should you conclude?","options":[{"text":"The model has a systematic bias on multi-label items and needs retraining with class weights.","correct":false,"tempting":"Reads a metric artefact as a model defect; 0.92 to the fourth power is 0.72, so this is the arithmetic behaving normally rather than evidence of bias."},{"text":"Exact match multiplies per-label accuracy across labels, so 72% is what a strong per-label model looks like, and the useful views are per-label and per-item scores.","correct":true},{"text":"Exact match is invalid for multi-label problems and should never be reported.","correct":false,"tempting":"Overcorrects: exact match is the right metric wherever the product needs the entire set correct, such as auto-filling a form nobody reviews."},{"text":"Hamming loss would show the same 28% failure rate more clearly, so it should replace exact match.","correct":false,"tempting":"Confuses the two: Hamming loss counts individual label decisions, so it would report something near 8% here, and on a long label list it flatters a model that predicts almost nothing."}]},"prev":{"slug":"the-confusion-matrix","title":"Precision and recall, worked on real numbers","url":"https://addaly.com/learn/evaluating-ai/the-confusion-matrix"},"next":{"slug":"reading-the-curve","title":"What a curve says, and what its one number leaves out","url":"https://addaly.com/learn/evaluating-ai/reading-the-curve"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When an item has several right answers', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/multi-label-and-partial-credit (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When an item has several right answers","courseTitle":"Knowing If It Works","module":{"index":2,"slug":"metrics-that-mean-something","title":"Metrics that mean something"},"answer":null,"keyIdea":"Exact-match scoring on multi-label tasks collapses under its own arithmetic, so score at the level the product acts on and audit predicted labels by hand, because the gold set is rarely complete.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"one-ticket-three-labels","runs":[{"text":"One ticket, three labels"}]},{"t":"p","runs":[{"text":"Real classification tasks are rarely one label per item. A support message says the payment failed, asks for a refund, and threatens to leave. A post is both political and abusive. A product belongs to three categories. The moment more than one label can be true at once, most of the arithmetic from the confusion-matrix lesson needs a decision before you can apply it."}]},{"t":"p","runs":[{"text":"Start with the strictest possible rule, "},{"text":"exact match","b":true},{"text":": the prediction is correct only if the predicted label set equals the true label set. It is easy to compute and it punishes you brutally, for a reason worth seeing in numbers. If items carry five labels and each is predicted correctly 90% of the time, independently:"}]},{"t":"code","lang":"","label":"Text","text":"exact match = 0.90 ** 5 = 0.59"},{"t":"p","runs":[{"text":"A system that is right about every individual label nine times in ten looks like a 59% system. Nothing is broken; the metric is multiplying. Report exact match only where the product needs the whole set right: populating a form that a human will not review, for instance."}]},{"t":"h2","id":"the-three-usable-alternatives","runs":[{"text":"The three usable alternatives"}]},{"t":"p","runs":[{"text":"Per-label precision and recall.","b":true},{"text":" Treat each label as its own binary problem. This is the diagnostic view and it is where you should always start, because it shows that the failure is concentrated in two labels out of forty."}]},{"t":"p","runs":[{"text":"Sample-averaged F1.","b":true},{"text":" For each item, compute the overlap between the predicted and true sets, then average across items. For an item where you predicted "},{"text":"{billing, refund}","code":true},{"text":" and the truth was "},{"text":"{billing, refund, churn-risk}","code":true},{"text":": precision 2/2, recall 2/3, F1 0.8. This answers “how good is a typical prediction”, which is closest to what a user experiences."}]},{"t":"p","runs":[{"text":"Hamming loss.","b":true},{"text":" The fraction of individual label decisions that are wrong across the whole grid. It is stable and it is nearly useless on its own, because with forty labels and two true ones, always predicting nothing gives a Hamming loss under 5%."}]},{"t":"code","lang":"python","label":"Python","text":"from sklearn.metrics import classification_report, f1_score\nprint(classification_report(Y_true, Y_pred, target_names=labels))   # per label\nf1_score(Y_true, Y_pred, average=\"samples\")                          # per item"},{"t":"h2","id":"thresholds-are-per-label-not-global","runs":[{"text":"Thresholds are per label, not global"}]},{"t":"p","runs":[{"text":"Multi-label models usually emit an independent score per label. A single global cutoff of 0.5 will be far too high for rare labels and too low for common ones, because the model’s score distribution differs per label with the training frequency."}]},{"t":"p","runs":[{"text":"Tune one threshold per label on the development set, maximising whatever that label needs: recall for the safety label, precision for the label that triggers a refund. It is a loop of ten lines and it routinely gains more than a model change. Record the thresholds with the model version; a threshold table that lives in somebody’s notebook is a production incident waiting to happen."}]},{"t":"h2","id":"taxonomies-wrong-and-wrong-by-how-much","runs":[{"text":"Taxonomies: wrong, and wrong by how much"}]},{"t":"p","runs":[{"text":"When labels form a hierarchy ("},{"text":"refund","code":true},{"text":" above "},{"text":"refund-partial","code":true},{"text":" and "},{"text":"refund-full","code":true},{"text":") flat scoring treats a "},{"text":"refund-partial","code":true},{"text":" prediction on a "},{"text":"refund-full","code":true},{"text":" item exactly like predicting "},{"text":"billing","code":true},{"text":". That is wrong in a way that matters, because the two errors have different consequences."}]},{"t":"p","runs":[{"text":"The fix is not a clever metric. It is to score at the level the product acts on:"}]},{"t":"ul","items":[[{"text":"If the label routes the ticket to a queue and both children go to the same queue, evaluate at the parent level, where the system is right."}],[{"text":"If the label sets the refund amount, evaluate at the leaf level, where the system is wrong."}]]},{"t":"p","runs":[{"text":"Compute both and report them as two lines: “parent-level accuracy 91%, leaf-level 68%”. This says precisely what works and what does not, and it stops the argument about whether the model is good."}]},{"t":"h2","id":"the-gold-set-is-probably-incomplete","runs":[{"text":"The gold set is probably incomplete"}]},{"t":"p","runs":[{"text":"Here is the failure specific to multi-label work, and it catches good teams. Annotators tag the labels they notice. On a long message they tag two and miss the third. Your model predicts the third correctly, and your evaluation counts it as a false positive."}]},{"t":"p","runs":[{"text":"The consequence is systematic: measured precision is biased downwards, and the bias grows with how good the model is. A model that finds labels humans overlook scores worse."}]},{"t":"p","runs":[{"text":"The way out is a targeted audit. Take 50 predicted labels that the gold set does not contain and have a person judge each one: not “did the annotator write this”, but “is this true of this item”. You get a corrected precision and, usually, a list of annotation-guide holes. Run this before you conclude that precision is falling; roughly a third of the time the model is right and the file is out of date."}]}],"check":{"question":[{"text":"A multi-label classifier is right about each individual label 92% of the time, and items carry four labels on average. Exact-match accuracy comes out near 72%. What should you conclude?"}],"options":[{"text":[{"text":"The model has a systematic bias on multi-label items and needs retraining with class weights."}],"correct":false,"tempting":[{"text":"Reads a metric artefact as a model defect; 0.92 to the fourth power is 0.72, so this is the arithmetic behaving normally rather than evidence of bias."}]},{"text":[{"text":"Exact match multiplies per-label accuracy across labels, so 72% is what a strong per-label model looks like, and the useful views are per-label and per-item scores."}],"correct":true,"tempting":null},{"text":[{"text":"Exact match is invalid for multi-label problems and should never be reported."}],"correct":false,"tempting":[{"text":"Overcorrects: exact match is the right metric wherever the product needs the entire set correct, such as auto-filling a form nobody reviews."}]},{"text":[{"text":"Hamming loss would show the same 28% failure rate more clearly, so it should replace exact match."}],"correct":false,"tempting":[{"text":"Confuses the two: Hamming loss counts individual label decisions, so it would report something near 8% here, and on a long label list it flatters a model that predicts almost nothing."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"the-confusion-matrix","title":"Precision and recall, worked on real numbers","minutes":9},"next":{"slug":"reading-the-curve","title":"What a curve says, and what its one number leaves out","minutes":9}}}