When an item has several right answers

One ticket, three labels

Real classification tasks are rarely one label per item. A support message says the payment failed, asks for a refund, and threatens to leave. A post is both political and abusive. A product belongs to three categories. The moment more than one label can be true at once, most of the arithmetic from the confusion-matrix lesson needs a decision before you can apply it.

Start with the strictest possible rule, exact match: the prediction is correct only if the predicted label set equals the true label set. It is easy to compute and it punishes you brutally, for a reason worth seeing in numbers. If items carry five labels and each is predicted correctly 90% of the time, independently:

exact match = 0.90 ** 5 = 0.59

A system that is right about every individual label nine times in ten looks like a 59% system. Nothing is broken; the metric is multiplying. Report exact match only where the product needs the whole set right: populating a form that a human will not review, for instance.

The three usable alternatives

Per-label precision and recall. Treat each label as its own binary problem. This is the diagnostic view and it is where you should always start, because it shows that the failure is concentrated in two labels out of forty.

Sample-averaged F1. For each item, compute the overlap between the predicted and true sets, then average across items. For an item where you predicted {billing, refund} and the truth was {billing, refund, churn-risk}: precision 2/2, recall 2/3, F1 0.8. This answers “how good is a typical prediction”, which is closest to what a user experiences.

Hamming loss. The fraction of individual label decisions that are wrong across the whole grid. It is stable and it is nearly useless on its own, because with forty labels and two true ones, always predicting nothing gives a Hamming loss under 5%.

from sklearn.metrics import classification_report, f1_score
print(classification_report(Y_true, Y_pred, target_names=labels))   # per label
f1_score(Y_true, Y_pred, average="samples")                          # per item

Thresholds are per label, not global

Multi-label models usually emit an independent score per label. A single global cutoff of 0.5 will be far too high for rare labels and too low for common ones, because the model’s score distribution differs per label with the training frequency.

Tune one threshold per label on the development set, maximising whatever that label needs: recall for the safety label, precision for the label that triggers a refund. It is a loop of ten lines and it routinely gains more than a model change. Record the thresholds with the model version; a threshold table that lives in somebody’s notebook is a production incident waiting to happen.

Taxonomies: wrong, and wrong by how much

When labels form a hierarchy (refund above refund-partial and refund-full) flat scoring treats a refund-partial prediction on a refund-full item exactly like predicting billing. That is wrong in a way that matters, because the two errors have different consequences.

The fix is not a clever metric. It is to score at the level the product acts on:

• If the label routes the ticket to a queue and both children go to the same queue, evaluate at the parent level, where the system is right.
• If the label sets the refund amount, evaluate at the leaf level, where the system is wrong.

Compute both and report them as two lines: “parent-level accuracy 91%, leaf-level 68%”. This says precisely what works and what does not, and it stops the argument about whether the model is good.

The gold set is probably incomplete

Here is the failure specific to multi-label work, and it catches good teams. Annotators tag the labels they notice. On a long message they tag two and miss the third. Your model predicts the third correctly, and your evaluation counts it as a false positive.

The consequence is systematic: measured precision is biased downwards, and the bias grows with how good the model is. A model that finds labels humans overlook scores worse.

The way out is a targeted audit. Take 50 predicted labels that the gold set does not contain and have a person judge each one: not “did the annotator write this”, but “is this true of this item”. You get a corrected precision and, usually, a list of annotation-guide holes. Run this before you conclude that precision is falling; roughly a third of the time the model is right and the file is out of date.