Images: the background the model is actually reading

The three tasks and their metrics

Classification uses accuracy per class and the same confusion matrix arithmetic as anywhere else.

Detection uses mean average precision at an intersection-over-union threshold. IoU is the overlap of the predicted and true boxes divided by their union, and a prediction counts as correct only above the threshold: commonly 0.5.

That threshold has a sharp consequence worth seeing. A box that is correct in every practical sense but shifted by a quarter of its width can fall below 0.5, and it is then counted twice against you: once as a false positive, once as a missed detection. So mAP at 0.5 and mAP at 0.75 can rank two models differently, and quoting one without saying which is not a result. Report the IoU threshold every time, and look at the localisation error separately from the classification error.

Segmentation uses mean IoU per class, where thin structures and boundaries dominate the error and human annotators disagree about them too.

Your test set probably leaks through the session

The classic vision leak is not a duplicated file. It is the photo session: forty images of the same object, taken in one place with one camera in one minute, split at random across train and test. The model then recognises the carpet.

Split by capture session, by device, by patient, by shop: whatever unit generated a batch of similar images. And run a near-duplicate check before trusting any split:

import imagehash
from PIL import Image
h = {p: imagehash.phash(Image.open(p)) for p in paths}
# pairs within a small Hamming distance are near-duplicates

Shortcuts: what the model is keyed on

Models learn whatever separates the classes fastest, which is often the background. Cows are photographed on grass, so a cow on a beach stops being a cow. Medical imaging models learn the scanner, the ward label burned into the corner, or the ruler that clinicians place beside a suspicious lesion.

The test is direct: build a slice where the correlation is broken (the object on an unusual background, images from a different site, the marker removed) and measure the gap. A model at 94% on the ordinary set and 61% on the broken-correlation set is not a 94% model. Two hundred such images will tell you more than ten thousand ordinary ones.

Gradient-based attribution maps are worth one afternoon here, not as evidence but as a lead: they show where the model is looking, and you check the suspicion properly with a slice.

The images your users will actually send

An evaluation built from clean catalogue photographs measures a product nobody has. Real intake is a phone camera, held at an angle, in a shop with a fluorescent tube, at 40% battery with the screen dimmed.

Build a perturbation suite and report accuracy at each level: Gaussian blur, JPEG compression at quality 40 and 20, brightness at half and double, 10-degree rotation, motion blur, and downscaling to 480 pixels. albumentations and PIL do all of it free, and the curve of accuracy against perturbation level is a more useful artefact than any single number: it tells the product team what to ask the camera screen to enforce.

Detection has a threshold problem of its own

Every detector emits a confidence per box, and mAP integrates over all of them, which means it describes no operating point you will ship. Pick the confidence threshold the product will use, and report ordinary precision and recall there, per class. A model with mAP 0.68 that must run at 0.9 precision to keep the review queue sane is a different proposition from one with mAP 0.66 that holds 0.9 precision at twice the recall.

Group slices, handled carefully

Where the images are of people, performance differs by skin tone, age and lighting, and reporting an average hides it. The complication is that measuring this requires the sensitive attributes you are trying to be careful about. The workable practice is to use an established annotation scale, collect it for the evaluation set only, keep it access-controlled, state in the report that it exists and why, and do not attach it to production records.

Two honest limits

Label noise in image sets is real, several per cent even in well-known public sets, so re-label 200 items yourself and measure the disagreement rate before treating the last two points of accuracy as signal. And a model evaluated only on your set tells you nothing about images from a new market, a new device generation, or a new season; that gap is measured after deployment, with the monitoring covered later in this course.