Why detection is the wrong hope

Four independent reasons, any one of which would be enough

Generalisation. A detector is a classifier trained on real and generated examples. It learns what the generators in its training set leave behind — characteristic frequency signatures, upsampling traces, statistical regularities in noise. Those fingerprints are specific to an architecture and to a version. A model released after the detector produces different traces, and measured performance on unseen generators falls sharply, sometimes to near chance. The detector is always chasing.

Fragility. The evidence lives in fine detail. Re-encoding a JPEG, resizing, cropping, applying a filter, uploading to a platform that recompresses, screenshotting, or photographing a screen — each of these removes a substantial part of the signal. By the time an image has passed through two social platforms, most detectors are guessing.

Adversarial pressure. A public detector is a target. Anyone who wants to evade it can test against it and adjust until it passes. This is not hypothetical; it is a standard research result across every detection domain.

Base rates. This is the one that matters operationally and the one people skip.

The arithmetic worth doing once

Suppose a detector is 95% accurate in both directions, which is better than most are in the field. Suppose one image in a thousand in some feed is generated.

Out of 100,000 images: 100 are generated, of which the detector flags 95. The other 99,900 are real, and it wrongly flags 5% of them — about 4,995.

So of roughly 5,090 flagged images, about 95 are actually generated. Fewer than two in a hundred flags are correct.

Nothing is wrong with the detector. The rarity of the thing being detected swamps a good error rate. Any screening system for a rare event has this property, and it is why medical screening programmes are designed around confirmatory testing rather than around the screen alone.

The consequence is that a positive result from a detector is a reason to look more carefully, and never a reason to accuse.

The precedent worth knowing

Text detection went through this cycle publicly and quickly. A widely-publicised detector for AI-written text was withdrawn by its own developer within seven months, on the stated grounds of low accuracy — it correctly identified a minority of generated text while wrongly flagging some genuine human writing.

Meanwhile students were being accused of cheating on detector output, and writing in a plain, structured style — which is what non-native English writers and careful technical writers produce — was flagged disproportionately. The harm landed on people with the least ability to contest it.

Image and video detection is subject to the same forces with worse fragility, because pixels get recompressed far more aggressively than text does.

What detectors are actually good for

They are not useless, and it is worth being precise about the fit.

Triage inside a system with a second step. Flag for human review, route to a slower path, request additional verification. The false positives then cost a check rather than an accusation.

Aggregate measurement. "Roughly what proportion of uploads this month were generated" tolerates a poor per-item error rate because the errors partly cancel.

Known-generator settings. A closed platform checking its own outputs against its own detector, where the generator is known and the file is unmodified.

What they are not good for: deciding whether the picture in front of you is real, in public, about a named person.

The direction that works instead

Detection asks "does this look generated". Provenance asks "where did this come from". The second question has an answer that improves as the generators improve, because it depends on the record rather than on the artefact.

That is the subject of the next three lessons. The important shift is one of framing: stop trying to prove a negative about the pixels and start establishing a positive about the origin.

One consequence is worth naming for anyone in a position of authority over other people — a teacher, an editor, a manager, a moderator. Do not act on a detector's output alone against an individual. Not because the tool is worthless, but because the arithmetic above means most of the people you flag will be innocent, and the cost falls entirely on them while the cost of your error falls on nobody. If a flag is the beginning of a conversation in which the person can show their working, the tool is doing something useful. If it is the end of the conversation, it is producing false accusations at a rate you would never accept if the number were printed on the screen.