Frames that were never photographed

Frames that were never photographed

Three related operations invent picture information that did not exist: interpolation makes new frames between real ones, upscaling makes new pixels within a frame, and video generation makes everything. All three are estimation, and all three fail in ways worth being able to name before you rely on them.

Interpolation, and where it breaks

Classical frame interpolation estimates optical flow (a vector per pixel describing where it moved between two frames) and warps the two frames toward an intermediate time. Learned methods such as RIFE and FILM replace the hand-built flow estimator with a network, and generally produce better results on difficult motion.

Both fail in the same four places, and all four follow from the same cause: flow assumes every pixel in one frame has a corresponding pixel in the other.

• Occlusion boundaries. Where something appears from behind something else, the newly revealed pixels have no correspondence in the earlier frame. The estimator has to invent them, and the result is a smearing or tearing around the edges of moving objects. This is the most common visible artefact.
• Thin structures. A railing, a wire, a spoke, a strand of hair. A thin object can move further than its own width between frames, so there is no overlap to match, and it breaks into fragments or disappears.
• Repetitive texture. A picket fence, brickwork, a striped shirt. Many candidate matches score equally well, and the flow field picks inconsistently across the region, producing a rippling wobble.
• Large motion and blur. Fast pans and heavily blurred frames give the estimator nothing sharp to correspond.

The useful rule: interpolation works well on smooth, moderate motion of large solid objects, and badly on everything else. For genuine slow motion, shoot at the higher rate; the frame-rate lesson makes the same point and it is the same reason.

Upscaling invents plausible detail

A super-resolution model has learned what high-resolution images look like, and given a low-resolution input it produces a high-resolution image consistent with it. Consistent is not the same as correct.

The consequence needs stating plainly: an upscaler does not reveal detail, it fabricates detail. If a number plate is six pixels across, the information identifying its characters is not in the file. An upscaler will produce sharp, legible characters, and they will be a plausible guess. Any use that treats upscaled output as evidence (forensic, journalistic, identifying a person) is treating an invention as a measurement.

The characteristic artefacts:

• Hallucinated text. Small illegible lettering becomes sharp nonsense, or sharp wrong words.
• Faces. Models trained heavily on faces reconstruct a face, which can be a different person’s face. This has real consequences and has been demonstrated repeatedly.
• Over-sharpened texture. Skin becomes plastic, foliage becomes a repeating pattern, film grain becomes a crawling texture.
• Temporal instability on video. A per-frame model reconstructs slightly differently each frame, so the invented detail shimmers. Video-specific models reduce this and do not eliminate it.

Where upscaling is legitimately useful: restoring archive material for viewing, enlarging for a delivery format, cleaning a compressed source. Where it is not: anything where the detail is the point.

Generated video

Text-to-video and image-to-video models produce clips of a few seconds. The state of the art moves quickly and the structural limitations have been persistent:

• Temporal consistency. Objects drift, change colour, and lose or gain parts across a clip. There is no persistent representation of a scene.
• Physics. Contact, weight and momentum are approximated from appearance rather than modelled. Things pass through each other and liquids behave oddly.
• Hands, text and counting. All three remain unreliable, for related reasons: they require a structural constraint the model has learned only statistically.
• Duration. Coherence degrades with length, which is why outputs are short and why longer pieces are assembled from shots.

Treated as a source of short, controllable elements inside an edit (a texture, a background plate, an abstract transition) this is useful. Treated as a replacement for shooting a sequence, it is not there.

Judging any of the three

Play it at full speed, and play it twice. All three failure classes are temporal: a frame-by-frame inspection shows nothing, and the wobble, the shimmer and the drift only exist in motion.

Then look at the specific places named above (occlusion edges, thin objects, text, hands, repeated texture) rather than at the picture in general. The artefacts are predictable, so you can check for them directly.

The free path

• RIFE and FILM: open-source interpolation, run locally, both available as command-line tools and as plugins for free editors.
• ffmpeg’s minterpolate filter: classical motion-compensated interpolation, free and built in, lower quality than the learned methods but present everywhere.
• Real-ESRGAN, BasicVSR++, waifu2x: open-source upscalers, including video-specific ones.
• DaVinci Resolve’s free edition has optical-flow retiming and a super-scale function.
• Open-weight video generation models exist and change too fast for a list here to be useful; check their licences, which frequently restrict commercial use.

The limitation

All three operations share one property: they produce confident output with no signal about how much of it is invented. There is no per-pixel uncertainty map in the file, no warning that this region was extrapolated, no distinction in the output between a pixel derived from the source and one produced from the prior.

That is manageable for entertainment and dangerous everywhere else. The discipline that follows is to keep the original, to say in any handover that the material has been interpolated or upscaled, and never to let a generated frame be used as a record of something that happened.