Your clip is compressed fifty times before anything happens

The number that explains most of the artefacts

Take a five-second clip at 1080p and 24 frames per second. That is 120 frames, each 1920 by 1080 with three colour channels: about 746 million numbers.

No transformer attends over 746 million numbers. So the first thing a video model does is hand your frames to an autoencoder (usually called a 3D VAE or a causal video VAE) which squeezes them. A typical configuration compresses by 8× in width, 8× in height, 4× along time, and expands the channel count from 3 to 16.

Run the arithmetic, setting aside for now the extra first frame that causal encoders keep (covered below). 120 frames become 30 latent frames, or 31 counting that first frame. 1920×1080 becomes 240×135. With 16 channels that is 30 × 240 × 135 × 16 ≈ 15.5 million numbers.

746 million in, 15.5 million out. Roughly forty-eight times smaller. Generation happens entirely in that small space. The decoder then expands it back to pixels at the end.

What survives the squeeze, and what does not

The encoder is not a zip file. It is lossy, and it is lossy in a specific, predictable way: it keeps what it was trained to consider important and discards the rest.

What survives well:

• Large shapes, silhouettes, overall colour, the direction of light.
• Smooth gradients and broad textures.
• Motion of large objects.

What does not survive:

• Anything smaller than about an 8×8 pixel block at output resolution. That is a letter in small type, the weave of a fabric, a thin wire, an eyelash.
• High-frequency noise, including real camera grain.
• Fine specular highlights: the tiny bright points on wet surfaces, metal edges, eyes.

When you see those things in the output, they were not preserved. The decoder invented plausible replacements. That is why generated footage has a slightly waxy, denoised surface even when nothing has obviously gone wrong. It is also why text melts: the letterform was never in the latent to begin with.

The round-trip test you can run for free

This is the single most convincing demonstration in the subject, and it costs nothing because it involves no generation at all.

In ComfyUI, build a short graph: load a real video, split it into frames, encode them with the model’s VAE (a Load VAE node feeds both the encoder and the decoder), decode them straight back out, and reassemble the video at the original frame rate. No diffusion, no prompt, no sampling. Compare input and output.

LoadVideo -> GetVideoComponents -> VAEEncode -> VAEDecode -> CreateVideo -> SaveVideo

What comes back is the absolute ceiling on quality for that model. Nothing generated through that VAE will ever be sharper than your round-tripped real footage. Do it once with a clip containing a sign, a patterned shirt and a face, and you will stop blaming the sampler for things the encoder did.

If you cannot run ComfyUI, several Hugging Face Spaces expose the same encode-decode path in a browser.

Causal encoders and why the first frame is different

Most current video VAEs are causal: latent frame n is computed from pixel frames at or before it, never from later ones. The practical consequence is that the very first frame is encoded on its own, without temporal compression, while every later group of four frames is squeezed together.

That asymmetry is part of why image-to-video works as well as it does. Your supplied still enters the latent space with less loss than any frame that follows it. It is the sharpest frame in the clip before generation even starts.

It is also why the first quarter-second often looks subtly different from the rest: the settle described in the previous lesson has an encoder component as well as an attention component.

What you do differently knowing this

1. Do not put load-bearing fine detail in the still. If the logo has to be legible, it goes on in the editor as real text over the top, not in the frame you animate.
2. Do not send a 4000-pixel still into a 720p model. It is downscaled on the way in. You paid for pixels that were deleted before the model saw them.
3. Do not expect an upscaler to recover what the VAE removed. Upscaling invents new detail; it cannot restore the specific detail that was discarded. It is a finishing step, not a repair.
4. Add grain at the end, deliberately. The encoder removed real grain. A uniform grain pass over the finished timeline replaces a texture the pipeline is structurally unable to keep, and it is why generated footage that has been grained reads as film while the raw export reads as CGI.
5. Judge a model’s VAE separately from its sampler. Two models with the same sampler quality can differ enormously in how much the decoder smears. The round-trip test isolates it.

The limitation

Bigger latents would help. Some newer models compress less aggressively along time, or use more channels, and they look visibly crisper. They also cost proportionally more to generate, because everything downstream scales with the size of the latent. The waxiness is not a bug somebody forgot to fix: it is the price paid to make the attention step affordable at all, and it moves only when somebody accepts a bigger bill.

Today: run the encode-decode round trip on ten seconds of your own phone footage. Whatever comes back is the best this model will ever give you.