{"course":{"slug":"ai-video","title":"Video with AI","url":"https://addaly.com/learn/ai-video","level":"intermediate","track":"create"},"lesson":{"slug":"latent-space-for-video","title":"Your clip is compressed fifty times before anything happens","url":"https://addaly.com/learn/ai-video/latent-space-for-video","position":2,"of":66,"minutes":10,"answer":"A video autoencoder throws away roughly ninety-eight per cent of the pixel data before generation begins, so any detail finer than an eight-pixel block is being reinvented by the decoder rather than preserved.","keyIdea":"A video autoencoder throws away roughly ninety-eight per cent of the pixel data before generation begins, so any detail finer than an eight-pixel block is being reinvented by the decoder rather than preserved."},"body":{"format":"markdown","text":"## The number that explains most of the artefacts\n\nTake a five-second clip at 1080p and 24 frames per second. That is 120 frames, each 1920 by 1080 with three colour channels: about 746 million numbers.\n\nNo transformer attends over 746 million numbers. So the first thing a video model does is hand your frames to an autoencoder (usually called a 3D VAE or a causal video VAE) which squeezes them. A typical configuration compresses by 8× in width, 8× in height, 4× along time, and expands the channel count from 3 to 16.\n\nRun the arithmetic, setting aside for now the extra first frame that causal encoders keep (covered below). 120 frames become 30 latent frames, or 31 counting that first frame. 1920×1080 becomes 240×135. With 16 channels that is 30 × 240 × 135 × 16 ≈ 15.5 million numbers.\n\n746 million in, 15.5 million out. Roughly forty-eight times smaller. Generation happens entirely in that small space. The decoder then expands it back to pixels at the end.\n\n> Figure: Where the 746 million numbers go. Layers, top to bottom: Pixels: 746 million numbers (A five-second 1080p clip at 24 frames per second. 120 frames, each 1920 by 1080, in three colour channels. No transformer attends over this.); Encoder: 8 by 8 by 4 (Eight times narrower, eight times shorter, four times fewer frames, with channels rising from 3 to 16. Lossy, and lossy in a predictable direction.); Latent: 15.5 million numbers (30 latent frames at 240 by 135 by 16 channels. Roughly forty-eight times smaller. Every step of generation happens in here and nowhere else.); Decoder: back to pixels (Anything finer than about an eight-pixel block was never carried, so the decoder invents a plausible replacement: small type, fabric weave, a thin wire, an eyelash.). The squeeze is the reason the attention step is affordable at all, and it is also the reason generated footage has a slightly waxy surface and melting text. Neither was preserved and then damaged. Neither was ever in the latent.\n\n## What survives the squeeze, and what does not\n\nThe encoder is not a zip file. It is lossy, and it is lossy in a specific, predictable way: it keeps what it was trained to consider important and discards the rest.\n\nWhat survives well:\n\n- Large shapes, silhouettes, overall colour, the direction of light.\n- Smooth gradients and broad textures.\n- Motion of large objects.\n\nWhat does not survive:\n\n- Anything smaller than about an 8×8 pixel block at output resolution. That is a letter in small type, the weave of a fabric, a thin wire, an eyelash.\n- High-frequency noise, including real camera grain.\n- Fine specular highlights: the tiny bright points on wet surfaces, metal edges, eyes.\n\nWhen you see those things in the output, they were not preserved. The decoder invented plausible replacements. That is why generated footage has a slightly waxy, denoised surface even when nothing has obviously gone wrong. It is also why text melts: the letterform was never in the latent to begin with.\n\n## The round-trip test you can run for free\n\nThis is the single most convincing demonstration in the subject, and it costs nothing because it involves no generation at all.\n\nIn ComfyUI, build a short graph: load a real video, split it into frames, encode them with the model's VAE (a Load VAE node feeds both the encoder and the decoder), decode them straight back out, and reassemble the video at the original frame rate. No diffusion, no prompt, no sampling. Compare input and output.\n\n```\nLoadVideo -> GetVideoComponents -> VAEEncode -> VAEDecode -> CreateVideo -> SaveVideo\n```\n\nWhat comes back is the absolute ceiling on quality for that model. Nothing generated through that VAE will ever be sharper than your round-tripped real footage. Do it once with a clip containing a sign, a patterned shirt and a face, and you will stop blaming the sampler for things the encoder did.\n\nIf you cannot run ComfyUI, several Hugging Face Spaces expose the same encode-decode path in a browser.\n\n## Causal encoders and why the first frame is different\n\nMost current video VAEs are causal: latent frame *n* is computed from pixel frames at or before it, never from later ones. The practical consequence is that the very first frame is encoded on its own, without temporal compression, while every later group of four frames is squeezed together.\n\nThat asymmetry is part of why image-to-video works as well as it does. Your supplied still enters the latent space with less loss than any frame that follows it. It is the sharpest frame in the clip before generation even starts.\n\nIt is also why the first quarter-second often looks subtly different from the rest: the settle described in the previous lesson has an encoder component as well as an attention component.\n\n## What you do differently knowing this\n\n1. **Do not put load-bearing fine detail in the still.** If the logo has to be legible, it goes on in the editor as real text over the top, not in the frame you animate.\n2. **Do not send a 4000-pixel still into a 720p model.** It is downscaled on the way in. You paid for pixels that were deleted before the model saw them.\n3. **Do not expect an upscaler to recover what the VAE removed.** Upscaling invents new detail; it cannot restore the specific detail that was discarded. It is a finishing step, not a repair.\n4. **Add grain at the end, deliberately.** The encoder removed real grain. A uniform grain pass over the finished timeline replaces a texture the pipeline is structurally unable to keep, and it is why generated footage that has been grained reads as film while the raw export reads as CGI.\n5. **Judge a model's VAE separately from its sampler.** Two models with the same sampler quality can differ enormously in how much the decoder smears. The round-trip test isolates it.\n\n## The limitation\n\nBigger latents would help. Some newer models compress less aggressively along time, or use more channels, and they look visibly crisper. They also cost proportionally more to generate, because everything downstream scales with the size of the latent. The waxiness is not a bug somebody forgot to fix: it is the price paid to make the attention step affordable at all, and it moves only when somebody accepts a bigger bill.\n\n---\n\nToday: run the encode-decode round trip on ten seconds of your own phone footage. Whatever comes back is the best this model will ever give you."},"check":{"question":"You generate a shot containing a shop sign. The letters shimmer and reshape across the clip. You raise the sampler steps from 20 to 50 and regenerate. The letters still shimmer. Why?","options":[{"text":"Fifty steps is still too few for text; letterforms typically need 80 to 120 steps before they stabilise.","correct":false,"tempting":"Treats a representational limit as a convergence problem, and invents a step count at which a discarded detail reappears."},{"text":"The letters are below the resolution the autoencoder preserves, so the decoder reinvents them each frame however well the latent was denoised.","correct":true},{"text":"The prompt did not specify the text clearly enough, so the model had no target to converge on.","correct":false,"tempting":"Assumes the failure is in conditioning rather than in the pixel budget; a perfectly specified sign fails identically."},{"text":"The temporal attention window is shorter than the clip, so the letters lose their anchor to the first frame partway through and begin drifting away from it.","correct":false,"tempting":"Names a real mechanism that causes other drift, but text shimmers even inside a single attention window because the detail was never in the latent."}]},"prev":{"slug":"why-clips-are-short","title":"Nobody is hiding the long version from you","url":"https://addaly.com/learn/ai-video/why-clips-are-short"},"next":{"slug":"denoising-a-block-of-time","title":"The whole clip is made at once, not frame by frame","url":"https://addaly.com/learn/ai-video/denoising-a-block-of-time"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Your clip is compressed fifty times before anything happens', in Video with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-video/latent-space-for-video (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Your clip is compressed fifty times before anything happens","courseTitle":"Video with AI","module":{"index":1,"slug":"how-video-models-work","title":"What the machine is actually doing"},"answer":null,"keyIdea":"A video autoencoder throws away roughly ninety-eight per cent of the pixel data before generation begins, so any detail finer than an eight-pixel block is being reinvented by the decoder rather than preserved.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Where the 746 million numbers go","layers":[{"label":"Pixels: 746 million numbers","detail":"A five-second 1080p clip at 24 frames per second. 120 frames, each 1920 by 1080, in three colour channels. No transformer attends over this."},{"label":"Encoder: 8 by 8 by 4","detail":"Eight times narrower, eight times shorter, four times fewer frames, with channels rising from 3 to 16. Lossy, and lossy in a predictable direction."},{"label":"Latent: 15.5 million numbers","detail":"30 latent frames at 240 by 135 by 16 channels. Roughly forty-eight times smaller. Every step of generation happens in here and nowhere else."},{"label":"Decoder: back to pixels","detail":"Anything finer than about an eight-pixel block was never carried, so the decoder invents a plausible replacement: small type, fabric weave, a thin wire, an eyelash."}],"caption":"The squeeze is the reason the attention step is affordable at all, and it is also the reason generated footage has a slightly waxy surface and melting text. Neither was preserved and then damaged. Neither was ever in the latent."},"alt":"Figure: Where the 746 million numbers go. Layers, top to bottom: Pixels: 746 million numbers (A five-second 1080p clip at 24 frames per second. 120 frames, each 1920 by 1080, in three colour channels. No transformer attends over this.); Encoder: 8 by 8 by 4 (Eight times narrower, eight times shorter, four times fewer frames, with channels rising from 3 to 16. Lossy, and lossy in a predictable direction.); Latent: 15.5 million numbers (30 latent frames at 240 by 135 by 16 channels. Roughly forty-eight times smaller. Every step of generation happens in here and nowhere else.); Decoder: back to pixels (Anything finer than about an eight-pixel block was never carried, so the decoder invents a plausible replacement: small type, fabric weave, a thin wire, an eyelash.). The squeeze is the reason the attention step is affordable at all, and it is also the reason generated footage has a slightly waxy surface and melting text. Neither was preserved and then damaged. Neither was ever in the latent."}},"blocks":[{"t":"h2","id":"the-number-that-explains-most-of-the-artefacts","runs":[{"text":"The number that explains most of the artefacts"}]},{"t":"p","runs":[{"text":"Take a five-second clip at 1080p and 24 frames per second. That is 120 frames, each 1920 by 1080 with three colour channels: about 746 million numbers."}]},{"t":"p","runs":[{"text":"No transformer attends over 746 million numbers. So the first thing a video model does is hand your frames to an autoencoder (usually called a 3D VAE or a causal video VAE) which squeezes them. A typical configuration compresses by 8× in width, 8× in height, 4× along time, and expands the channel count from 3 to 16."}]},{"t":"p","runs":[{"text":"Run the arithmetic, setting aside for now the extra first frame that causal encoders keep (covered below). 120 frames become 30 latent frames, or 31 counting that first frame. 1920×1080 becomes 240×135. With 16 channels that is 30 × 240 × 135 × 16 ≈ 15.5 million numbers."}]},{"t":"p","runs":[{"text":"746 million in, 15.5 million out. Roughly forty-eight times smaller. Generation happens entirely in that small space. The decoder then expands it back to pixels at the end."}]},{"t":"h2","id":"what-survives-the-squeeze-and-what-does-not","runs":[{"text":"What survives the squeeze, and what does not"}]},{"t":"p","runs":[{"text":"The encoder is not a zip file. It is lossy, and it is lossy in a specific, predictable way: it keeps what it was trained to consider important and discards the rest."}]},{"t":"p","runs":[{"text":"What survives well:"}]},{"t":"ul","items":[[{"text":"Large shapes, silhouettes, overall colour, the direction of light."}],[{"text":"Smooth gradients and broad textures."}],[{"text":"Motion of large objects."}]]},{"t":"p","runs":[{"text":"What does not survive:"}]},{"t":"ul","items":[[{"text":"Anything smaller than about an 8×8 pixel block at output resolution. That is a letter in small type, the weave of a fabric, a thin wire, an eyelash."}],[{"text":"High-frequency noise, including real camera grain."}],[{"text":"Fine specular highlights: the tiny bright points on wet surfaces, metal edges, eyes."}]]},{"t":"p","runs":[{"text":"When you see those things in the output, they were not preserved. The decoder invented plausible replacements. That is why generated footage has a slightly waxy, denoised surface even when nothing has obviously gone wrong. It is also why text melts: the letterform was never in the latent to begin with."}]},{"t":"h2","id":"the-round-trip-test-you-can-run-for-free","runs":[{"text":"The round-trip test you can run for free"}]},{"t":"p","runs":[{"text":"This is the single most convincing demonstration in the subject, and it costs nothing because it involves no generation at all."}]},{"t":"p","runs":[{"text":"In ComfyUI, build a short graph: load a real video, split it into frames, encode them with the model’s VAE (a Load VAE node feeds both the encoder and the decoder), decode them straight back out, and reassemble the video at the original frame rate. No diffusion, no prompt, no sampling. Compare input and output."}]},{"t":"code","lang":"","label":"Text","text":"LoadVideo -> GetVideoComponents -> VAEEncode -> VAEDecode -> CreateVideo -> SaveVideo"},{"t":"p","runs":[{"text":"What comes back is the absolute ceiling on quality for that model. Nothing generated through that VAE will ever be sharper than your round-tripped real footage. Do it once with a clip containing a sign, a patterned shirt and a face, and you will stop blaming the sampler for things the encoder did."}]},{"t":"p","runs":[{"text":"If you cannot run ComfyUI, several Hugging Face Spaces expose the same encode-decode path in a browser."}]},{"t":"h2","id":"causal-encoders-and-why-the-first-frame-is-different","runs":[{"text":"Causal encoders and why the first frame is different"}]},{"t":"p","runs":[{"text":"Most current video VAEs are causal: latent frame "},{"text":"n","i":true},{"text":" is computed from pixel frames at or before it, never from later ones. The practical consequence is that the very first frame is encoded on its own, without temporal compression, while every later group of four frames is squeezed together."}]},{"t":"p","runs":[{"text":"That asymmetry is part of why image-to-video works as well as it does. Your supplied still enters the latent space with less loss than any frame that follows it. It is the sharpest frame in the clip before generation even starts."}]},{"t":"p","runs":[{"text":"It is also why the first quarter-second often looks subtly different from the rest: the settle described in the previous lesson has an encoder component as well as an attention component."}]},{"t":"h2","id":"what-you-do-differently-knowing-this","runs":[{"text":"What you do differently knowing this"}]},{"t":"ol","start":1,"items":[[{"text":"Do not put load-bearing fine detail in the still.","b":true},{"text":" If the logo has to be legible, it goes on in the editor as real text over the top, not in the frame you animate."}],[{"text":"Do not send a 4000-pixel still into a 720p model.","b":true},{"text":" It is downscaled on the way in. You paid for pixels that were deleted before the model saw them."}],[{"text":"Do not expect an upscaler to recover what the VAE removed.","b":true},{"text":" Upscaling invents new detail; it cannot restore the specific detail that was discarded. It is a finishing step, not a repair."}],[{"text":"Add grain at the end, deliberately.","b":true},{"text":" The encoder removed real grain. A uniform grain pass over the finished timeline replaces a texture the pipeline is structurally unable to keep, and it is why generated footage that has been grained reads as film while the raw export reads as CGI."}],[{"text":"Judge a model’s VAE separately from its sampler.","b":true},{"text":" Two models with the same sampler quality can differ enormously in how much the decoder smears. The round-trip test isolates it."}]]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"Bigger latents would help. Some newer models compress less aggressively along time, or use more channels, and they look visibly crisper. They also cost proportionally more to generate, because everything downstream scales with the size of the latent. The waxiness is not a bug somebody forgot to fix: it is the price paid to make the attention step affordable at all, and it moves only when somebody accepts a bigger bill."}]},{"t":"rule"},{"t":"p","runs":[{"text":"Today: run the encode-decode round trip on ten seconds of your own phone footage. Whatever comes back is the best this model will ever give you."}]}],"check":{"question":[{"text":"You generate a shot containing a shop sign. The letters shimmer and reshape across the clip. You raise the sampler steps from 20 to 50 and regenerate. The letters still shimmer. Why?"}],"options":[{"text":[{"text":"Fifty steps is still too few for text; letterforms typically need 80 to 120 steps before they stabilise."}],"correct":false,"tempting":[{"text":"Treats a representational limit as a convergence problem, and invents a step count at which a discarded detail reappears."}]},{"text":[{"text":"The letters are below the resolution the autoencoder preserves, so the decoder reinvents them each frame however well the latent was denoised."}],"correct":true,"tempting":null},{"text":[{"text":"The prompt did not specify the text clearly enough, so the model had no target to converge on."}],"correct":false,"tempting":[{"text":"Assumes the failure is in conditioning rather than in the pixel budget; a perfectly specified sign fails identically."}]},{"text":[{"text":"The temporal attention window is shorter than the clip, so the letters lose their anchor to the first frame partway through and begin drifting away from it."}],"correct":false,"tempting":[{"text":"Names a real mechanism that causes other drift, but text shimmers even inside a single attention window because the detail was never in the latent."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"why-clips-are-short","title":"Nobody is hiding the long version from you","minutes":8},"next":{"slug":"denoising-a-block-of-time","title":"The whole clip is made at once, not frame by frame","minutes":10}}}