{"course":{"slug":"ai-video","title":"Video with AI","url":"https://addaly.com/learn/ai-video","level":"intermediate","track":"create"},"lesson":{"slug":"attention-across-frames","title":"Why a chair stays a chair and a face does not","url":"https://addaly.com/learn/ai-video/attention-across-frames","position":4,"of":66,"minutes":10,"answer":"Attention lets every patch of every frame see every other, which is what holds a scene together, but it costs the square of the token count and carries no object identity.","keyIdea":"Attention lets every patch of every frame see every other, which is what holds a scene together, but it costs the square of the token count and carries no object identity, so coherence is an emergent statistical effect, not a guarantee about any particular thing."},"body":{"format":"markdown","text":"## What attention is doing here\n\nAfter the encoder, the latent is cut into patches: typically 2×2 in space, sometimes across a pair of latent frames as well. Each patch becomes a token. For our five-second 1080p clip: 30 latent frames, 240×135 latent pixels, patched 2×2, gives 30 × 120 × 68 ≈ 244,800 tokens.\n\nIn **full 3D attention**, every one of those tokens computes a relationship with every other one. That is 244,800² ≈ 60 billion pairs, per attention layer, per step. Modern kernels do not materialise that matrix, but the work still scales that way.\n\nThis is the expensive thing in the whole pipeline, and it is also the thing that makes video models work. A patch on frame 118 can look directly at a patch on frame 3. That is how the wall stays where it was.\n\n## Full, factorised, and windowed\n\nThree arrangements, with a clear trade-off.\n\n- **Full 3D attention.** Every token sees every token. Best coherence, highest cost. Used by the models with the best physical consistency, and the reason they are expensive.\n- **Factorised attention.** Attend across space within each frame, then across time at each spatial position, in alternating layers. Cost drops enormously. Objects that move diagonally across the frame are handled worse, because a token only attends along its own row through time.\n- **Windowed or sparse attention.** Tokens attend within a local neighbourhood in space and time, with occasional global layers. Cheapest. Long-range agreement weakens, which shows up as slow drift over a clip rather than sudden breakage.\n\nYou cannot usually pick. It is baked into the model. But you can read the symptom: if a model handles fast diagonal motion badly while holding static scenes well, you are probably looking at factorisation. If everything holds for three seconds and then slowly slides, suspect a window.\n\n## Attention has no idea what an object is\n\nThis is the part that matters most, and it is routinely misunderstood.\n\nThere is no variable anywhere in the model holding \"the red mug\". There are tokens whose values happen to correlate strongly across frames because they were trained on real footage where mugs persist. Persistence is a statistical habit, not a stored fact.\n\nThree consequences follow directly.\n\n**Occlusion is dangerous.** When something passes behind something else, the tokens that carried it stop existing for a while. When it reappears, nothing retrieves the original; the model re-samples something consistent with context. A person who walks behind a pillar can come back with a different shirt, and no setting prevents it.\n\n**Leaving frame is worse than occlusion.** An object that exits the edge and returns has been gone from every token for many frames. Expect it to come back changed.\n\n**Count is not conserved.** Three birds can become four. There is nothing counting.\n\nThe production response is to design around it. Do not write shots where a specific object leaves and returns. Cut instead: the cut is free and the re-entry is not.\n\n## Why faces break first\n\nThe model is no worse at faces than at chairs. You are far better at faces.\n\nHuman vision has dedicated machinery for facial identity, tuned to differences of a few per cent in the geometry between features. A chair whose back changes by thirty per cent still reads as the same chair. A face whose inter-ocular distance shifts by three per cent reads as a different person.\n\nSo the same drift rate produces a shrug on furniture and a rejected take on a face. That asymmetry is in your visual cortex, not in the model, and knowing it tells you where to spend: shorter shots on faces, longer shots on everything else.\n\n## Text conditioning is a small part of this\n\nYour prompt enters through **cross-attention**: text tokens from an encoder such as T5 or a CLIP variant, attended to by the video tokens. There are a few hundred text tokens against a quarter of a million video tokens.\n\nThat ratio is worth sitting with. The prompt is a light steering influence over an enormous amount of structure that is mostly determined by the first frame and the model's priors. It explains why adding a fourth clause to a prompt often changes nothing, and why the still frame overwhelms the text every time the two disagree.\n\n## What you can act on\n\n1. Shots where an object leaves and returns are not shots. Split them.\n2. Put the face in fewer, shorter shots than your instinct says.\n3. Prefer motion that stays roughly in the same region of frame: a push-in holds better than a fast lateral whip, on factorised models especially.\n4. When a model drifts slowly rather than breaking suddenly, do not fight it with prompts. Shorten the clip.\n5. Expect the prompt to lose any argument with the first frame.\n\n---\n\nToday: generate one shot where a person walks behind a column and out the other side. Watch what comes back. That single clip teaches the whole lesson better than the arithmetic does."},"check":{"question":"In a generated shot, a woman carrying a blue bag walks behind a pillar and emerges with a green bag. What does this reveal about the model?","options":[{"text":"Nothing stores the bag, so once its tokens are occluded the model re-samples something merely consistent with the surroundings.","correct":true},{"text":"The temporal attention window ended at the pillar, so the frames after it had no access to the frames before.","correct":false,"tempting":"Assumes a window boundary landed exactly at the occlusion; the same failure occurs well inside a window."},{"text":"The prompt described the bag only once, and prompt influence decays across the length of a clip so later frames were effectively unconditioned.","correct":false,"tempting":"Imagines the text conditioning weakening over time; cross-attention applies equally to every frame in the block."},{"text":"The autoencoder discarded the bag's colour because it fell below the preserved spatial resolution, and the decoder chose a new one.","correct":false,"tempting":"Misapplies the VAE limit, which affects fine detail rather than the colour of a large object."}]},"prev":{"slug":"denoising-a-block-of-time","title":"The whole clip is made at once, not frame by frame","url":"https://addaly.com/learn/ai-video/denoising-a-block-of-time"},"next":{"slug":"how-models-learned-motion","title":"The prompt vocabulary was written by whoever captioned the training clips","url":"https://addaly.com/learn/ai-video/how-models-learned-motion"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Why a chair stays a chair and a face does not', in Video with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-video/attention-across-frames (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Why a chair stays a chair and a face does not","courseTitle":"Video with AI","module":{"index":1,"slug":"how-video-models-work","title":"What the machine is actually doing"},"answer":null,"keyIdea":"Attention lets every patch of every frame see every other, which is what holds a scene together, but it costs the square of the token count and carries no object identity, so coherence is an emergent statistical effect, not a guarantee about any particular thing.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-attention-is-doing-here","runs":[{"text":"What attention is doing here"}]},{"t":"p","runs":[{"text":"After the encoder, the latent is cut into patches: typically 2×2 in space, sometimes across a pair of latent frames as well. Each patch becomes a token. For our five-second 1080p clip: 30 latent frames, 240×135 latent pixels, patched 2×2, gives 30 × 120 × 68 ≈ 244,800 tokens."}]},{"t":"p","runs":[{"text":"In "},{"text":"full 3D attention","b":true},{"text":", every one of those tokens computes a relationship with every other one. That is 244,800² ≈ 60 billion pairs, per attention layer, per step. Modern kernels do not materialise that matrix, but the work still scales that way."}]},{"t":"p","runs":[{"text":"This is the expensive thing in the whole pipeline, and it is also the thing that makes video models work. A patch on frame 118 can look directly at a patch on frame 3. That is how the wall stays where it was."}]},{"t":"h2","id":"full-factorised-and-windowed","runs":[{"text":"Full, factorised, and windowed"}]},{"t":"p","runs":[{"text":"Three arrangements, with a clear trade-off."}]},{"t":"ul","items":[[{"text":"Full 3D attention.","b":true},{"text":" Every token sees every token. Best coherence, highest cost. Used by the models with the best physical consistency, and the reason they are expensive."}],[{"text":"Factorised attention.","b":true},{"text":" Attend across space within each frame, then across time at each spatial position, in alternating layers. Cost drops enormously. Objects that move diagonally across the frame are handled worse, because a token only attends along its own row through time."}],[{"text":"Windowed or sparse attention.","b":true},{"text":" Tokens attend within a local neighbourhood in space and time, with occasional global layers. Cheapest. Long-range agreement weakens, which shows up as slow drift over a clip rather than sudden breakage."}]]},{"t":"p","runs":[{"text":"You cannot usually pick. It is baked into the model. But you can read the symptom: if a model handles fast diagonal motion badly while holding static scenes well, you are probably looking at factorisation. If everything holds for three seconds and then slowly slides, suspect a window."}]},{"t":"h2","id":"attention-has-no-idea-what-an-object-is","runs":[{"text":"Attention has no idea what an object is"}]},{"t":"p","runs":[{"text":"This is the part that matters most, and it is routinely misunderstood."}]},{"t":"p","runs":[{"text":"There is no variable anywhere in the model holding “the red mug”. There are tokens whose values happen to correlate strongly across frames because they were trained on real footage where mugs persist. Persistence is a statistical habit, not a stored fact."}]},{"t":"p","runs":[{"text":"Three consequences follow directly."}]},{"t":"p","runs":[{"text":"Occlusion is dangerous.","b":true},{"text":" When something passes behind something else, the tokens that carried it stop existing for a while. When it reappears, nothing retrieves the original; the model re-samples something consistent with context. A person who walks behind a pillar can come back with a different shirt, and no setting prevents it."}]},{"t":"p","runs":[{"text":"Leaving frame is worse than occlusion.","b":true},{"text":" An object that exits the edge and returns has been gone from every token for many frames. Expect it to come back changed."}]},{"t":"p","runs":[{"text":"Count is not conserved.","b":true},{"text":" Three birds can become four. There is nothing counting."}]},{"t":"p","runs":[{"text":"The production response is to design around it. Do not write shots where a specific object leaves and returns. Cut instead: the cut is free and the re-entry is not."}]},{"t":"h2","id":"why-faces-break-first","runs":[{"text":"Why faces break first"}]},{"t":"p","runs":[{"text":"The model is no worse at faces than at chairs. You are far better at faces."}]},{"t":"p","runs":[{"text":"Human vision has dedicated machinery for facial identity, tuned to differences of a few per cent in the geometry between features. A chair whose back changes by thirty per cent still reads as the same chair. A face whose inter-ocular distance shifts by three per cent reads as a different person."}]},{"t":"p","runs":[{"text":"So the same drift rate produces a shrug on furniture and a rejected take on a face. That asymmetry is in your visual cortex, not in the model, and knowing it tells you where to spend: shorter shots on faces, longer shots on everything else."}]},{"t":"h2","id":"text-conditioning-is-a-small-part-of-this","runs":[{"text":"Text conditioning is a small part of this"}]},{"t":"p","runs":[{"text":"Your prompt enters through "},{"text":"cross-attention","b":true},{"text":": text tokens from an encoder such as T5 or a CLIP variant, attended to by the video tokens. There are a few hundred text tokens against a quarter of a million video tokens."}]},{"t":"p","runs":[{"text":"That ratio is worth sitting with. The prompt is a light steering influence over an enormous amount of structure that is mostly determined by the first frame and the model’s priors. It explains why adding a fourth clause to a prompt often changes nothing, and why the still frame overwhelms the text every time the two disagree."}]},{"t":"h2","id":"what-you-can-act-on","runs":[{"text":"What you can act on"}]},{"t":"ol","start":1,"items":[[{"text":"Shots where an object leaves and returns are not shots. Split them."}],[{"text":"Put the face in fewer, shorter shots than your instinct says."}],[{"text":"Prefer motion that stays roughly in the same region of frame: a push-in holds better than a fast lateral whip, on factorised models especially."}],[{"text":"When a model drifts slowly rather than breaking suddenly, do not fight it with prompts. Shorten the clip."}],[{"text":"Expect the prompt to lose any argument with the first frame."}]]},{"t":"rule"},{"t":"p","runs":[{"text":"Today: generate one shot where a person walks behind a column and out the other side. Watch what comes back. That single clip teaches the whole lesson better than the arithmetic does."}]}],"check":{"question":[{"text":"In a generated shot, a woman carrying a blue bag walks behind a pillar and emerges with a green bag. What does this reveal about the model?"}],"options":[{"text":[{"text":"Nothing stores the bag, so once its tokens are occluded the model re-samples something merely consistent with the surroundings."}],"correct":true,"tempting":null},{"text":[{"text":"The temporal attention window ended at the pillar, so the frames after it had no access to the frames before."}],"correct":false,"tempting":[{"text":"Assumes a window boundary landed exactly at the occlusion; the same failure occurs well inside a window."}]},{"text":[{"text":"The prompt described the bag only once, and prompt influence decays across the length of a clip so later frames were effectively unconditioned."}],"correct":false,"tempting":[{"text":"Imagines the text conditioning weakening over time; cross-attention applies equally to every frame in the block."}]},{"text":[{"text":"The autoencoder discarded the bag’s colour because it fell below the preserved spatial resolution, and the decoder chose a new one."}],"correct":false,"tempting":[{"text":"Misapplies the VAE limit, which affects fine detail rather than the colour of a large object."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"denoising-a-block-of-time","title":"The whole clip is made at once, not frame by frame","minutes":10},"next":{"slug":"how-models-learned-motion","title":"The prompt vocabulary was written by whoever captioned the training clips","minutes":9}}}