Why a chair stays a chair and a face does not

What attention is doing here

After the encoder, the latent is cut into patches: typically 2×2 in space, sometimes across a pair of latent frames as well. Each patch becomes a token. For our five-second 1080p clip: 30 latent frames, 240×135 latent pixels, patched 2×2, gives 30 × 120 × 68 ≈ 244,800 tokens.

In full 3D attention, every one of those tokens computes a relationship with every other one. That is 244,800² ≈ 60 billion pairs, per attention layer, per step. Modern kernels do not materialise that matrix, but the work still scales that way.

This is the expensive thing in the whole pipeline, and it is also the thing that makes video models work. A patch on frame 118 can look directly at a patch on frame 3. That is how the wall stays where it was.

Full, factorised, and windowed

Three arrangements, with a clear trade-off.

• Full 3D attention. Every token sees every token. Best coherence, highest cost. Used by the models with the best physical consistency, and the reason they are expensive.
• Factorised attention. Attend across space within each frame, then across time at each spatial position, in alternating layers. Cost drops enormously. Objects that move diagonally across the frame are handled worse, because a token only attends along its own row through time.
• Windowed or sparse attention. Tokens attend within a local neighbourhood in space and time, with occasional global layers. Cheapest. Long-range agreement weakens, which shows up as slow drift over a clip rather than sudden breakage.

You cannot usually pick. It is baked into the model. But you can read the symptom: if a model handles fast diagonal motion badly while holding static scenes well, you are probably looking at factorisation. If everything holds for three seconds and then slowly slides, suspect a window.

Attention has no idea what an object is

This is the part that matters most, and it is routinely misunderstood.

There is no variable anywhere in the model holding “the red mug”. There are tokens whose values happen to correlate strongly across frames because they were trained on real footage where mugs persist. Persistence is a statistical habit, not a stored fact.

Three consequences follow directly.

Occlusion is dangerous. When something passes behind something else, the tokens that carried it stop existing for a while. When it reappears, nothing retrieves the original; the model re-samples something consistent with context. A person who walks behind a pillar can come back with a different shirt, and no setting prevents it.

Leaving frame is worse than occlusion. An object that exits the edge and returns has been gone from every token for many frames. Expect it to come back changed.

Count is not conserved. Three birds can become four. There is nothing counting.

The production response is to design around it. Do not write shots where a specific object leaves and returns. Cut instead: the cut is free and the re-entry is not.

Why faces break first

The model is no worse at faces than at chairs. You are far better at faces.

Human vision has dedicated machinery for facial identity, tuned to differences of a few per cent in the geometry between features. A chair whose back changes by thirty per cent still reads as the same chair. A face whose inter-ocular distance shifts by three per cent reads as a different person.

So the same drift rate produces a shrug on furniture and a rejected take on a face. That asymmetry is in your visual cortex, not in the model, and knowing it tells you where to spend: shorter shots on faces, longer shots on everything else.

Text conditioning is a small part of this

Your prompt enters through cross-attention: text tokens from an encoder such as T5 or a CLIP variant, attended to by the video tokens. There are a few hundred text tokens against a quarter of a million video tokens.

That ratio is worth sitting with. The prompt is a light steering influence over an enormous amount of structure that is mostly determined by the first frame and the model’s priors. It explains why adding a fourth clause to a prompt often changes nothing, and why the still frame overwhelms the text every time the two disagree.

What you can act on

1. Shots where an object leaves and returns are not shots. Split them.
2. Put the face in fewer, shorter shots than your instinct says.
3. Prefer motion that stays roughly in the same region of frame: a push-in holds better than a fast lateral whip, on factorised models especially.
4. When a model drifts slowly rather than breaking suddenly, do not fight it with prompts. Shorten the clip.
5. Expect the prompt to lose any argument with the first frame.

Today: generate one shot where a person walks behind a column and out the other side. Watch what comes back. That single clip teaches the whole lesson better than the arithmetic does.