Depth, pose and flow as instructions

Beyond language and beyond sliders

The strongest motion control available is not a word or a number. It is another video.

Extract a structural signal from a reference clip (depth per frame, a pose skeleton per frame, or the optical flow between frames) and feed that alongside your prompt and first frame. The model then generates content that follows that structure. This is the video descendant of ControlNet, and it is how most of the striking transformation work you have seen was made.

Three signals, three different jobs.

Depth. A per-frame depth map from the reference. Gives you the reference’s camera move and spatial layout, while leaving surfaces free. Good for putting your scene through somebody else’s camera move.

Pose. A skeleton (joints and limbs) tracked through the reference. Gives you the reference’s performance. This is how a generated character does a specific dance, a specific gesture, a specific walk.

Optical flow or edges. Per-pixel motion, or per-frame outlines. The tightest control and the least freedom; output follows the reference almost exactly, which is what you want for a style transformation and not what you want for anything else.

Why this beats prompting for certain shots

A prompt cannot say “she raises her right hand to shoulder height over 1.4 seconds, pauses, then lowers it”. A pose sequence says exactly that, frame by frame, and the model obeys.

So any shot where the timing of an action matters is a candidate. Earlier in the course we said action timed to a beat is not currently something you can ask for. With pose conditioning, it is, because you are no longer asking; you are supplying.

The reference can be you, filmed on a phone in a kitchen. That is the thing worth sitting with: the hard part of this technique is free and requires no talent for prompting at all. It requires being willing to stand in front of a phone and do the movement.

What it imports along with the motion

The honest limitation, and it is not small.

A conditioning signal carries everything about the reference’s motion, including the parts you did not want:

• Proportions. A pose skeleton from a tall person drives a tall character. Applied to a character with different proportions, limbs stretch or the figure warps to reach the joint positions.
• Camera shake. Handheld wobble in the reference becomes wobble in the output. Stabilise the reference first, in Resolve or with FFmpeg’s vidstab, before extracting anything.
• Framing. Depth conditioning brings the reference’s composition. If the reference has the subject at the left edge, so will your output.
• Errors. A pose tracker that loses an arm behind a torso for four frames produces four frames where the generated arm does something impossible. Inspect the extracted signal as a video before you generate from it.

That last point is the one people skip. Render the conditioning signal and watch it. It takes thirty seconds and it is where most of these failures are visible.

Where the presets fit

The named camera-move libraries you find on consumer platforms (crash zoom, orbit, bullet time, drone flight) sit on top of this machinery or on top of tuned prompt-and-parameter sets. That is why they are repeatable in a way loose prose is not.

Their value is real: a vocabulary handed to you rather than learned by burning credits. Their limit is equally real: a preset is a look, not a shot. It applies the same move regardless of whether your frame supports it, and an orbit around a subject standing against a flat wall reveals that the wall is not a space, because it is not one.

The free path, which is complete here

This is one of the areas where open tooling is not a compromise.

• Depth: Depth Anything, free, as a Hugging Face Space or a ComfyUI node.
• Pose: DWPose and OpenPose, free, same routes. MediaPipe runs pose tracking on a phone in a browser.
• Flow: RAFT and similar, free, in ComfyUI.
• Stabilisation and extraction: FFmpeg, free, on any machine.
• The controllable video models themselves (the open-weights lines with control adapters) run in ComfyUI on the hardware discussed earlier.

The binding constraint is a GPU rather than a budget. Colab and Kaggle sessions close some of that gap; Hugging Face Spaces demonstrating specific control pipelines close more.

When not to use it

If the motion is simple and generic (a slow push-in, a figure standing in a breeze) conditioning is a great deal of machinery for something a sentence already does. Reach for it when the motion is specific, when timing matters, or when you need the same movement repeated across several shots.

Today: film ten seconds of yourself performing one gesture on a phone. Extract a pose sequence in a free Space and watch the skeleton. That file is a piece of direction no prompt could have written.