{"course":{"slug":"ai-video","title":"Video with AI","url":"https://addaly.com/learn/ai-video","level":"intermediate","track":"create"},"lesson":{"slug":"conditioned-motion-controls","title":"Depth, pose and flow as instructions","url":"https://addaly.com/learn/ai-video/conditioned-motion-controls","position":26,"of":66,"minutes":10,"answer":"A conditioning signal supplies the model with a per-frame structural target (depth, pose skeleton or optical flow) which is why it produces motion a prompt cannot describe.","keyIdea":"A conditioning signal supplies the model with a per-frame structural target (depth, pose skeleton or optical flow) which is why it produces motion a prompt cannot describe, and why it also imports every flaw of the source video."},"body":{"format":"markdown","text":"## Beyond language and beyond sliders\n\nThe strongest motion control available is not a word or a number. It is another video.\n\nExtract a structural signal from a reference clip (depth per frame, a pose skeleton per frame, or the optical flow between frames) and feed that alongside your prompt and first frame. The model then generates content that follows that structure. This is the video descendant of ControlNet, and it is how most of the striking transformation work you have seen was made.\n\nThree signals, three different jobs.\n\n**Depth.** A per-frame depth map from the reference. Gives you the reference's camera move and spatial layout, while leaving surfaces free. Good for putting your scene through somebody else's camera move.\n\n**Pose.** A skeleton (joints and limbs) tracked through the reference. Gives you the reference's performance. This is how a generated character does a specific dance, a specific gesture, a specific walk.\n\n**Optical flow or edges.** Per-pixel motion, or per-frame outlines. The tightest control and the least freedom; output follows the reference almost exactly, which is what you want for a style transformation and not what you want for anything else.\n\n## Why this beats prompting for certain shots\n\nA prompt cannot say \"she raises her right hand to shoulder height over 1.4 seconds, pauses, then lowers it\". A pose sequence says exactly that, frame by frame, and the model obeys.\n\nSo any shot where the *timing* of an action matters is a candidate. Earlier in the course we said action timed to a beat is not currently something you can ask for. With pose conditioning, it is, because you are no longer asking; you are supplying.\n\nThe reference can be you, filmed on a phone in a kitchen. That is the thing worth sitting with: the hard part of this technique is free and requires no talent for prompting at all. It requires being willing to stand in front of a phone and do the movement.\n\n## What it imports along with the motion\n\nThe honest limitation, and it is not small.\n\nA conditioning signal carries everything about the reference's motion, including the parts you did not want:\n\n- **Proportions.** A pose skeleton from a tall person drives a tall character. Applied to a character with different proportions, limbs stretch or the figure warps to reach the joint positions.\n- **Camera shake.** Handheld wobble in the reference becomes wobble in the output. Stabilise the reference first, in Resolve or with FFmpeg's `vidstab`, before extracting anything.\n- **Framing.** Depth conditioning brings the reference's composition. If the reference has the subject at the left edge, so will your output.\n- **Errors.** A pose tracker that loses an arm behind a torso for four frames produces four frames where the generated arm does something impossible. Inspect the extracted signal as a video before you generate from it.\n\nThat last point is the one people skip. **Render the conditioning signal and watch it.** It takes thirty seconds and it is where most of these failures are visible.\n\n## Where the presets fit\n\nThe named camera-move libraries you find on consumer platforms (crash zoom, orbit, bullet time, drone flight) sit on top of this machinery or on top of tuned prompt-and-parameter sets. That is why they are repeatable in a way loose prose is not.\n\nTheir value is real: a vocabulary handed to you rather than learned by burning credits. Their limit is equally real: a preset is a look, not a shot. It applies the same move regardless of whether your frame supports it, and an orbit around a subject standing against a flat wall reveals that the wall is not a space, because it is not one.\n\n## The free path, which is complete here\n\nThis is one of the areas where open tooling is not a compromise.\n\n- **Depth**: Depth Anything, free, as a Hugging Face Space or a ComfyUI node.\n- **Pose**: DWPose and OpenPose, free, same routes. MediaPipe runs pose tracking on a phone in a browser.\n- **Flow**: RAFT and similar, free, in ComfyUI.\n- **Stabilisation and extraction**: FFmpeg, free, on any machine.\n- **The controllable video models themselves** (the open-weights lines with control adapters) run in ComfyUI on the hardware discussed earlier.\n\nThe binding constraint is a GPU rather than a budget. Colab and Kaggle sessions close some of that gap; Hugging Face Spaces demonstrating specific control pipelines close more.\n\n## When not to use it\n\nIf the motion is simple and generic (a slow push-in, a figure standing in a breeze) conditioning is a great deal of machinery for something a sentence already does. Reach for it when the motion is specific, when timing matters, or when you need the same movement repeated across several shots.\n\n---\n\nToday: film ten seconds of yourself performing one gesture on a phone. Extract a pose sequence in a free Space and watch the skeleton. That file is a piece of direction no prompt could have written."},"check":{"question":"A character generated from a pose sequence has arms that stretch unnaturally during a reaching movement. The pose extraction looked clean. What is the most likely cause?","options":[{"text":"Optical flow was used instead of a skeleton, so per-pixel motion was applied to a figure with different geometry.","correct":false,"tempting":"Contradicts the premise, which states a pose sequence was used; flow would distort the whole frame, not the limbs alone."},{"text":"The reference performer's limb proportions differ from the generated character's, so the model deforms the figure to reach the joint positions.","correct":true},{"text":"The pose tracker sampled at a lower frame rate than the generation, so intermediate joint positions were interpolated linearly through impossible configurations.","correct":false,"tempting":"Names a real sampling issue that produces jerky timing rather than sustained limb stretching."},{"text":"Pose conditioning overrides the first frame, so the character's build was re-sampled mid-clip to match the skeleton rather than the supplied still.","correct":false,"tempting":"Overstates the conditioning's authority; the still still governs appearance, which is precisely why the conflict resolves as deformation."}]},"prev":{"slug":"motion-strength-and-brushes","title":"Taking motion out of language","url":"https://addaly.com/learn/ai-video/motion-strength-and-brushes"},"next":{"slug":"negative-prompts-and-their-limits","title":"What a negative prompt can and cannot remove","url":"https://addaly.com/learn/ai-video/negative-prompts-and-their-limits"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Depth, pose and flow as instructions', in Video with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-video/conditioned-motion-controls (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Depth, pose and flow as instructions","courseTitle":"Video with AI","module":{"index":3,"slug":"directing-the-shot","title":"Directing, not describing"},"answer":null,"keyIdea":"A conditioning signal supplies the model with a per-frame structural target (depth, pose skeleton or optical flow) which is why it produces motion a prompt cannot describe, and why it also imports every flaw of the source video.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"beyond-language-and-beyond-sliders","runs":[{"text":"Beyond language and beyond sliders"}]},{"t":"p","runs":[{"text":"The strongest motion control available is not a word or a number. It is another video."}]},{"t":"p","runs":[{"text":"Extract a structural signal from a reference clip (depth per frame, a pose skeleton per frame, or the optical flow between frames) and feed that alongside your prompt and first frame. The model then generates content that follows that structure. This is the video descendant of ControlNet, and it is how most of the striking transformation work you have seen was made."}]},{"t":"p","runs":[{"text":"Three signals, three different jobs."}]},{"t":"p","runs":[{"text":"Depth.","b":true},{"text":" A per-frame depth map from the reference. Gives you the reference’s camera move and spatial layout, while leaving surfaces free. Good for putting your scene through somebody else’s camera move."}]},{"t":"p","runs":[{"text":"Pose.","b":true},{"text":" A skeleton (joints and limbs) tracked through the reference. Gives you the reference’s performance. This is how a generated character does a specific dance, a specific gesture, a specific walk."}]},{"t":"p","runs":[{"text":"Optical flow or edges.","b":true},{"text":" Per-pixel motion, or per-frame outlines. The tightest control and the least freedom; output follows the reference almost exactly, which is what you want for a style transformation and not what you want for anything else."}]},{"t":"h2","id":"why-this-beats-prompting-for-certain-shots","runs":[{"text":"Why this beats prompting for certain shots"}]},{"t":"p","runs":[{"text":"A prompt cannot say “she raises her right hand to shoulder height over 1.4 seconds, pauses, then lowers it”. A pose sequence says exactly that, frame by frame, and the model obeys."}]},{"t":"p","runs":[{"text":"So any shot where the "},{"text":"timing","i":true},{"text":" of an action matters is a candidate. Earlier in the course we said action timed to a beat is not currently something you can ask for. With pose conditioning, it is, because you are no longer asking; you are supplying."}]},{"t":"p","runs":[{"text":"The reference can be you, filmed on a phone in a kitchen. That is the thing worth sitting with: the hard part of this technique is free and requires no talent for prompting at all. It requires being willing to stand in front of a phone and do the movement."}]},{"t":"h2","id":"what-it-imports-along-with-the-motion","runs":[{"text":"What it imports along with the motion"}]},{"t":"p","runs":[{"text":"The honest limitation, and it is not small."}]},{"t":"p","runs":[{"text":"A conditioning signal carries everything about the reference’s motion, including the parts you did not want:"}]},{"t":"ul","items":[[{"text":"Proportions.","b":true},{"text":" A pose skeleton from a tall person drives a tall character. Applied to a character with different proportions, limbs stretch or the figure warps to reach the joint positions."}],[{"text":"Camera shake.","b":true},{"text":" Handheld wobble in the reference becomes wobble in the output. Stabilise the reference first, in Resolve or with FFmpeg’s "},{"text":"vidstab","code":true},{"text":", before extracting anything."}],[{"text":"Framing.","b":true},{"text":" Depth conditioning brings the reference’s composition. If the reference has the subject at the left edge, so will your output."}],[{"text":"Errors.","b":true},{"text":" A pose tracker that loses an arm behind a torso for four frames produces four frames where the generated arm does something impossible. Inspect the extracted signal as a video before you generate from it."}]]},{"t":"p","runs":[{"text":"That last point is the one people skip. "},{"text":"Render the conditioning signal and watch it.","b":true},{"text":" It takes thirty seconds and it is where most of these failures are visible."}]},{"t":"h2","id":"where-the-presets-fit","runs":[{"text":"Where the presets fit"}]},{"t":"p","runs":[{"text":"The named camera-move libraries you find on consumer platforms (crash zoom, orbit, bullet time, drone flight) sit on top of this machinery or on top of tuned prompt-and-parameter sets. That is why they are repeatable in a way loose prose is not."}]},{"t":"p","runs":[{"text":"Their value is real: a vocabulary handed to you rather than learned by burning credits. Their limit is equally real: a preset is a look, not a shot. It applies the same move regardless of whether your frame supports it, and an orbit around a subject standing against a flat wall reveals that the wall is not a space, because it is not one."}]},{"t":"h2","id":"the-free-path-which-is-complete-here","runs":[{"text":"The free path, which is complete here"}],"role":"free"},{"t":"p","runs":[{"text":"This is one of the areas where open tooling is not a compromise."}]},{"t":"ul","items":[[{"text":"Depth","b":true},{"text":": Depth Anything, free, as a Hugging Face Space or a ComfyUI node."}],[{"text":"Pose","b":true},{"text":": DWPose and OpenPose, free, same routes. MediaPipe runs pose tracking on a phone in a browser."}],[{"text":"Flow","b":true},{"text":": RAFT and similar, free, in ComfyUI."}],[{"text":"Stabilisation and extraction","b":true},{"text":": FFmpeg, free, on any machine."}],[{"text":"The controllable video models themselves","b":true},{"text":" (the open-weights lines with control adapters) run in ComfyUI on the hardware discussed earlier."}]]},{"t":"p","runs":[{"text":"The binding constraint is a GPU rather than a budget. Colab and Kaggle sessions close some of that gap; Hugging Face Spaces demonstrating specific control pipelines close more."}]},{"t":"h2","id":"when-not-to-use-it","runs":[{"text":"When not to use it"}]},{"t":"p","runs":[{"text":"If the motion is simple and generic (a slow push-in, a figure standing in a breeze) conditioning is a great deal of machinery for something a sentence already does. Reach for it when the motion is specific, when timing matters, or when you need the same movement repeated across several shots."}]},{"t":"rule"},{"t":"p","runs":[{"text":"Today: film ten seconds of yourself performing one gesture on a phone. Extract a pose sequence in a free Space and watch the skeleton. That file is a piece of direction no prompt could have written."}]}],"check":{"question":[{"text":"A character generated from a pose sequence has arms that stretch unnaturally during a reaching movement. The pose extraction looked clean. What is the most likely cause?"}],"options":[{"text":[{"text":"Optical flow was used instead of a skeleton, so per-pixel motion was applied to a figure with different geometry."}],"correct":false,"tempting":[{"text":"Contradicts the premise, which states a pose sequence was used; flow would distort the whole frame, not the limbs alone."}]},{"text":[{"text":"The reference performer’s limb proportions differ from the generated character’s, so the model deforms the figure to reach the joint positions."}],"correct":true,"tempting":null},{"text":[{"text":"The pose tracker sampled at a lower frame rate than the generation, so intermediate joint positions were interpolated linearly through impossible configurations."}],"correct":false,"tempting":[{"text":"Names a real sampling issue that produces jerky timing rather than sustained limb stretching."}]},{"text":[{"text":"Pose conditioning overrides the first frame, so the character’s build was re-sampled mid-clip to match the skeleton rather than the supplied still."}],"correct":false,"tempting":[{"text":"Overstates the conditioning’s authority; the still still governs appearance, which is precisely why the conflict resolves as deformation."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"motion-strength-and-brushes","title":"Taking motion out of language","minutes":9},"next":{"slug":"negative-prompts-and-their-limits","title":"What a negative prompt can and cannot remove","minutes":8}}}