{"course":{"slug":"ai-video","title":"Video with AI","url":"https://addaly.com/learn/ai-video","level":"intermediate","track":"create"},"lesson":{"slug":"lip-sync-mechanics","title":"What the model is actually reshaping","url":"https://addaly.com/learn/ai-video/lip-sync-mechanics","position":41,"of":66,"minutes":9,"answer":"Audio-driven lip-sync maps sound to a small set of visually distinct mouth shapes and inpaints the mouth region frame by frame, which is why it works at medium shots and falls apart on plosives.","keyIdea":"Audio-driven lip-sync maps sound to a small set of visually distinct mouth shapes and inpaints the mouth region frame by frame, which is why it works at medium shots and falls apart on plosives, profiles and close-ups."},"body":{"format":"markdown","text":"## Phonemes and visemes\n\nSpeech contains roughly forty distinguishable sounds in English. The mouth makes far fewer distinguishable shapes: around a dozen. Those shapes are called visemes, and several phonemes share one: **p**, **b** and **m** look identical from outside, as do **f** and **v**, and as do **t**, **d** and **n**.\n\nA lip-sync model learns the mapping from audio features to those shapes, then reconstructs the mouth region of each frame to match. The Wav2Lip family and its successors work essentially this way: detect the face, crop the lower half, generate a mouth that matches the audio, blend it back.\n\n> Figure: What a lip-sync model actually reshapes. Steps: 1. Forty sounds, a dozen shapes (English has roughly forty distinguishable sounds and the mouth makes about a dozen distinguishable shapes. P, B and M look identical from outside, and so do F and V, and T, D and N.); 2. Detect the face, crop the mouth (The original Wav2Lip worked on a 96 by 96 pixel crop of the lower face. Later models are larger and all of them are far smaller than your frame.); 3. Generate a mouth per frame (Reconstructed at the crop's resolution to match the audio, with no knowledge of the cheeks, the jaw angle, the eyebrows or the head. That is why the mouth speaks and the face does not.); 4. Scale it back and blend (On a 1080p face this leaves a visibly softer rectangle around the mouth, which is the clearest tell of a lip-synced clip once you know to look for it.). Everything these tools do well and badly follows from this description. A medium shot with clean, slightly slowed speech works. A close-up, a profile past about thirty degrees, or a run of plosives does not, and no setting changes that.\n\nEverything about how these tools behave follows from that description.\n\n## Why it is only the mouth\n\nThe model changes a small rectangle and leaves the rest of the frame alone. That is why the technique is cheap, why it works on any footage including real footage, and why it produces a specific uncanny quality: the mouth is speaking and the face is not.\n\nReal speech involves the cheeks, the jaw angle, the eyes narrowing on emphasis, the eyebrows, the head moving slightly on stressed syllables. A mouth-only sync gives you none of that, and viewers register the absence as \"dubbed\" without being able to say why.\n\nNewer hosted products (Hedra, HeyGen, Synthesia and the sync features inside larger platforms) animate more: head motion, blinks, some eye direction. Better, and still a generated performance rather than a captured one.\n\n## The four places it breaks\n\n**Plosives.** For **b**, **p** and **m** the lips must close completely. Many models under-close them, and the result reads exactly like badly dubbed foreign footage. Most viewers cannot name what is wrong; all of them notice. Slowing the delivery slightly improves it, because the closure gets more frames.\n\n**Profiles.** The mouth has to be visible and roughly frontal. Past about thirty degrees off-axis the model is reconstructing geometry it cannot see, and the jaw slides. Keep any speaking shot near frontal.\n\n**Close-ups.** The interior of the mouth (teeth, tongue, the dark space behind) is where these models are weakest, and a close-up shows all of it. Stay at a medium shot and most of the problem disappears. This is also why so much AI-presenter content is framed at chest height: it is not a style choice.\n\n**Long takes.** Drift, as everywhere. The reconstructed mouth region slowly diverges from the surrounding skin in colour and texture. Cut every few seconds.\n\n## Resolution, and the reason for the soft square\n\nMany open lip-sync models operate on a small crop: 96 by 96 pixels in the original Wav2Lip, larger in later ones. That crop is generated at low resolution and scaled back up into the frame.\n\nOn a 1080p face, that leaves a visibly softer rectangle around the mouth. It is the clearest tell of a lip-synced clip and you can see it on a great deal of published content once you know to look.\n\nMitigations: work at a shot size where the face is small enough that the crop covers it adequately; run a face restoration pass afterwards at low strength, accepting the flicker trade-off from the previous module; or use a higher-resolution model and pay for it.\n\n## Getting a better result from the audio side\n\nHalf of lip-sync quality is the audio you feed it, and this part is free:\n\n- **Clean speech, no music, no background.** The model is extracting phonetic features; anything else is noise to it. Sync against the dry voice track and add music afterwards in the edit.\n- **Consistent level.** Compress and normalise the voice first. Audacity does both in about a minute and runs on very old machines.\n- **Slightly slower delivery.** More frames per phoneme, better closures, fewer smears.\n- **Trim leading and trailing silence.** Some models produce odd mouth movement over silence.\n\n## The free path\n\nWav2Lip, LatentSync and their current successors are open and free, available as Hugging Face Spaces for browser use and as local installs. Audacity handles every audio preparation step above. FFmpeg splits and rejoins audio and video.\n\nThe paid products are better, particularly on head motion and resolution. They are not a prerequisite for learning any of this, and a medium shot with clean audio from a free model beats a close-up from a paid one.\n\n---\n\nToday: sync thirty seconds of clean speech to a medium shot and to a close-up of the same character, and compare where the eye goes in each."},"check":{"question":"A lip-synced clip reads as badly dubbed even though the timing is correct. The character is saying a line with several b and p sounds. What is the most likely cause?","options":[{"text":"The model under-closes the lips on plosives, which require full closure, and incomplete closure is the classic signature of poor dubbing.","correct":true},{"text":"The audio was normalised too aggressively, so the transients that mark plosives were flattened and the model had no cue to close the lips on.","correct":false,"tempting":"Names a real audio processing risk; the closure failure occurs on well-levelled audio too, because it is a reconstruction limit."},{"text":"The viseme mapping is ambiguous for b, p and m, so the model selected the wrong mouth shape for those phonemes.","correct":false,"tempting":"Those three phonemes share one viseme, so ambiguity between them cannot produce a wrong shape: the shape is the same."},{"text":"The frame rate is too low to represent the brief closure, and interpolating the clip to a higher frame rate before syncing would restore the missing frames of contact.","correct":false,"tempting":"Interpolation invents frames from existing ones and cannot add a closure the source never contained."}]},"prev":{"slug":"faces-voices-consent","title":"Lip-sync is easy; the permission is not","url":"https://addaly.com/learn/ai-video/faces-voices-consent"},"next":{"slug":"performance-transfer","title":"Acting is timing, and timing has to come from somewhere","url":"https://addaly.com/learn/ai-video/performance-transfer"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What the model is actually reshaping', in Video with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-video/lip-sync-mechanics (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What the model is actually reshaping","courseTitle":"Video with AI","module":{"index":5,"slug":"sound-and-speech","title":"The half of video that is not picture"},"answer":null,"keyIdea":"Audio-driven lip-sync maps sound to a small set of visually distinct mouth shapes and inpaints the mouth region frame by frame, which is why it works at medium shots and falls apart on plosives, profiles and close-ups.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Forty sounds, a dozen shapes","detail":"English has roughly forty distinguishable sounds and the mouth makes about a dozen distinguishable shapes. P, B and M look identical from outside, and so do F and V, and T, D and N."},{"label":"Detect the face, crop the mouth","detail":"The original Wav2Lip worked on a 96 by 96 pixel crop of the lower face. Later models are larger and all of them are far smaller than your frame."},{"label":"Generate a mouth per frame","detail":"Reconstructed at the crop’s resolution to match the audio, with no knowledge of the cheeks, the jaw angle, the eyebrows or the head. That is why the mouth speaks and the face does not."},{"label":"Scale it back and blend","detail":"On a 1080p face this leaves a visibly softer rectangle around the mouth, which is the clearest tell of a lip-synced clip once you know to look for it."}],"title":"What a lip-sync model actually reshapes","caption":"Everything these tools do well and badly follows from this description. A medium shot with clean, slightly slowed speech works. A close-up, a profile past about thirty degrees, or a run of plosives does not, and no setting changes that."},"alt":"Figure: What a lip-sync model actually reshapes. Steps: 1. Forty sounds, a dozen shapes (English has roughly forty distinguishable sounds and the mouth makes about a dozen distinguishable shapes. P, B and M look identical from outside, and so do F and V, and T, D and N.); 2. Detect the face, crop the mouth (The original Wav2Lip worked on a 96 by 96 pixel crop of the lower face. Later models are larger and all of them are far smaller than your frame.); 3. Generate a mouth per frame (Reconstructed at the crop’s resolution to match the audio, with no knowledge of the cheeks, the jaw angle, the eyebrows or the head. That is why the mouth speaks and the face does not.); 4. Scale it back and blend (On a 1080p face this leaves a visibly softer rectangle around the mouth, which is the clearest tell of a lip-synced clip once you know to look for it.). Everything these tools do well and badly follows from this description. A medium shot with clean, slightly slowed speech works. A close-up, a profile past about thirty degrees, or a run of plosives does not, and no setting changes that."}},"blocks":[{"t":"h2","id":"phonemes-and-visemes","runs":[{"text":"Phonemes and visemes"}]},{"t":"p","runs":[{"text":"Speech contains roughly forty distinguishable sounds in English. The mouth makes far fewer distinguishable shapes: around a dozen. Those shapes are called visemes, and several phonemes share one: "},{"text":"p","b":true},{"text":", "},{"text":"b","b":true},{"text":" and "},{"text":"m","b":true},{"text":" look identical from outside, as do "},{"text":"f","b":true},{"text":" and "},{"text":"v","b":true},{"text":", and as do "},{"text":"t","b":true},{"text":", "},{"text":"d","b":true},{"text":" and "},{"text":"n","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"A lip-sync model learns the mapping from audio features to those shapes, then reconstructs the mouth region of each frame to match. The Wav2Lip family and its successors work essentially this way: detect the face, crop the lower half, generate a mouth that matches the audio, blend it back."}]},{"t":"p","runs":[{"text":"Everything about how these tools behave follows from that description."}]},{"t":"h2","id":"why-it-is-only-the-mouth","runs":[{"text":"Why it is only the mouth"}]},{"t":"p","runs":[{"text":"The model changes a small rectangle and leaves the rest of the frame alone. That is why the technique is cheap, why it works on any footage including real footage, and why it produces a specific uncanny quality: the mouth is speaking and the face is not."}]},{"t":"p","runs":[{"text":"Real speech involves the cheeks, the jaw angle, the eyes narrowing on emphasis, the eyebrows, the head moving slightly on stressed syllables. A mouth-only sync gives you none of that, and viewers register the absence as “dubbed” without being able to say why."}]},{"t":"p","runs":[{"text":"Newer hosted products (Hedra, HeyGen, Synthesia and the sync features inside larger platforms) animate more: head motion, blinks, some eye direction. Better, and still a generated performance rather than a captured one."}]},{"t":"h2","id":"the-four-places-it-breaks","runs":[{"text":"The four places it breaks"}]},{"t":"p","runs":[{"text":"Plosives.","b":true},{"text":" For "},{"text":"b","b":true},{"text":", "},{"text":"p","b":true},{"text":" and "},{"text":"m","b":true},{"text":" the lips must close completely. Many models under-close them, and the result reads exactly like badly dubbed foreign footage. Most viewers cannot name what is wrong; all of them notice. Slowing the delivery slightly improves it, because the closure gets more frames."}]},{"t":"p","runs":[{"text":"Profiles.","b":true},{"text":" The mouth has to be visible and roughly frontal. Past about thirty degrees off-axis the model is reconstructing geometry it cannot see, and the jaw slides. Keep any speaking shot near frontal."}]},{"t":"p","runs":[{"text":"Close-ups.","b":true},{"text":" The interior of the mouth (teeth, tongue, the dark space behind) is where these models are weakest, and a close-up shows all of it. Stay at a medium shot and most of the problem disappears. This is also why so much AI-presenter content is framed at chest height: it is not a style choice."}]},{"t":"p","runs":[{"text":"Long takes.","b":true},{"text":" Drift, as everywhere. The reconstructed mouth region slowly diverges from the surrounding skin in colour and texture. Cut every few seconds."}]},{"t":"h2","id":"resolution-and-the-reason-for-the-soft-square","runs":[{"text":"Resolution, and the reason for the soft square"}]},{"t":"p","runs":[{"text":"Many open lip-sync models operate on a small crop: 96 by 96 pixels in the original Wav2Lip, larger in later ones. That crop is generated at low resolution and scaled back up into the frame."}]},{"t":"p","runs":[{"text":"On a 1080p face, that leaves a visibly softer rectangle around the mouth. It is the clearest tell of a lip-synced clip and you can see it on a great deal of published content once you know to look."}]},{"t":"p","runs":[{"text":"Mitigations: work at a shot size where the face is small enough that the crop covers it adequately; run a face restoration pass afterwards at low strength, accepting the flicker trade-off from the previous module; or use a higher-resolution model and pay for it."}]},{"t":"h2","id":"getting-a-better-result-from-the-audio-side","runs":[{"text":"Getting a better result from the audio side"}]},{"t":"p","runs":[{"text":"Half of lip-sync quality is the audio you feed it, and this part is free:"}]},{"t":"ul","items":[[{"text":"Clean speech, no music, no background.","b":true},{"text":" The model is extracting phonetic features; anything else is noise to it. Sync against the dry voice track and add music afterwards in the edit."}],[{"text":"Consistent level.","b":true},{"text":" Compress and normalise the voice first. Audacity does both in about a minute and runs on very old machines."}],[{"text":"Slightly slower delivery.","b":true},{"text":" More frames per phoneme, better closures, fewer smears."}],[{"text":"Trim leading and trailing silence.","b":true},{"text":" Some models produce odd mouth movement over silence."}]]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"Wav2Lip, LatentSync and their current successors are open and free, available as Hugging Face Spaces for browser use and as local installs. Audacity handles every audio preparation step above. FFmpeg splits and rejoins audio and video."}]},{"t":"p","runs":[{"text":"The paid products are better, particularly on head motion and resolution. They are not a prerequisite for learning any of this, and a medium shot with clean audio from a free model beats a close-up from a paid one."}]},{"t":"rule"},{"t":"p","runs":[{"text":"Today: sync thirty seconds of clean speech to a medium shot and to a close-up of the same character, and compare where the eye goes in each."}]}],"check":{"question":[{"text":"A lip-synced clip reads as badly dubbed even though the timing is correct. The character is saying a line with several b and p sounds. What is the most likely cause?"}],"options":[{"text":[{"text":"The model under-closes the lips on plosives, which require full closure, and incomplete closure is the classic signature of poor dubbing."}],"correct":true,"tempting":null},{"text":[{"text":"The audio was normalised too aggressively, so the transients that mark plosives were flattened and the model had no cue to close the lips on."}],"correct":false,"tempting":[{"text":"Names a real audio processing risk; the closure failure occurs on well-levelled audio too, because it is a reconstruction limit."}]},{"text":[{"text":"The viseme mapping is ambiguous for b, p and m, so the model selected the wrong mouth shape for those phonemes."}],"correct":false,"tempting":[{"text":"Those three phonemes share one viseme, so ambiguity between them cannot produce a wrong shape: the shape is the same."}]},{"text":[{"text":"The frame rate is too low to represent the brief closure, and interpolating the clip to a higher frame rate before syncing would restore the missing frames of contact."}],"correct":false,"tempting":[{"text":"Interpolation invents frames from existing ones and cannot add a closure the source never contained."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"faces-voices-consent","title":"Lip-sync is easy; the permission is not","minutes":10},"next":{"slug":"performance-transfer","title":"Acting is timing, and timing has to come from somewhere","minutes":9}}}