What the model is actually reshaping

Phonemes and visemes

Speech contains roughly forty distinguishable sounds in English. The mouth makes far fewer distinguishable shapes: around a dozen. Those shapes are called visemes, and several phonemes share one: p, b and m look identical from outside, as do f and v, and as do t, d and n.

A lip-sync model learns the mapping from audio features to those shapes, then reconstructs the mouth region of each frame to match. The Wav2Lip family and its successors work essentially this way: detect the face, crop the lower half, generate a mouth that matches the audio, blend it back.

Everything about how these tools behave follows from that description.

Why it is only the mouth

The model changes a small rectangle and leaves the rest of the frame alone. That is why the technique is cheap, why it works on any footage including real footage, and why it produces a specific uncanny quality: the mouth is speaking and the face is not.

Real speech involves the cheeks, the jaw angle, the eyes narrowing on emphasis, the eyebrows, the head moving slightly on stressed syllables. A mouth-only sync gives you none of that, and viewers register the absence as “dubbed” without being able to say why.

Newer hosted products (Hedra, HeyGen, Synthesia and the sync features inside larger platforms) animate more: head motion, blinks, some eye direction. Better, and still a generated performance rather than a captured one.

The four places it breaks

Plosives. For b, p and m the lips must close completely. Many models under-close them, and the result reads exactly like badly dubbed foreign footage. Most viewers cannot name what is wrong; all of them notice. Slowing the delivery slightly improves it, because the closure gets more frames.

Profiles. The mouth has to be visible and roughly frontal. Past about thirty degrees off-axis the model is reconstructing geometry it cannot see, and the jaw slides. Keep any speaking shot near frontal.

Close-ups. The interior of the mouth (teeth, tongue, the dark space behind) is where these models are weakest, and a close-up shows all of it. Stay at a medium shot and most of the problem disappears. This is also why so much AI-presenter content is framed at chest height: it is not a style choice.

Long takes. Drift, as everywhere. The reconstructed mouth region slowly diverges from the surrounding skin in colour and texture. Cut every few seconds.

Resolution, and the reason for the soft square

Many open lip-sync models operate on a small crop: 96 by 96 pixels in the original Wav2Lip, larger in later ones. That crop is generated at low resolution and scaled back up into the frame.

On a 1080p face, that leaves a visibly softer rectangle around the mouth. It is the clearest tell of a lip-synced clip and you can see it on a great deal of published content once you know to look.

Mitigations: work at a shot size where the face is small enough that the crop covers it adequately; run a face restoration pass afterwards at low strength, accepting the flicker trade-off from the previous module; or use a higher-resolution model and pay for it.

Getting a better result from the audio side

Half of lip-sync quality is the audio you feed it, and this part is free:

• Clean speech, no music, no background. The model is extracting phonetic features; anything else is noise to it. Sync against the dry voice track and add music afterwards in the edit.
• Consistent level. Compress and normalise the voice first. Audacity does both in about a minute and runs on very old machines.
• Slightly slower delivery. More frames per phoneme, better closures, fewer smears.
• Trim leading and trailing silence. Some models produce odd mouth movement over silence.

The free path

Wav2Lip, LatentSync and their current successors are open and free, available as Hugging Face Spaces for browser use and as local installs. Audacity handles every audio preparation step above. FFmpeg splits and rejoins audio and video.

The paid products are better, particularly on head motion and resolution. They are not a prerequisite for learning any of this, and a medium shot with clean audio from a free model beats a close-up from a paid one.

Today: sync thirty seconds of clean speech to a medium shot and to a close-up of the same character, and compare where the eye goes in each.