Acting is timing, and timing has to come from somewhere

What it does differently

Instead of deriving mouth shapes from audio, performance transfer derives the whole facial performance from a video of a person acting. You record yourself on a webcam delivering the line; the model maps your expression, head movement, blinks and eye direction onto a generated character.

Runway’s Act-One is the best-known example of the class, and similar features exist elsewhere under other names.

The difference in output quality is not subtle, and the reason is not technical sophistication. It is that delivery is most of acting, and no phoneme model invents delivery.

What delivery actually consists of

Watch somebody say a sentence they mean. The mouth is doing the least interesting part.

• Pauses. Where somebody stops, and for how long. A quarter-second before a word changes what the word means.
• Emphasis. Which syllable gets weight, and the tiny head movement that accompanies it.
• Eyebrows. Raised on a question, drawn together on difficulty, lifted on surprise. These carry as much meaning as the words.
• Blinks. People blink at clause boundaries and at moments of thought. Regular mechanical blinking reads as wrong.
• Eye direction. Looking away while thinking, back on the point being made.
• Micro-stillness. Genuine listening is very still. Generated faces tend to fidget.

A lip-sync model produces none of these. A performance transfer produces all of them, because a person did them.

Which means the webcam is the important part

This is the unintuitive conclusion and the practically useful one. The quality of your talking-head shot depends mostly on how well the person in front of the webcam performs, and only secondarily on the model.

So:

• Perform the line properly. Read it aloud several times first. Know what it means. Decide where the pauses are.
• Sit still and frontal, well lit, camera at eye level. The model tracks your face, so give it a face to track: even light, no strong shadow across the features, no backlight.
• Do several takes and pick the one where the delivery is right. This is free: it is a webcam recording, not a generation.
• Overplay very slightly. Expression loses a little amplitude in transfer, as it does in animation generally.

If you cannot act, find somebody who can. A friend who does theatre will produce a better result than any amount of prompt engineering, and they will do it in ten minutes.

The limits

• Proportions transfer, and they should not. A driver with a long face drives the target toward a long face. Pick a driver whose head shape is roughly compatible, or expect the character to shift toward you.
• Head rotation range is limited. Large turns lose tracking. Keep within perhaps thirty degrees of frontal.
• Occlusion breaks it. Hands near the face, hair falling across it, glasses catching the light.
• It is still a generated face. Skin behaves slightly wrongly in motion, and long takes drift like everything else. Cut.
• Duration limits apply exactly as they do to any generation.

Where it fits in a workflow

A talking-head sequence that works usually looks like this:

1. Record the audio properly, as a separate clean track: a phone in a quiet room beats a webcam microphone every time.
2. Record the performance on the webcam, listening to or delivering the same line.
3. Transfer the performance onto the character.
4. Replace the webcam audio with the clean recording in the edit, aligned to the performance.
5. Cut every few seconds, using cutaways and reaction shots so no single generation has to run long.

Step four is the one people skip, and it is the difference between something that sounds like a video call and something that sounds produced.

The consent question is unchanged

Performance transfer onto a generated character is your own face driving a fictional one, which is straightforward. Performance transfer onto a real person’s likeness is synthesising that person saying something they did not say, which requires their written, specific permission and a visible label on the output.

The technique does not change the rule and the quality of the result does not change it either. If anything, a better result raises the stakes, because a convincing synthetic performance is exactly what the disclosure obligations exist for.

The free path

Performance transfer is currently mostly a hosted, paid feature, and that is an honest limitation rather than something to talk around. The free approximations: open lip-sync plus a separately generated head-motion pass; pose and face-landmark conditioning in ComfyUI, which is fiddlier and does work; or the oldest answer, which is to film a real person and composite them, covered earlier in the course.

Recording the audio well, which is half the result, is free on any phone.

Today: record yourself saying one line three ways: flat, with a pause before the last word, and with an eyebrow. Watch the three back. That difference is what transfer captures and lip-sync cannot.