Drawing the matte by hand

When there is no screen, somebody draws it

A great deal of professional compositing has no green screen at all. The shot exists, it was filmed on location, and you need a matte around an actor who was never keyed. The answer is rotoscoping: drawing the shape by hand, and animating it so it follows the subject.

It is slow, it is the entry-level job in every visual effects house, and it is the technique that makes everything else possible when the shoot did not cooperate.

Working practice that halves the time

Beginners roto frame by frame, forward, from frame 1. That is the slowest possible method.

Work in binary. Set the shape on the first frame and the last frame of the range. Go to the middle and correct it. Then the middle of each half. Then the middle of each quarter. The software interpolates between your keyframes, so you are only fixing what interpolation got wrong, and the error is largest at the midpoint. Typically you need a keyframe every 5 to 15 frames rather than every frame.

Split the subject into parts. One shape for the forearm, one for the upper arm, one for the torso, one per finger if the shot demands it. A single shape trying to follow an entire articulated body needs a keyframe on every frame; separate shapes each move and can be interpolated over long stretches. Parent them so the hand follows the forearm.

Match the point count to the shape, and keep it low. Every extra control point is another thing to animate. Most limbs need four to eight points. A shape with forty points is a shape you will not finish.

Do not track the outline; track the form. The outline of a rotating arm changes; the arm does not. Place points where features are, not evenly around the silhouette.

Motion blur is the real difficulty

A crisp spline against a crisply-filmed subject is manageable. A fast-moving limb is smeared across twenty pixels, and there is no line where the arm ends: there is a gradient from arm to background.

Three approaches, all partial:

• Feather the shape by an amount that varies along its length: heavier on the leading edge, lighter where the motion is small. Every roto tool supports per-point feather for exactly this reason.
• Render the matte with motion blur, letting the tool blur the shape according to how fast its points are moving between frames. This is accurate when the shape’s motion matches the subject’s, and wrong when it does not.
• Roto the shape as if there were no blur, then blur the resulting matte directionally to match. Crude, quick, and adequate for a matte that will sit under a heavy effect.

A shot with severe motion blur is a shot that should have been keyed or shot differently, and saying so early is part of the job.

Planar tracking: the technique that changed the discipline

For any part of a subject that is roughly flat (a forehead, a phone screen, a wall, a door, a clothing panel, a licence plate) you do not have to animate the shape at all. A planar tracker analyses the movement of a whole region of pixels and solves for the four-corner perspective transform that maps frame A onto frame B. You draw the shape once and the tracker carries it through the shot, including perspective change.

This is what Mocha does, and it is why “Mocha” appears in job listings as a skill of its own. It works on surfaces that stay planar and fails on things that deform: a face turning through profile, cloth folding, anything organic that bends out of plane.

The free equivalent is Blender’s motion tracking with a plane track: define a plane from four tracked markers and it solves the same corner pin, and you can parent a mask to it. It is capable, and it is the strongest free entry in this module.

When machine segmentation helps, and when it does not

Segmentation models will now produce a person matte from arbitrary footage, and the free ones (Segment Anything and its video successors, the rembg family, the person-segmentation features in DaVinci Resolve’s free edition) are remarkable at the body.

Two failure modes matter, and both follow from how the models work.

Temporal instability. A per-frame model has no knowledge of the previous frame, so the matte’s edge shifts slightly every frame even on a static subject. Played back, this reads as a crawling, boiling edge. Video-aware models reduce it and do not eliminate it.

Edges. These models are trained and scored on region-level accuracy, so being right about 99 per cent of the pixels (the interior) scores well while getting hair and motion blur wrong. Those are precisely the pixels a composite is judged on.

The productive use is therefore not “generate the final matte”. It is: use the model to produce a garbage matte and a core matte in seconds, and spend your hand-roto time only on the edge band. That is a genuine several-fold speed-up and it is available free today.

Quality control

Check roto by playing the matte, not by scrubbing it. A shape that looks right on every individual frame can chatter when played, because the control points are jittering by a fraction of a pixel. Chatter is far more visible in motion than a shape that is consistently a pixel too small.

View the composite with the background replaced by a flat, saturated colour (magenta is traditional) and play it at full speed. Anything that flickers is a problem; anything that holds steady is done.

The free path

Blender has a full mask editor with bezier shapes, per-point feather, animation and plane tracking, free. Fusion in DaVinci Resolve’s free edition has polygon and B-spline masks with tracking. Natron has a roto node modelled on Nuke’s. Segment Anything and rembg are open source and run on modest hardware. Mocha and Silhouette are the paid industry names, worth knowing for a CV, not required for the skill.

The limitation

Rotoscoping is honest manual labour and no technique removes that. A complex shot is measured in days per second of footage, which is why it is outsourced in volume and why the economics of the discipline are what they are.

It is also a reconstruction, not a measurement. You are deciding where the edge is, and on a blurred or low-contrast edge that decision is a judgement. Two artists produce two different mattes, both defensible, and the shot only needs the one that survives playback.