The greyscale image that decides everything

One greyscale image decides everything

Every composite in every film you have seen comes down to the same operation, repeated: two images, and a third greyscale image that decides how much of each one shows at every pixel. That third image is the matte.

White means show the foreground. Black means show the background. The grey values in between are the entire art of compositing, because they are the edges (hair, motion blur, glass, smoke) and edges are where a bad composite gives itself away.

The operation is called over, and in its straight form it is:

result = foreground × alpha + background × (1 − alpha)

Read it once and the whole discipline unpacks. At alpha 1 you get the foreground. At alpha 0 you get the background. At alpha 0.4 you get a weighted mix, which is exactly what a real semi-transparent edge looks like.

The vocabulary, which is not standardised

Different tools use different words for the same things, which is why documentation is confusing.

• Alpha channel: a matte stored as a fourth channel inside the image, alongside red, green and blue.
• Matte or mask: a matte held separately, often as a greyscale image or a drawn shape.
• Key: a matte generated automatically from the image’s own content, usually colour.
• Luma matte: using another image’s brightness as the matte.
• Alpha matte: using another image’s alpha as the matte.
• Garbage matte: a rough hand-drawn shape that throws away everything you know you do not want: the edge of the green screen, a light stand, the boom.
• Holdout matte: the inverse, used to stop one element drawing over another.
• Core matte: a shape safely inside the subject, guaranteed solid.

The technique that solves most difficult keys

Almost every hard key is solved the same way, and it is not by finding better keyer settings.

You build three mattes and combine them:

1. A garbage matte, drawn by hand, roughly, that removes everything outside the subject. Keyers work far better when they are not being asked to deal with a C-stand in the corner.
2. A core matte, drawn inside the subject, which is pure white. This guarantees the interior is solid even if the keyer produces holes in it: a dark jacket against a dark-lit green screen will always produce holes.
3. An edge matte from the keyer, used only in the band between the core and the garbage matte, where the soft detail lives.

Combine: core OR (key AND garbage). The keyer is now responsible for a thin band of edge rather than the whole frame, and you can push it much harder in that band than you could globally.

This is the single most useful structural idea in compositing, and it is the reason experienced compositors do not spend long fiddling with keyer sliders.

Reading a matte properly

You cannot judge a matte by looking at the composite. Look at the matte on its own, and look at it twice.

Gain it up. Multiply the matte by 5 or 10. Areas that should be solid white and are actually 0.95 will now be visible as grey: those are the holes that will show as a faint transparency of the background through your subject, and they are invisible at normal exposure.

Gain it down, or view the inverse. Areas of the background that should be pure black but are 0.02 will show up. That is background contamination, which appears as a faint glow of the subject where there should be nothing.

In practice: a matte that looks correct at normal viewing and falls apart when gained is the normal state of a first attempt. Nuke, Fusion, After Effects and Blender’s compositor all have a one-key matte view; use it constantly.

Where mattes come from

• Keyed: from colour, covered in the green-screen lesson.
• Drawn: rotoscoping, frame by frame, with splines.
• Rendered: a 3D render supplies a perfect alpha channel for free, plus ID mattes per object if you ask for them.
• Depth-derived: a depth pass or a depth-sensing camera can threshold into a matte. Consumer depth maps are low-resolution and noisy at edges, which is exactly where you need them to be good.
• Machine-learned: segmentation models now produce a usable person matte from arbitrary footage. They are astonishing at the body and poor at hair and motion blur, and they are temporally unstable: the matte wobbles between frames even when nothing moves. Useful as a garbage or core matte; rarely usable as an edge matte without work.

The free path

Blender’s compositor does everything in this lesson (nodes, keyers, mask shapes with animation, matte viewing) and is free and scriptable. Natron is an open-source node compositor modelled on Nuke; be aware its development has been intermittent, so check the state of the project before committing to it. DaVinci Resolve’s free edition includes Fusion, a full node compositor, which is the most capable free option by some distance. GIMP and Krita handle still composites. Nuke and After Effects are named in job adverts and neither is required to learn any of this.

The limitation

A matte is a single number per pixel, and reality is not. A strand of hair thinner than a pixel is a mixture of hair colour and background colour, and no matte can separate them: the information is gone, averaged together by the sensor. That is why hair composites over a new background often keep a faint memory of the old one, and why the fix is not a better keyer but a despill and an edge treatment that replaces the contaminated colour rather than trying to recover it.

The same applies to motion blur, glass, smoke and water. Compositing does not undo the mixing the camera did; it decides what to mix with instead.