What a tracker can and cannot see

What a tracker is actually doing

A 2D point tracker does one thing per frame: it takes a small rectangle of pixels from the previous frame (the pattern) and searches a larger rectangle in the current frame for the position where those pixels match best. It reports that position, usually to a fraction of a pixel by fitting a curve to the correlation scores around the best whole-pixel match.

That is the whole mechanism. Every tracking failure follows from it, and once you can name the mechanism you stop guessing at settings.

The aperture problem, which explains most failures

Put a tracker on a long straight edge: the top of a table, the join between two walls, a cable against the sky. The track slides along the edge and the result is useless.

The reason is geometric. Through a small window onto a straight edge, motion along the edge produces no change in the pixels at all. The pattern at position x looks identical to the pattern at position x+5. The tracker can measure the component of motion perpendicular to the edge and has no information whatsoever about the component parallel to it. This is the aperture problem, and it is not a limitation of the software.

The consequence is the single most useful rule in tracking: a good feature has contrast in two directions. A corner. A crossing. A bolt head. A high-contrast blemish. Not an edge, not a smooth gradient, not a repeating texture like brickwork: repeats give the search several equally good answers and the track jumps between them.

Blender’s tracker and most others can show you a correlation score per frame. A feature that is trackable holds above 0.9; one that drifts along an edge falls off steadily and tells you before the result is visibly wrong.

Pattern size and search size

Two rectangles, two different jobs.

The pattern is what is being matched. Make it large enough to contain a distinctive feature and small enough that everything inside it moves together. If your pattern spans two objects at different depths, they move by different amounts and no single position matches both.

The search area is where the tracker looks. It must be large enough to contain the frame-to-frame movement and no larger: a bigger search area is slower and offers more chances to find a false match. If the camera whips, the movement between frames can be a hundred pixels and a default search area will lose the feature.

When a track fails on a fast move, the cause is nearly always the search area, not the pattern.

The four things that break a track

Motion blur. A smeared feature has no sharp structure to correlate against. Trackers commonly survive mild blur and fail on heavy blur, and the usual answer is to track a different, larger feature that stays distinguishable when smeared.

Occlusion. Something passes in front. Every tracker offers a way to keyframe the track manually through the occlusion and resume; on a short occlusion, interpolating between the position before and after is usually accurate enough.

Lighting change. The pattern is matched on pixel values, so a feature that moves through a shadow changes its values. Most trackers can normalise for brightness, which handles a gradual change and not a hard shadow edge.

Parallax inside the pattern. A feature near a depth discontinuity (a corner of a building against a distant background) changes its surroundings as the camera moves. Shrink the pattern until it contains only the near object.

Using a track

Once you have a path, there are three standard applications.

• Stabilise. Apply the inverse of the track to the footage. This crops, because the frame is being moved around inside itself, so a stabilised shot is always smaller than its source: typically 5 to 15 per cent. Decide this before framing the shot, not after.
• Match-move. Apply the track to an inserted element so it appears attached. One point gives position; two points give position, rotation and scale; four points give a full perspective corner pin.
• Drive an effect. A blur, a censor patch, a colour correction that follows a moving object.

Planar over point, when you can

For any surface that is flat and stays flat, a planar track is more robust than four point tracks, because it uses thousands of pixels rather than four small patches and can tolerate parts of the surface being occluded or badly lit. A phone screen, a poster, a wall, a face’s forehead: all planar enough. This is the practical reason planar tracking took over so much of the work that used to be point tracking.

The free path

Blender’s motion tracking is a full implementation (2D tracks with adjustable pattern and search, correlation display, plane tracks, and a solver) and it is free with no watermark and no resolution cap. It is the single best free tool in this module. Fusion in DaVinci Resolve’s free edition has a tracker and a planar tracker. Kdenlive and Shotcut both expose simple motion tracking for effects. Mocha is the paid standard and worth naming on a CV.

The limitation

A tracker reports where a pattern of pixels went. It does not know what an object is. It will happily track a reflection, a shadow, or a specular highlight that is sliding across a surface rather than attached to it, and a highlight is a superb-looking track that is physically wrong, because its position depends on the light and the camera rather than on the object.

The result looks convincing in the tracker’s own display and slips visibly once you attach something to it. The defence is to check what you are tracking with your eyes before you press the button, and to test the result by attaching a crosshair and playing the shot at full speed.