Solving for a camera, and when you cannot

From flat tracks to a camera in space

A 2D track tells you where a feature moved on the screen. A camera solve (also called matchmoving or a 3D track) works out where the camera was in three dimensions on every frame, and where a cloud of tracked points sits in the world. Once you have that, you can put a 3D object into the shot and it will stay put as the camera moves, with correct perspective.

The solver is doing structure from motion: it has many features, each seen from many camera positions, and it searches for the set of 3D point positions and camera poses that best explains all the 2D observations at once. The measure of “best” is reprojection error: project each solved 3D point back through each solved camera and measure how far it lands from where the feature actually was, in pixels.

That number is the one thing to look at. Under about 0.5 pixels on a 1080p plate is a solve you can build on. Between 1 and 2 is marginal and will slide. Above 3, something is wrong and no amount of adding more tracks will fix it.

The limitation that surprises everybody

A camera that only rotates cannot be solved for 3D.

If the camera is on a tripod and pans, every feature in the frame moves in a way that depends only on the rotation. A near object and a distant mountain shift by the same amount on the sensor. There is no parallax, so there is no depth information, so the solver has nothing to triangulate with. It is not a software limitation: the information does not exist in the footage.

This is why solvers offer a separate nodal or tripod mode: it solves rotation only, gives you no point cloud and no translation, and is entirely adequate for adding a distant element like a sky replacement or a far-off building. What it cannot do is let you place an object on the floor in front of the camera, because there is no floor in the solution.

The practical consequence on set: if you want a 3D solve, move the camera sideways. Even a metre of lateral movement over a shot turns an unsolvable pan into a solvable one. A handheld shot usually solves well for precisely this reason: the operator’s sway supplies parallax for free.

What the solver needs from you

Enough tracks, spread through depth. Twenty to forty good tracks is typical. They must not all be on one plane: tracks that all sit on a flat wall give an ambiguous solution, because a plane seen from different angles has multiple consistent interpretations. Get near objects and far objects in the set.

Tracks that last. A solve is stronger when features persist across many frames, because each one is then constraining many camera positions. A shot where every track lives for eight frames is a weak solve even if there are two hundred of them.

The focal length, if you have it. Solving for the focal length as well as the geometry is possible and less accurate. If you know the lens was 35mm on a Super 35 sensor, tell the solver and it has one fewer unknown. Most modern cameras and phones record the focal length in metadata; exiftool will read it out of the file, free.

Lens distortion handled. A real lens bends straight lines, most visibly at wide angles. The solver assumes a pinhole camera. If you feed it distorted footage, the error shows up as a solve that fits the centre of frame and drifts at the edges. The standard workflow is: estimate the distortion, undistort the plate, solve and composite on the undistorted image, then redistort the finished comp so it matches the original lens. Blender, Fusion and Nuke all have this loop, and skipping it is a common cause of an insert that looks glued on near the frame edge.

Setting the scene up afterwards

A solve gives you a camera and a point cloud in arbitrary units, with an arbitrary origin and orientation. It does not know which way is up or how big anything is. Before you can place an object you have to constrain it:

• Pick three or more points you know lie on the ground and define them as the floor plane.
• Pick two points a known distance apart and set the scale: the gap between two paving slabs, a person’s height, a door.
• Set an origin so your scene’s coordinates are meaningful.

Skipping this is why a first solve produces a cube that drifts through the floor at forty-five degrees.

Checking the solve honestly

Do not judge it by looking at the error number alone. Put a simple 3D object in the scene (a grid on the floor, a cube on a surface, a sphere at a tracked point) and play the shot at full speed. A solve with acceptable numbers can still slide if the distortion was not handled or the scene orientation is slightly off, and playback shows it in a second.

Then check the individual track errors and delete the worst offenders. A single bad track with a large error can distort the whole solution, and removing three bad tracks is usually worth more than adding thirty good ones.

The free path

Blender has a complete matchmoving pipeline: 2D tracking, lens distortion estimation, a camera solver with per-track error readout, scene orientation tools, and then the 3D scene and compositor in the same application. It is used on real productions and costs nothing. This is a case where the free tool is a first-class option rather than a compromise. COLMAP and Meshroom are open-source structure-from-motion packages aimed at photogrammetry that can also produce camera solutions. PFTrack, 3DEqualizer and SynthEyes are the paid specialists.

The limitation

A solve describes a rigid world seen by a moving camera. Anything in the scene that moves independently is a contaminant, and the solver has no way to know which tracks are on the moving car and which are on the road. You remove them by hand, and on a busy shot that is most of the work.

Rolling shutter is the other reliable spoiler. Most CMOS cameras expose the frame line by line over several milliseconds, so during fast movement the top and bottom of the frame are from different moments and straight verticals lean. A solver assuming a global shutter fits an average that is wrong everywhere. Some tools compensate; the compensation is a model, not a measurement, and heavy rolling shutter remains a reason a shot cannot be solved to a usable error.