Canny, lineart, depth, softedge, pose, normal

One question decides it

Not “which control is best”. What must survive from my input, and what must be free to change?

Answer that and the control picks itself.

The types, and what each keeps

Canny detects hard edges and produces a thin black-and-white line image. It preserves every edge, including texture edges, noise and the grain of a wall. Extremely faithful and correspondingly rigid. Use it when the source is clean and you want the output to match it closely: a product outline, a logo shape, an architectural photograph you are restyling. Its threshold settings matter enormously: too low and you capture noise, too high and you lose the object.

Lineart is a learned edge extractor that produces drawing-like lines. It ignores texture and keeps object boundaries. Better than Canny for anything organic, and the right choice for turning a clean drawing into an image.

Scribble is the loosest. Trained on rough, wobbly, human lines. Forgiving of bad drawing, which is exactly what you want from a five-minute sketch.

Softedge (HED and its relatives) sits between lineart and depth: soft, thick boundaries that convey form without dictating exact contours. It is the most forgiving control for photographic sources and my default when I want “roughly this arrangement” rather than “exactly this shape”.

Depth keeps distance and volume, discards edges, texture and colour. The right tool for spatial arrangement and perspective, as the previous lesson covered.

Normal encodes surface orientation: which way each bit of a surface is facing. It preserves three-dimensional form and how light will fall on it, more finely than depth. Useful for detailed objects, faces and anything where the sculptural quality matters.

OpenPose extracts a skeleton: joints and limbs, nothing else. It keeps a body’s position and gesture and discards body shape, clothing, age and identity entirely. Unmatched for posing a figure, and useless for anything that is not a body.

Segmentation labels regions by category: sky here, building there, road below. Keeps layout at the level of what-kind-of-thing-goes-where, discards all form. Good for landscape and urban scenes.

Matching a job to a type

• Turn my sketch into a photograph → scribble
• Restyle this photograph but keep the exact objects → canny or lineart
• Keep this room’s geometry, change everything about its look → depth
• Put a different person in this exact pose → openpose
• Keep this figure’s pose and the room’s layout → openpose plus depth
• Keep the sculptural form of this object → normal
• Keep the general arrangement, let details change → softedge
• Recolour a product but hold its silhouette exactly → canny, high strength

The two settings that matter more than the type

Strength: how hard the control pushes, usually 0 to 2 with 1.0 as neutral. Below 0.5 it is a suggestion; above 1.3 it dominates and the output starts to look like a coloured-in control image. The working range is 0.6 to 1.0 for most jobs.

Start and end steps: when the control is active. Ending it at 0.6 to 0.8 of the schedule is right for most work, for the reason the sketch lesson gave: the last portion of sampling should be free to build texture. Starting late (around 0.2) is an unusual but useful trick when you want the model to establish its own composition and then be nudged toward your structure.

People reach for a different control type when the real problem is that strength is at 1.0 and should be at 0.7. Change the settings before the type.

The failure mode of every control

All of them share one: too much control produces something that looks constructed. Flat, posed, slightly dead, with the subject’s outline just a bit too exactly matching the input.

The cause is straightforward. The generative process has degrees of freedom, and each constraint removes some. Constrain the edges, the depth, the pose and the colour and there is very little left for the model to decide, so the output stops having the incidental richness that makes an image look real: the small asymmetries, the details nobody specified.

The practical rule that follows: constrain only what the brief actually requires, at the lowest strength that holds it. If the brief needs the pose, use pose and nothing else. Adding depth “for safety” costs you the liveliness and buys you nothing the brief asked for.