Every generation is a guess about where things go
The prompt says a woman reaching upward for a book on a high shelf. The model produces a woman, a shelf and some books, and arranges them however the training data suggests. Run it again and the arrangement changes. You are sampling from a distribution of plausible compositions, not specifying one.
Conditioning changes that. Instead of hoping, you hand the model the composition directly.
The mechanism, in one paragraph
Take a reference image and run it through a preprocessor that throws away almost everything and keeps one kind of structure: a stick figure of the joints, a greyscale near-and-far map, a map of edges. A second network reads that structure and, at every denoising step, nudges the image toward matching it. The prompt still decides what things look like. The map decides where they are.
This is the ControlNet family, released in 2023 by Lvmin Zhang, and it is the single biggest reason working practitioners use ComfyUI or Forge rather than a hosted box.
Which map for which job
- OpenPose — a skeleton of body, hand and face keypoints. Use it when the pose is the thing and nothing else about the reference should carry over. You can pose a stick figure by hand and generate from that, with no source photograph at all.
- Depth — a greyscale image where near is bright and far is dark. Use it to keep a three-dimensional layout while changing everything's appearance: same room, different decade. The best choice for architecture, interiors and putting a product into a scene.
- Canny — a strict edge map. Keeps every line it finds. Use for turning a clean line drawing into a rendered image.
- Lineart and Softedge — looser edge detection. Softedge in particular gives the model room to reinterpret while keeping the shape.
- Scribble — deliberately crude. Draw with a mouse in thirty seconds, get a composition.
- Segmentation — regions labelled by category: sky here, building there, road there.
The two settings that decide whether it looks alive
Control weight. At 1.0 the conditioning is enforced hard. Drop to 0.6-0.8 and the model has room to make the image work as an image.
Start and end percent. This is the one people miss. You can apply the control for only part of the denoising process. Composition is decided early; detail and texture are decided late. So run the control from 0% to about 60% of the steps and then let go. The layout is locked in by then, and the model spends the final steps resolving the picture naturally instead of tracing.
An image conditioned at weight 1.0 for 100% of the steps looks stiff, flat and traced, because it is. Ending the control early is the difference between a render and a colouring-in exercise.
The failure modes
Stacked controls fighting each other. Pose plus depth plus canny, all at full weight, leaves the model no freedom and no way to satisfy all three. The result is muddy and rigid. Use one control. Add a second only when you can name what it is contributing.
Canny eating artefacts. Canny finds edges, and JPEG compression blocks, paper grain and scanner noise all look like edges. Run canny on a compressed photo and you get thin phantom lines in the sky. Raise the canny thresholds, or clean the source first, or use softedge instead.
Depth flattening. Depth maps from a photo with a plain background often read the whole background as one distance, so your generated scene has a wall where you wanted a street.
Where hosted tools stand
Some expose a sketch-to-image or pose-reference feature that is conditioning under a friendlier name. Most do not expose it at all, and none of them give you the weight and scheduling controls. If your work needs a pose held across a series, this is usually the moment you install something local.
The free path is the whole path
There is no paid version of this that is better. The ControlNet models are free downloads on Hugging Face. ComfyUI and Forge expose every setting. Krita with AI Diffusion exposes them in a way that will not frighten you, with a canvas you can draw the scribble on directly. Free GPU hours on Colab or Kaggle run all of it. See running models yourself.
Today: draw a stick figure or a five-line scribble, run it through scribble or openpose conditioning at weight 0.8 ending at 60%, and watch a composition you chose come back rendered. That is the moment most people stop treating generation as a slot machine.
Before you move on