Noise in, picture out: inpainting explained

The training idea, in one paragraph

A diffusion model is trained by taking real images and adding noise to them in steps until nothing remains but static. The model learns to predict, at any noise level, what noise was added, which is equivalent to learning how to remove a little of it. Run that prediction repeatedly starting from pure static and you get an image, because each step nudges the static towards something the model considers image-like.

Text conditioning steers the nudging. A text encoder turns your prompt into a vector, and the denoising network is trained so that its predictions depend on that vector. “A ceramic bowl” pushes each step towards arrangements the model associates with those words.

Inpainting: the same thing with a mask

When you select an area and ask for generative fill, four things go in:

1. The masked region, which is initialised with noise.
2. The surrounding pixels, which are kept fixed and re-imposed at every denoising step, so the generated area is continuously pulled towards agreeing with its edges.
3. Your prompt, if you gave one.
4. A denoising strength, which decides how much of the original content under the mask survives. At 1.0 the region is generated from scratch; at 0.4 the original is only partially overwritten, which is useful for changing a texture while keeping a shape.

Most systems work in a latent space rather than on pixels: an encoder compresses the image by a factor of eight in each direction, the diffusion happens in that smaller space, and a decoder expands the result back to pixels. This is why the whole thing is fast enough to be a button, and it is the source of several of the artefacts below.

What follows from the mechanism

Generated regions are smoother and cleaner than photographs. The model produces what it considers a likely image, and likely images are not grainy. A filled region in a high-ISO photograph is conspicuously noise-free. Add matched grain over the whole picture afterwards, as in the compositing block.

The latent grid produces a resolution ceiling. Because diffusion happens at one-eighth scale, fine detail is decoded rather than generated, and the result is often slightly soft compared with the surrounding photograph. On a 45-megapixel file with a small fill, the difference is obvious.

Seams appear when the mask boundary is too tight. The surrounding context is what keeps the generation consistent, and a mask drawn exactly at the edge of the object gives the model almost no overlap to work with. Expand the selection by 10 to 30 pixels and the seam usually disappears.

Text comes out wrong. The model has learned the visual statistics of letterforms, not a writing system. A generated shopfront has letter-shaped marks that are not letters. This has improved and it has not been solved, and for anything with real text the answer is to composite the real text in.

Regular geometry drifts. Brick courses that do not line up, window frames that change width, tiles that lose their grid. The model has no representation of a straight line continuing; it has a statistical sense of what brickwork looks like locally.

Hands, small faces and reflections are the classic failures, all for the same reason: they are high-complexity structures that occupy few pixels, so errors there cost the training objective very little.

It is not deterministic. The same prompt on the same mask with a different random seed gives a different result. Three attempts cost three seconds and one of them is usually much better than the others, which is why every implementation offers variations.

The controls worth learning

• Mask feather and expansion. The single most effective control. Feather 5 to 15 pixels, expand 10 to 30.
• Denoising strength, where exposed. High for replacing, low for modifying.
• Prompt or no prompt. An empty prompt tells the model to continue the surroundings, which is what you want for removing an object. A described prompt is for adding something.
• Generate in several small passes rather than one large one. Each pass has more surrounding context relative to its size, so each is more consistent.
• Seed, where exposed, so you can reproduce a result you liked.

Where this sits against the earlier tools

The previous module’s content-aware fill copies patches from your own photograph. Diffusion inpainting generates new content from a model trained on other people’s photographs. The practical difference:

• Content-aware is limited to what is in the frame, so it cannot invent a doorway, and everything it produces was in your image.
• Diffusion can produce a doorway, and that doorway was never there.

Neither is better in general. For removing a bin against a plain wall, content-aware is faster and safer. For removing a car from a complex street, diffusion will produce a more convincing result and will have invented the shopfront behind it.

What it does not do

It does not know what is true. The model has no representation of your client’s product, this particular person’s face, the actual building behind the van, or what was written on the sign. It produces the most probable-looking content given the surroundings and the prompt, and probability is not evidence.

That statement is the whole of the ethical discussion later in this module, and it is worth holding as a technical fact rather than a moral one: the output is a sample from a distribution, not a measurement of a scene. Everything else follows from deciding whether the picture in front of you is allowed to contain samples.