A picture, out of noise

Not the same trick as the chatbot

A chatbot writes one token at a time, left to right. That does not work for a picture. A photograph has no first pixel, and a face drawn one pixel at a time would have no way of knowing, when it reached the second eye, what the first one looked like. Image models needed a different idea, and the one that won in 2022 is odd enough to be worth understanding properly.

Learn to clean, then run it backwards

Take a photograph. Add a little random static to it. Add a little more. Keep going for a thousand steps and you have pure noise: every trace of the picture is gone. That part is trivial. Anyone can ruin a photograph.

Now train a network on the reverse: show it a slightly noisy picture and ask it to predict the noise that was added, so the noise can be subtracted. Show it millions of pictures at every level of noisiness. What it learns is not any particular picture but what pictures in general look like, well enough to say which parts of a smudge are static and which parts are the beginnings of an edge.

Generation is that skill run from the wrong end. Start with a canvas of pure random noise. Ask the network what noise to remove. Remove a little. Ask again. After twenty to fifty rounds, what is left is a picture that was never in the training set, assembled by a machine that only ever learned to clean. The technical name is diffusion, after the physics of ink spreading in water, and the design dates from a 2015 paper that almost nobody read and a 2020 paper that everybody did.

Why it works at all

The first time most people hear this they ask the right question: where does the picture come from, if you started from static? The answer is that the network was trained on pictures with some real structure under the noise, and it has no way to tell whether the structure it thinks it sees is real. Shown pure static, it does what it always does (finds the most picture-like interpretation of the smudge) and commits to it. Every subsequent step sharpens whatever it committed to. The image is a hallucination in the most literal sense, and this time that is the point.

The trick that made it cheap

The version that reached the public in August 2022, which you can still download and run for nothing, added one thing. Working on every pixel of a large image is slow. So the picture is first compressed by a separate network into a much smaller grid (a 512-pixel square becomes a 64-by-64 grid of numbers that describe its content) and the noise-removal happens on that. At the end a decoder turns the small grid back into pixels. Denoising sixty-four squared is sixty-four times less work than five hundred and twelve squared. That is why it ran on a gaming card in a bedroom, and why a photograph-quality image now costs a fraction of a cent.

How words come in

The description above draws a random picture. To draw your picture, the noise-removal network is given the text as well, at every step, and steers toward an interpretation of the static that matches it. How the text is turned into something the network can use, and why the prompt works the way it does, is the next lesson.

Where to try it

The paid tools named in job advertisements are Midjourney, Adobe’s Firefly (built into Photoshop), and the image mode inside the big chatbots. Every one of them is a diffusion model of this shape.

The free path is the same mechanism in your own hands. Stable Diffusion in its several versions and Flux from a German lab are open-weights image models. Fooocus runs them behind a one-box interface; ComfyUI exposes every step as a node you can rewire; the Krita image editor has a free plugin that paints with them inside the canvas. The smaller models run on a graphics card with four gigabytes of memory, or on a laptop without one at a couple of minutes per image, or free on a hosted notebook.

What the mechanism cannot do

It has no idea what it drew. The network never represents “a cat” as an object; it represents what cat-like regions of a noisy grid tend to resolve to. That is why it cannot count (three cats is a texture of catness, not three things) and why asking it to change one detail in a finished image tends to change everything, because there is no “detail” in there to hold still. Editing tools work around this by masking a region and re-running the process on that region alone. The workaround is good. The limit underneath it is the mechanism.