Diffusion is not the only way

Four families, and what each is for

"AI image generator" covers several distinct machines. Knowing which you are using explains behaviour that otherwise looks arbitrary.

GANs. Two networks in a contest: a generator makes images, a discriminator tries to spot fakes, and both improve. GANs dominated until about 2021 and are still the fastest thing available — a single forward pass, milliseconds, no loop. StyleGAN's faces were startling in 2019 and still are. Their weakness is coverage: a GAN trained on faces makes superb faces and nothing else, and training one on everything tends to collapse onto a narrow slice of the data. They remain the right tool for narrow, high-volume jobs, and much free face-restoration and upscaling software is still GAN-based.

Autoregressive token models. Chop an image into a grid of discrete tokens, then predict them one at a time in sequence, exactly as a language model predicts words. This was DALL·E's original design, and it has returned: the image generation now built into several multimodal chat assistants works this way. Because the same network handles text and image tokens, it can use everything it knows about language, which is why these models are much better at rendering readable text in an image and at following long, fussy instructions. The cost is speed — one token at a time is a long sequence — and a different failure signature, where the picture drifts as it is written.

Diffusion. The subject of the last four lessons. Strong at texture and photographic realism, parallel over the whole image, controllable through a rich set of conditioning methods because the loop offers thirty opportunities to intervene.

Flow matching. The current direction, and the basis of most 2024-onward open models. Instead of learning to remove noise at many discrete levels, the model learns a straight-line velocity field from noise to data. In practice it is a cleaner training objective that converges faster and needs fewer steps. From the outside it behaves like diffusion with different default numbers, which is why guidance values that suit older models are wrong for these.

Why the distinction reaches you

Three practical consequences.

Text in images. If you need a poster with correct spelling, an autoregressive or hybrid model will usually beat a pure diffusion model, because it generates the picture as a sequence with the language machinery attached. This is a real capability difference, not a matter of prompting harder.

Editing. Diffusion models offer masks, structural conditioning and strength dials because the loop can be interrupted. Sequential token models are typically edited by asking again in conversation. If your work is iterative retouching, the diffusion toolchain gives you more handles.

Running it yourself. Open weights are overwhelmingly diffusion and flow-matching. The strongest autoregressive image models are, at the time of writing, API-only. Anyone who needs offline work, private data or unlimited volume is choosing from the diffusion family whether they intended to or not.

The honest limitation

None of these families has solved the same set of problems. The autoregressive models write text well and still miscount objects. Diffusion models render skin and cloth better and still cannot spell. Both fail on the same negation prompts. This is evidence that the remaining failures are not artefacts of one architecture but something deeper about learning images from captions, which is worth remembering when a vendor claims their new architecture has solved understanding.

A useful habit: when you meet a new tool, find out which family it belongs to before you form an opinion about it. The model card, the research paper or a single search will tell you. Then test the three things that family is known to be weak at. You will know in ten minutes what would otherwise take a month of confused experimenting.

Increasingly the answer is "more than one". Several current systems are hybrids: a language model reads your request and writes a detailed internal description, which a diffusion or flow model then renders. That is why some assistants follow a complicated instruction well while still producing an image with diffusion's characteristic strengths and weaknesses — you are talking to one machine and being drawn for by another. It also explains a behaviour people find eerie, where the picture reflects something you implied rather than said. The intermediate description was written by a model that infers, and you never see it. When a result is oddly interpreted, asking the system to state the description it used is often available, and it is usually the fastest way to find where your meaning was lost.