How words steer the picture

Two models, not one

When you type a prompt into an image generator, the diffusion network you met in the previous lesson never sees your words. A separate model reads them first and converts them into a set of numbers: the same idea as the embeddings in the chatbot module, where meaning became a position in space. Those numbers are handed to the denoiser at every step, and the denoiser steers the emerging image toward them.

Everything strange about image prompts follows from what that reader is, and what it was trained to do.

The reader that learned from captions

The reader used by the first public models was trained in 2021 on four hundred million pairs of images and their captions, scraped from the web. Its job was simple: given an image and a caption, place them near each other in the space; given an image and someone else’s caption, place them far apart. It learned which words go with which pictures.

It did not learn grammar, because captions do not need it. “A dog chasing a cat” and “a cat chasing a dog” have the same words and, in this reader, nearly the same position. It did not learn negation, because almost no caption says what is not in the picture. And it did not learn letters, because it saw words as whole tokens and never needed to know what a “B” looks like.

Three consequences you will meet immediately

Prompts behave like a list. Since the reader keeps concepts and drops structure, a prompt that is a comma-separated list of nouns and styles (“portrait, elderly fisherman, harbour at dawn, film grain”) works about as well as a sentence, and often better. The folk wisdom is right; the mechanism is a reader that cannot use the sentence anyway.

”No elephants” draws elephants. The word “elephant” enters the space as the concept of an elephant; the word “no” has almost no pull. The picture is steered toward elephants. Image tools added a separate box (a negative prompt) for exactly this reason. Technically the process is run twice, once toward your prompt and once toward the negative one, and the second is subtracted from the first. That subtraction is also how the strength of the steering is set: a “guidance” number, typically around seven, that says how far to push toward the words. Set it low and you get a loose, varied picture that ignores you; set it high and you get an over-saturated one that obeys every noun too hard.

Order of words matters a little, and position in the image not at all. “A red cube on a blue sphere” comes out as red and blue and a cube and a sphere in some arrangement. The reader has no left, right, above or below.

What fixed it

From 2022 onward the labs replaced or supplemented the caption-trained reader with a language model of the kind in the chatbot module: one trained on text, with grammar, that knows what “on” and “not” mean. Google’s 2022 model did this first; the third generation of Stable Diffusion and the Flux models did it in 2024. Two other changes worked alongside it. Training captions were rewritten by a vision model into long, detailed descriptions, because the web’s own captions are short and vague, and a model trained on “IMG_4032.jpg” learns nothing from it. And the readers began to see letters, which is a large part of why text in images went from gibberish to usable.

None of this made the model understand the sentence. It made the reader’s positions carry more of the sentence’s structure than before. Ask a 2025 model for a red cube on a blue sphere and it will usually oblige. Ask for five objects in a specific arrangement with two of them upside down and you will discover where the improvement stops.

What this means in practice

Describe what you want to see, not what you want to avoid. Put the subject first. Name the medium and the light. If the tool has a negative box, use it for the things that keep appearing. And when a model refuses to do what a sentence says, it is not being difficult; the reader has flattened your sentence into a cloud of concepts, and you are asking the cloud to have a shape it cannot hold. Rephrase toward the list, or reach for an editing tool that lets you place things by hand: every free tool named in the previous lesson has one.