Interrogating a picture for vocabulary

What the tools do

Point a CLIP interrogator or a tagger at a photograph and it returns text: a caption, or a list of tags. The free options are the CLIP Interrogator, the WD tagger family, and any general vision model you can run locally or through a free tier.

What comes back looks like this:

a woman in a green sari sorting chillies, sitting on the floor,
whitewashed wall, harsh sunlight, shadows, documentary photography,
35mm, film grain, warm tones, india, natural light

Useful. And it will not regenerate the photograph.

Why it cannot give you the image back

An interrogator is a description model, not an inversion. It answers “what words fit this picture” by comparing the image against text embeddings and returning close matches. That is a different operation from recovering the specific noise, seed, model and prompt that produced a particular image, which, for a photograph, never existed anyway.

So the description is lossy in exactly the way descriptions always are. It captures category and salient attributes and loses arrangement, proportion, the precise light, the specific gesture. Feed the output back into a generator and you get an image of a woman sorting chillies in sunlight. Not that one.

People discover this and conclude the tool is broken. It is doing what it does; the expectation was wrong.

What it is good for

Recovering vocabulary you lack. You have a reference whose look you cannot name. The interrogator says chiaroscuro, rim lighting, tenebrism; and now you have three terms to use deliberately. This is the main value and it is a real one, especially early on.

Checking your assumptions. You think the reference is softly lit. The tagger says hard light. Look again; it is usually right, and your eye is being fooled by a bright background.

Building a caption style for LoRA training. When you caption a training set, consistency matters more than accuracy, and a tagger applied to every image gives you consistency for free. The consistency block returns to this.

Bulk sorting. Tag two hundred images and you can group them by style, which is how you build a reference library rather than a folder.

How to use the output properly

Never paste it in whole. It arrives full of redundancy and noise: best quality, 1girl, duplicated concepts, and terms that came from the tagger’s training vocabulary rather than from the image.

Read it, take the two or three words you did not have, and put them in the appropriate slot of your own structured prompt. The interrogator is a thesaurus, not a source.

The thing to be careful about

If the reference is somebody’s copyrighted work, an interrogated prompt fed straight back is an attempt to reproduce it. Whether that is lawful depends on jurisdiction, on how close the result is, and on questions that are unresolved and being argued in several courts right now. This course does not pretend to know the answer.

What can be said is narrower and more useful: taking vocabulary from a reference (hard side light, muted palette, 35mm) is ordinary practice, the same as any art director describing a look. Attempting to regenerate a specific existing picture is a different act with different exposure, and the difference is obvious to you when you are doing it.

A final practical note. The interrogator’s output tells you what the tagger sees, which reflects its own training data and biases. It will describe people in its own vocabulary, which is sometimes crude and sometimes worse, particularly about ethnicity and body type. Read the output before you use it rather than pasting it forward unseen, because whatever it hands you becomes your prompt and then becomes your picture.

A better use: interrogate your own output

Turn the tool around. Run it on an image you generated and compare what it says against what you asked for.

If your prompt said hard side light and the interrogator says soft studio lighting, one of two things is true: the model ignored that part of the prompt, or your eye is wrong about what came out. Either is worth knowing, and both are invisible without the comparison.

This is a surprisingly good calibration exercise in the first month. Generate ten images, interrogate all ten, and read the captions against your prompts. The slots that consistently fail to appear in the description are the ones the model is not acting on, and they are where your prompt-writing time should go.

The free options, concretely

• CLIP Interrogator: runs on Hugging Face Spaces without installing anything, or locally on a modest GPU.
• WD taggers: tag-style output, fast, well suited to bulk captioning for training sets.
• A small local vision-language model: the most flexible option, because you can ask a specific question rather than accepting a generic caption. “Describe only the lighting in this image” is far more useful than a full caption when lighting is what you are after.

That last approach is the one worth learning. A general caption answers a question you did not ask; a targeted question answers the one you did.