Transferring a look from an image without training anything

A third way to use a reference

You have three mechanisms for giving a model a picture, and they do different things.

• img2img starts from the image, so content and layout survive.
• A control extracts one structural property and enforces it.
• An image-prompt adapter (IP-Adapter is the common family) injects the image’s appearance into the generation as though it were part of the prompt.

The third is the one people find hardest to place, and it is the one that answers “make it look like this, but of something else”.

What it actually does

The adapter runs the reference through an image encoder and feeds the resulting features into the model’s cross-attention layers, alongside the text. The model is then attending to a picture and a sentence at once.

The practical consequence is that it transfers a blend of everything the encoder noticed: palette, lighting, texture, mood, and (unless you take steps) composition and subject too. That blending is both the appeal and the weakness.

The variants worth knowing

Most implementations offer several modes, and choosing correctly matters more than tuning strength:

• Standard: transfers everything. Expect the subject to come along with the style.
• Style only: transfers palette, texture and light, leaves composition alone. This is the one you want most of the time.
• Composition only: the reverse; arrangement without appearance.
• Face variants: specialised for identity, covered in the consistency block.

Weight sits around 0.5 to 0.8 for a noticeable effect without domination. Above 1.0 the reference starts reproducing itself, which is both an aesthetic problem and, if the reference is somebody else’s work, a different kind of problem.

Adapter or LoRA

The question comes up constantly and the answer is a decision rule, not a preference.

Use an adapter when: you have one reference, you need it now, you want an approximate look, and it is a one-off. Setup is seconds and there is nothing to train.

Use a LoRA when: you need the same look across many images over time, you need it precise, you have 15 to 30 example images, and you can spend an hour training. The result is stronger, more consistent and reusable.

The rough shape: an adapter gives you perhaps 70% of a look in 30 seconds; a LoRA gives you 95% in an hour. For a single image, take the 70%. For a campaign, train.

Using it well

One reference, not five. Stacking adapters averages them into something characterless.

Match the medium. A photographic reference for a photographic output. Mixing an illustration reference into a photographic prompt produces a muddled result that is neither.

Crop the reference to what you want. If you want the palette, crop to a region that is mostly the palette. The encoder attends to the whole image, so a reference containing a person will contribute a person.

Combine with a control for structure. The adapter handles look, a depth or pose control handles arrangement. This pairing is the workhorse combination for consistent sets.

The limitations, stated plainly

You cannot say which attribute to take. “Take the palette but not the composition” is only available as far as the mode switch allows. Unlike a slot-structured prompt, you have no fine control over what transfers. This is the main frustration and it is inherent to the mechanism: the encoder produces one feature vector, not a labelled list of attributes.

It weakens fine prompt control. The image features compete with the text for attention, so detailed prompt instructions land less reliably when an adapter is running at high weight. If your prompt suddenly stops being followed, lower the adapter.

It can reproduce the reference too closely. At high weight with a distinctive source, the output can be recognisably derived from it. That is a practical risk when the reference is not yours, and the rights block returns to it. The working rule: if you can look at the output and identify the reference, the weight is too high for commercial work.

Quality depends on the encoder. These adapters are trained against a specific image encoder and base model. An adapter built for SD 1.5 will not work on SDXL, and mismatched combinations produce weak or bizarre results rather than clean errors.