LoRA for image models, and the likeness question

The same method, a different model

LoRA transferred from language models to image generators almost unchanged. You attach low-rank adapters to the attention projections of the denoising network, freeze everything else, and train on a small set of images with captions.

The scale is startling compared with anything else in this course: 10 to 30 images, 1,000 to 1,500 steps, 15 to 40 minutes on a consumer GPU or a free T4. A learning rate around 1e-4. The resulting adapter is tens of megabytes.

Three related techniques:

• LoRA: the general case. A subject, a style, an object, a composition.
• DreamBooth: a method for teaching one specific subject, using a rare token as its name and a set of class images to stop the concept swallowing the category.
• Textual inversion: trains only a new embedding vector, nothing else. A few kilobytes, weaker, and it composes unusually well with other adapters.

Every tool here is free: diffusers, kohya_ss, OneTrainer, ComfyUI.

The captioning rule almost nobody is told

This is the mechanism that decides whether a LoRA is usable.

What you caption becomes controllable. What you leave uncaptioned becomes baked in.

If every training image shows your subject against a white background and none of your captions mention the background, the model learns that white background is part of the subject. Ask for the subject on a beach and you will get a beach with a suspiciously white patch.

If instead your captions read “sks chair, white background”, the model learns that the white background is a separate, nameable attribute, and you can then ask for something else.

So: caption everything that varies and that you want to control. Leave uncaptioned only the thing you are actually training, and use a rare token for it so it does not collide with an existing concept.

Regularisation images

Without them, DreamBooth’s fine-tuning of a person tends to bleed into the whole category: you train on one face and afterwards every generated person resembles them. Class prior preservation counteracts this by mixing in generated images of the generic class, with generic captions, during training.

It is a real effect with a clear mechanism, it costs a few hundred generated images and some extra steps, and it is the difference between an adapter you can compose with others and one that dominates every prompt.

Overfitting, and what it looks like

Image LoRAs overfit visibly and fast. Symptoms:

• Every output has the same pose or composition as a training image.
• Backgrounds or artefacts from the training photos reappear regardless of the prompt.
• Prompt terms unrelated to the subject stop having any effect.

The fixes are the familiar ones (fewer steps, lower rank, lower learning rate, more varied training images) and the diagnostic is to generate at several checkpoints and look. As with text, save every few hundred steps and choose by eye rather than by loss; diffusion training loss is famously uninformative about sample quality.

Likeness, consent, and what is not settled

Training an adapter on a real person’s face is technically trivial and legally and ethically the most fraught thing in this course.

What can be said plainly:

• Training on somebody’s likeness without their consent is a decision you are making about them, not about a dataset.
• Presenting a generated image of a real person as a photograph of that person is a misrepresentation regardless of what any law says about it.

What cannot be said plainly, because it is unresolved and differs by country:

• Rights of publicity and personality exist in some jurisdictions and not others, with different scope and different remedies; in the United States they vary state by state.
• In the European Union a photograph of an identifiable person is personal data, and facial data used for identification is a special category with stricter conditions, but how those rules apply to model weights trained on such images, as opposed to the images themselves, is argued and not settled.
• Several jurisdictions have introduced or proposed specific rules on synthetic likenesses and disclosure, and they do not agree with each other.

This is a live area. Anybody who gives you a confident single answer is describing one jurisdiction, or their preference. If the question is commercially significant, it goes to a lawyer.

The practical position, which is defensible under any of these regimes: get consent, in writing, scoped to what you will actually do; do not train on faces scraped from the internet; label synthetic images as synthetic; and where you need a face for a persona or an avatar, use an illustrated one, because an illustration does not assert that a person exists.

Style, and the other unsettled question

Training a LoRA on a living artist’s work to reproduce their style is legally contested: copyright generally protects particular works rather than styles, but whether training on those works is itself an infringing use is exactly the question being litigated in several countries, with different answers so far. It is also, separately from the law, something many artists object to strongly. Both of those facts are worth holding at once.