{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"diffusion-lora-and-likeness","title":"LoRA for image models, and the likeness question","url":"https://addaly.com/learn/fine-tuning/diffusion-lora-and-likeness","position":63,"of":84,"minutes":10,"answer":"Image LoRAs train on 10 to 30 images in under an hour, and what you caption becomes controllable while what you leave uncaptioned is baked into the subject.","keyIdea":"Image LoRAs train on 10 to 30 images in under an hour, and what you caption becomes controllable while what you leave uncaptioned is baked into the subject, while likeness training is technically trivial and legally unsettled in ways that differ by jurisdiction, which makes consent and labelling the defensible position rather than a legal conclusion."},"body":{"format":"markdown","text":"## The same method, a different model\n\nLoRA transferred from language models to image generators almost unchanged. You attach low-rank adapters to the attention projections of the denoising network, freeze everything else, and train on a small set of images with captions.\n\nThe scale is startling compared with anything else in this course: **10 to 30 images**, 1,000 to 1,500 steps, 15 to 40 minutes on a consumer GPU or a free T4. A learning rate around 1e-4. The resulting adapter is tens of megabytes.\n\nThree related techniques:\n\n- **LoRA**: the general case. A subject, a style, an object, a composition.\n- **DreamBooth**: a method for teaching one specific subject, using a rare token as its name and a set of class images to stop the concept swallowing the category.\n- **Textual inversion**: trains only a new embedding vector, nothing else. A few kilobytes, weaker, and it composes unusually well with other adapters.\n\nEvery tool here is free: `diffusers`, kohya_ss, OneTrainer, ComfyUI.\n\n## The captioning rule almost nobody is told\n\nThis is the mechanism that decides whether a LoRA is usable.\n\n**What you caption becomes controllable. What you leave uncaptioned becomes baked in.**\n\nIf every training image shows your subject against a white background and none of your captions mention the background, the model learns that white background is part of the subject. Ask for the subject on a beach and you will get a beach with a suspiciously white patch.\n\nIf instead your captions read \"sks chair, white background\", the model learns that the white background is a separate, nameable attribute, and you can then ask for something else.\n\nSo: caption everything that varies and that you want to control. Leave uncaptioned only the thing you are actually training, and use a rare token for it so it does not collide with an existing concept.\n\n## Regularisation images\n\nWithout them, DreamBooth's fine-tuning of a person tends to bleed into the whole category: you train on one face and afterwards every generated person resembles them. Class prior preservation counteracts this by mixing in generated images of the generic class, with generic captions, during training.\n\nIt is a real effect with a clear mechanism, it costs a few hundred generated images and some extra steps, and it is the difference between an adapter you can compose with others and one that dominates every prompt.\n\n## Overfitting, and what it looks like\n\nImage LoRAs overfit visibly and fast. Symptoms:\n\n- Every output has the same pose or composition as a training image.\n- Backgrounds or artefacts from the training photos reappear regardless of the prompt.\n- Prompt terms unrelated to the subject stop having any effect.\n\nThe fixes are the familiar ones (fewer steps, lower rank, lower learning rate, more varied training images) and the diagnostic is to generate at several checkpoints and look. As with text, save every few hundred steps and choose by eye rather than by loss; diffusion training loss is famously uninformative about sample quality.\n\n## Likeness, consent, and what is not settled\n\nTraining an adapter on a real person's face is technically trivial and legally and ethically the most fraught thing in this course.\n\nWhat can be said plainly:\n\n- Training on somebody's likeness without their consent is a decision you are making about them, not about a dataset.\n- Presenting a generated image of a real person as a photograph of that person is a misrepresentation regardless of what any law says about it.\n\nWhat cannot be said plainly, because it is unresolved and differs by country:\n\n- Rights of publicity and personality exist in some jurisdictions and not others, with different scope and different remedies; in the United States they vary state by state.\n- In the European Union a photograph of an identifiable person is personal data, and facial data used for identification is a special category with stricter conditions, but how those rules apply to model weights trained on such images, as opposed to the images themselves, is argued and not settled.\n- Several jurisdictions have introduced or proposed specific rules on synthetic likenesses and disclosure, and they do not agree with each other.\n\nThis is a live area. Anybody who gives you a confident single answer is describing one jurisdiction, or their preference. If the question is commercially significant, it goes to a lawyer.\n\nThe practical position, which is defensible under any of these regimes: get consent, in writing, scoped to what you will actually do; do not train on faces scraped from the internet; label synthetic images as synthetic; and where you need a face for a persona or an avatar, use an illustrated one, because an illustration does not assert that a person exists.\n\n## Style, and the other unsettled question\n\nTraining a LoRA on a living artist's work to reproduce their style is legally contested: copyright generally protects particular works rather than styles, but whether training on those works is itself an infringing use is exactly the question being litigated in several countries, with different answers so far. It is also, separately from the law, something many artists object to strongly. Both of those facts are worth holding at once."},"check":{"question":"A LoRA trained on 20 photographs of a chair, all shot against a white backdrop, with captions reading only 'sks chair'. Prompted for the chair in a garden, the outputs show a garden with a white patch behind the chair. What is the mechanism?","options":[{"text":"The backdrop was never named in a caption, so the model absorbed it into the subject token's meaning.","correct":true},{"text":"Twenty images is too few, and the model interpolated between them rather than generalising to new scenes.","correct":false,"tempting":"Blames sample count for an effect that persists at 200 images when the backdrop is still uncaptioned."},{"text":"The rank was too low to represent both the chair and a varied background, so the model reused the training background.","correct":false,"tempting":"Frames it as a capacity limit; raising the rank without changing the captions reproduces the same backdrop, often more strongly."},{"text":"Class prior preservation was omitted, so the chair concept bled into the generic category of furniture and carried its setting with it.","correct":false,"tempting":"Names a real technique for a different failure (concept bleed across a category) rather than the caption-driven binding described here."}]},"prev":{"slug":"vision-language-models","title":"Fine-tuning a vision-language model","url":"https://addaly.com/learn/fine-tuning/vision-language-models"},"next":{"slug":"structure-without-training","title":"Getting structure without training for it","url":"https://addaly.com/learn/fine-tuning/structure-without-training"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'LoRA for image models, and the likeness question', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/diffusion-lora-and-likeness (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"LoRA for image models, and the likeness question","courseTitle":"Fine-Tuning, and When Not To","module":{"index":7,"slug":"beyond-text-generation","title":"Fine-tuning beyond a chat model"},"answer":null,"keyIdea":"Image LoRAs train on 10 to 30 images in under an hour, and what you caption becomes controllable while what you leave uncaptioned is baked into the subject, while likeness training is technically trivial and legally unsettled in ways that differ by jurisdiction, which makes consent and labelling the defensible position rather than a legal conclusion.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-same-method-a-different-model","runs":[{"text":"The same method, a different model"}]},{"t":"p","runs":[{"text":"LoRA transferred from language models to image generators almost unchanged. You attach low-rank adapters to the attention projections of the denoising network, freeze everything else, and train on a small set of images with captions."}]},{"t":"p","runs":[{"text":"The scale is startling compared with anything else in this course: "},{"text":"10 to 30 images","b":true},{"text":", 1,000 to 1,500 steps, 15 to 40 minutes on a consumer GPU or a free T4. A learning rate around 1e-4. The resulting adapter is tens of megabytes."}]},{"t":"p","runs":[{"text":"Three related techniques:"}]},{"t":"ul","items":[[{"text":"LoRA","b":true},{"text":": the general case. A subject, a style, an object, a composition."}],[{"text":"DreamBooth","b":true},{"text":": a method for teaching one specific subject, using a rare token as its name and a set of class images to stop the concept swallowing the category."}],[{"text":"Textual inversion","b":true},{"text":": trains only a new embedding vector, nothing else. A few kilobytes, weaker, and it composes unusually well with other adapters."}]]},{"t":"p","runs":[{"text":"Every tool here is free: "},{"text":"diffusers","code":true},{"text":", kohya_ss, OneTrainer, ComfyUI."}]},{"t":"h2","id":"the-captioning-rule-almost-nobody-is-told","runs":[{"text":"The captioning rule almost nobody is told"}]},{"t":"p","runs":[{"text":"This is the mechanism that decides whether a LoRA is usable."}]},{"t":"p","runs":[{"text":"What you caption becomes controllable. What you leave uncaptioned becomes baked in.","b":true}]},{"t":"p","runs":[{"text":"If every training image shows your subject against a white background and none of your captions mention the background, the model learns that white background is part of the subject. Ask for the subject on a beach and you will get a beach with a suspiciously white patch."}]},{"t":"p","runs":[{"text":"If instead your captions read “sks chair, white background”, the model learns that the white background is a separate, nameable attribute, and you can then ask for something else."}]},{"t":"p","runs":[{"text":"So: caption everything that varies and that you want to control. Leave uncaptioned only the thing you are actually training, and use a rare token for it so it does not collide with an existing concept."}]},{"t":"h2","id":"regularisation-images","runs":[{"text":"Regularisation images"}]},{"t":"p","runs":[{"text":"Without them, DreamBooth’s fine-tuning of a person tends to bleed into the whole category: you train on one face and afterwards every generated person resembles them. Class prior preservation counteracts this by mixing in generated images of the generic class, with generic captions, during training."}]},{"t":"p","runs":[{"text":"It is a real effect with a clear mechanism, it costs a few hundred generated images and some extra steps, and it is the difference between an adapter you can compose with others and one that dominates every prompt."}]},{"t":"h2","id":"overfitting-and-what-it-looks-like","runs":[{"text":"Overfitting, and what it looks like"}]},{"t":"p","runs":[{"text":"Image LoRAs overfit visibly and fast. Symptoms:"}]},{"t":"ul","items":[[{"text":"Every output has the same pose or composition as a training image."}],[{"text":"Backgrounds or artefacts from the training photos reappear regardless of the prompt."}],[{"text":"Prompt terms unrelated to the subject stop having any effect."}]]},{"t":"p","runs":[{"text":"The fixes are the familiar ones (fewer steps, lower rank, lower learning rate, more varied training images) and the diagnostic is to generate at several checkpoints and look. As with text, save every few hundred steps and choose by eye rather than by loss; diffusion training loss is famously uninformative about sample quality."}]},{"t":"h2","id":"likeness-consent-and-what-is-not-settled","runs":[{"text":"Likeness, consent, and what is not settled"}]},{"t":"p","runs":[{"text":"Training an adapter on a real person’s face is technically trivial and legally and ethically the most fraught thing in this course."}]},{"t":"p","runs":[{"text":"What can be said plainly:"}]},{"t":"ul","items":[[{"text":"Training on somebody’s likeness without their consent is a decision you are making about them, not about a dataset."}],[{"text":"Presenting a generated image of a real person as a photograph of that person is a misrepresentation regardless of what any law says about it."}]]},{"t":"p","runs":[{"text":"What cannot be said plainly, because it is unresolved and differs by country:"}]},{"t":"ul","items":[[{"text":"Rights of publicity and personality exist in some jurisdictions and not others, with different scope and different remedies; in the United States they vary state by state."}],[{"text":"In the European Union a photograph of an identifiable person is personal data, and facial data used for identification is a special category with stricter conditions, but how those rules apply to model weights trained on such images, as opposed to the images themselves, is argued and not settled."}],[{"text":"Several jurisdictions have introduced or proposed specific rules on synthetic likenesses and disclosure, and they do not agree with each other."}]]},{"t":"p","runs":[{"text":"This is a live area. Anybody who gives you a confident single answer is describing one jurisdiction, or their preference. If the question is commercially significant, it goes to a lawyer."}]},{"t":"p","runs":[{"text":"The practical position, which is defensible under any of these regimes: get consent, in writing, scoped to what you will actually do; do not train on faces scraped from the internet; label synthetic images as synthetic; and where you need a face for a persona or an avatar, use an illustrated one, because an illustration does not assert that a person exists."}]},{"t":"h2","id":"style-and-the-other-unsettled-question","runs":[{"text":"Style, and the other unsettled question"}]},{"t":"p","runs":[{"text":"Training a LoRA on a living artist’s work to reproduce their style is legally contested: copyright generally protects particular works rather than styles, but whether training on those works is itself an infringing use is exactly the question being litigated in several countries, with different answers so far. It is also, separately from the law, something many artists object to strongly. Both of those facts are worth holding at once."}]}],"check":{"question":[{"text":"A LoRA trained on 20 photographs of a chair, all shot against a white backdrop, with captions reading only ‘sks chair’. Prompted for the chair in a garden, the outputs show a garden with a white patch behind the chair. What is the mechanism?"}],"options":[{"text":[{"text":"The backdrop was never named in a caption, so the model absorbed it into the subject token’s meaning."}],"correct":true,"tempting":null},{"text":[{"text":"Twenty images is too few, and the model interpolated between them rather than generalising to new scenes."}],"correct":false,"tempting":[{"text":"Blames sample count for an effect that persists at 200 images when the backdrop is still uncaptioned."}]},{"text":[{"text":"The rank was too low to represent both the chair and a varied background, so the model reused the training background."}],"correct":false,"tempting":[{"text":"Frames it as a capacity limit; raising the rank without changing the captions reproduces the same backdrop, often more strongly."}]},{"text":[{"text":"Class prior preservation was omitted, so the chair concept bled into the generic category of furniture and carried its setting with it."}],"correct":false,"tempting":[{"text":"Names a real technique for a different failure (concept bleed across a category) rather than the caption-driven binding described here."}]}],"order":[2,3,0,1],"why":null},"prev":{"slug":"vision-language-models","title":"Fine-tuning a vision-language model","minutes":9},"next":{"slug":"structure-without-training","title":"Getting structure without training for it","minutes":9}}}