Four dataset faults, and the LoRA each one ruins

The rule that explains every failure

A LoRA learns what does not change.

That is the whole thing. If the face is the only constant, the face is what it learns. If the face and the lighting and the background and the camera angle are all constant, it learns all four as one inseparable concept, and you get a LoRA that can only produce your character in that light, in that place, from that angle.

Every dataset fault is a violation of this rule.

Fault one: no variety in lighting

Twenty images shot in the same session under the same light. The LoRA now believes soft frontal window light is part of the person’s identity. Prompt for dramatic side lighting and you get a fight between the LoRA and the prompt, which the LoRA usually wins, producing a flat face awkwardly embedded in a dramatic scene.

Fix: vary it deliberately. Some soft, some hard, some outdoors, some indoors, some backlit. If you are generating the training images from a character sheet, generate them under different lighting on purpose.

Fault two: no variety in angle and distance

All front-facing at the same crop. The LoRA produces a good front view and falls apart in profile, because it has never seen one.

Fix: roughly half front and three-quarter, a quarter profile or near-profile, a quarter at greater distance including full body. The distant ones matter more than people expect: without them, the LoRA has no idea what the character’s proportions are and will produce a correct face on an arbitrary body.

Fault three: near-duplicate images

Five frames from the same burst, or the same image cropped five ways. These are nearly identical, so they count five times toward whatever they have in common, and the set is effectively much smaller than its file count.

Fix: one image per moment. Fifteen different images beat forty near-duplicates comfortably. If you are unsure, lay the set out as a contact sheet and look for anything that reads as the same picture twice.

Fault four: other people, or other versions of the subject

A second face in the background. A reflection. A photograph of the person at a very different age or with a very different haircut, mixed into a set that is otherwise consistent.

Fix: crop out other faces or discard the image. Pick one era and stay in it: a LoRA trained across a fifteen-year span learns an average that resembles neither.

The quality floor

Resolution: at least your training resolution, so 1024 for SDXL. Upscaling a small image to meet it teaches the model upscaling artefacts as part of the identity.

Sharpness: avoid heavily blurred or motion-blurred images. Some softness is fine and even useful, but a set full of soft images produces a soft LoRA.

Compression: avoid heavily compressed JPEGs. The blocking gets learned. This matters more than it sounds, and it is why a LoRA trained from images saved off a messaging app often looks subtly bad in a way nobody can identify.

A checklist before you train

Lay the set out as a grid and ask:

1. Is the subject the only thing constant across all of these?
2. Are there at least three distinct lighting situations?
3. Are there at least three distinct angles, including one distant?
4. Would any two of these be described as the same picture?
5. Is there another face anywhere in frame?
6. Is every image at or above training resolution, and reasonably sharp?
7. Do the captions name everything that varies and nothing that should not?

Ten minutes on this checklist saves the two hours of training and testing a bad LoRA and then working out why it is bad.

The honest constraint

You cannot always get variety. Sometimes you have eight photographs of a real person, all from one afternoon, and that is all that exists. Train on them anyway (a limited LoRA is better than none), but know what you have: a LoRA that works in that lighting and fails elsewhere, and a job where you should plan to match the scene lighting to the training lighting rather than the other way round.

And a note that belongs here rather than in the legal block, because it is a working decision rather than a legal one. If the training images are of a real person, you need their agreement, and “I found them online” is not agreement. The images also carry the rights of whoever took them. A dataset assembled from a search engine is a problem you are building into an asset you intend to reuse, and it is much cheaper to resolve before training than after a campaign has shipped.