Training a character LoRA on hardware you can borrow

What you are actually doing

A LoRA adds small trainable matrices alongside the model’s existing weights and trains only those. The base model is untouched. That is why the result is 20 to 150 MB rather than several gigabytes, and why it trains in minutes rather than weeks.

For a character, you are teaching the model to associate a rare token (ohwx, sks, or any string it does not already have a meaning for) with a specific face.

The recipe

Reasonable starting settings for an SDXL character LoRA:

images:            20  (15 minimum, 30 is plenty)
resolution:        1024
repeats:           10
epochs:            10
total steps:       20 x 10 x 10 / batch 2  = 1000
network rank:      16      (8 for a simple style, 32 for a complex identity)
network alpha:     8       (half the rank is a common convention)
learning rate:     1e-4    (unet), 5e-5 (text encoder)
scheduler:         cosine with 10% warmup
optimiser:         AdamW8bit
save every epoch:  yes

Save every epoch. This is not optional advice: it is how you find the right amount of training. You will end up with ten files and the best one is usually not the last.

Running it free

Kaggle Notebooks give about 30 GPU-hours a week, enough for many training runs. Google Colab free tier works for SD 1.5 comfortably and SDXL with care. Both have ready-made notebooks for the common trainers: Kohya’s scripts and OneTrainer are the usual choices, both free and open source.

Expect, for 1000 steps at 1024 resolution: roughly 25 to 40 minutes on a T4, 12 to 20 on a better card. Session limits are the real constraint, not capability. Upload the dataset to the session, train, and download the resulting files before the session ends, because it will end.

Locally, 12 GB of VRAM trains SDXL LoRAs comfortably; 8 GB works with gradient checkpointing and a batch size of 1, slowly.

Finding the right epoch

Generate the same prompt and seed with each saved epoch and lay them out as a grid.

• Early epochs (1 to 3): identity weak, prompt followed well.
• Middle (4 to 7): identity present, prompt still working. This is where the good one is.
• Late (8 to 10): identity strong, prompt increasingly ignored, backgrounds from your training images appearing.

That last symptom is the definition of overfitting, and it is unmistakable once you know it: you ask for your character on a beach and get your character in front of the grey wall that was in twelve of your training photographs. The model has learned the wall as part of the concept.

Pick the middle epoch. Almost everyone’s first LoRA is overtrained, because more training feels like more learning.

Captions

Each training image gets a caption. The rule is counter-intuitive and worth stating clearly: caption what should vary, not what should stay.

ohwx woman, three-quarter view, grey background, neutral expression
ohwx woman, smiling, outdoors, blue jacket

The token ohwx is constant, so the model attaches the invariant part (the face) to it. Everything you do caption becomes separable and controllable afterwards. If you caption the jacket, you can change the jacket later. If you do not, the jacket becomes part of the character.

This is the single most useful thing to understand about captioning, and it explains a whole class of “my LoRA always puts her in the same shirt” complaints.

Style LoRAs differ

For a style rather than a character: 30 to 100 images, lower rank (8 to 16), fewer steps, and captions that describe content thoroughly while never naming the style. The logic is the same: the style is the invariant, so leave it uncaptioned and it attaches to the trigger token.

The honest assessment

A character LoRA takes about two hours end to end the first time: 30 minutes assembling and captioning, 30 training, 30 testing epochs, 30 tuning strength in real prompts. After a few, it is under an hour.

It gives you an identity that holds across dozens of images, works with any prompt, composes with controls, and is a file you own. For a campaign, a recurring character, or anything you will return to, it repays the hour many times.

It is the wrong answer for one image, and it is the wrong answer when you have not first checked that an adapter is good enough. And it has a specific failure worth knowing: a LoRA trained on 20 images of one face in similar conditions will reproduce those conditions, so the identity arrives carrying the lighting it was trained under. That is what the next lesson is about, and it is the difference between a LoRA that works and one that does not.