Fine-tuning a vision-language model

Three parts, and you train one of them

A vision-language model has a vision encoder that turns an image into a set of embeddings, a projector that maps those into the language model’s embedding space, and the language model itself.

The standard fine-tuning arrangement:

• Freeze the vision encoder. It knows what images look like, it was trained on far more images than you have, and unfreezing it is the fastest way to damage the model.
• Train the projector, fully. It is small, and it is the piece that aligns the two spaces for your kind of image.
• LoRA the language model. Attention and feed-forward projections, as usual.

for p in model.vision_tower.parameters():
    p.requires_grad = False

That one loop is most of the difference between a run that works and one that does not.

Images cost tokens, and it is more than you think

An image does not occupy one position. It becomes a block of embeddings in the language model’s sequence, and the count depends on resolution and on whether the model tiles large images.

Depending on the architecture and resolution, a single image can occupy anywhere from about 64 to well over 1,000 positions, and models that split a high-resolution image into tiles multiply that by the tile count. A document page at full resolution can consume several thousand tokens on its own.

The consequences are immediate:

• Activation memory scales with sequence length, so a batch of four images at high resolution can exceed what a batch of thirty-two text examples needs.
• Your effective batch size will be small: one or two per device, with heavy gradient accumulation.
• Lowering the input resolution is the most effective memory lever available, and it costs exactly the detail you lowered. For photographs of products this is usually free; for reading small print in a scanned document it is not.

Check the number before you plan the run: encode one representative image and print how many positions it produced.

The data

The messages format extends to images:

{"messages": [
  {"role": "user", "content": [
     {"type": "image"},
     {"type": "text", "text": "Which shelf is the damaged item on?"}]},
  {"role": "assistant", "content": "The second shelf from the top, on the left."}
]}

The image itself is passed alongside via the processor. Masking works exactly as in text: loss on the assistant turn only, and the image positions are input.

Volume is modest for a specific task: 500 to 3,000 image-and-answer pairs changes behaviour markedly, because you are adapting an already-aligned model rather than teaching it to see.

The free path

SmolVLM comes in sizes from a few hundred million parameters up, is small enough to fine-tune on a free T4, and is permissively licensed. Qwen’s VL family and Llama’s vision models are larger and stronger; both have LoRA recipes that fit on a single rented card.

For inference afterwards, llama.cpp supports several vision architectures on CPU, so the final artefact runs on a laptop.

The question to ask before any of it

For document work specifically (invoices, forms, scanned reports) a vision-language model is frequently the wrong tool, and it is the most common misapplication in this area.

The alternative is a pipeline: a dedicated OCR engine extracts text with layout, and a text model reads the result. Tesseract, PaddleOCR and the docling family are free. The pipeline is cheaper per page by a wide margin, its failures are inspectable (you can look at the OCR output and see exactly what went wrong) and on printed documents its accuracy is typically better, because OCR engines are specialists.

The vision model earns its cost when layout itself carries meaning that survives no text extraction: handwriting, photographs, charts to be interpreted, screenshots of interfaces, damage assessment.

Run both on fifty pages before committing. The comparison takes an afternoon and it decides an architecture.