{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"vision-language-models","title":"Fine-tuning a vision-language model","url":"https://addaly.com/learn/fine-tuning/vision-language-models","position":62,"of":84,"minutes":9,"answer":"Freeze the vision encoder, train the projector and LoRA the language model, and check how many sequence positions one image actually consumes before planning memory. And for printed documents.","keyIdea":"Freeze the vision encoder, train the projector and LoRA the language model, and check how many sequence positions one image actually consumes before planning memory. And for printed documents, an OCR pipeline into a text model is usually cheaper, more accurate and far easier to debug."},"body":{"format":"markdown","text":"## Three parts, and you train one of them\n\nA vision-language model has a vision encoder that turns an image into a set of embeddings, a projector that maps those into the language model's embedding space, and the language model itself.\n\nThe standard fine-tuning arrangement:\n\n- **Freeze the vision encoder.** It knows what images look like, it was trained on far more images than you have, and unfreezing it is the fastest way to damage the model.\n- **Train the projector**, fully. It is small, and it is the piece that aligns the two spaces for your kind of image.\n- **LoRA the language model.** Attention and feed-forward projections, as usual.\n\n```python\nfor p in model.vision_tower.parameters():\n    p.requires_grad = False\n```\n\nThat one loop is most of the difference between a run that works and one that does not.\n\n## Images cost tokens, and it is more than you think\n\nAn image does not occupy one position. It becomes a block of embeddings in the language model's sequence, and the count depends on resolution and on whether the model tiles large images.\n\nDepending on the architecture and resolution, a single image can occupy anywhere from about 64 to well over 1,000 positions, and models that split a high-resolution image into tiles multiply that by the tile count. A document page at full resolution can consume several thousand tokens on its own.\n\nThe consequences are immediate:\n\n- Activation memory scales with sequence length, so a batch of four images at high resolution can exceed what a batch of thirty-two text examples needs.\n- Your effective batch size will be small: one or two per device, with heavy gradient accumulation.\n- Lowering the input resolution is the most effective memory lever available, and it costs exactly the detail you lowered. For photographs of products this is usually free; for reading small print in a scanned document it is not.\n\nCheck the number before you plan the run: encode one representative image and print how many positions it produced.\n\n## The data\n\nThe messages format extends to images:\n\n```json\n{\"messages\": [\n  {\"role\": \"user\", \"content\": [\n     {\"type\": \"image\"},\n     {\"type\": \"text\", \"text\": \"Which shelf is the damaged item on?\"}]},\n  {\"role\": \"assistant\", \"content\": \"The second shelf from the top, on the left.\"}\n]}\n```\n\nThe image itself is passed alongside via the processor. Masking works exactly as in text: loss on the assistant turn only, and the image positions are input.\n\nVolume is modest for a specific task: 500 to 3,000 image-and-answer pairs changes behaviour markedly, because you are adapting an already-aligned model rather than teaching it to see.\n\n## The free path\n\nSmolVLM comes in sizes from a few hundred million parameters up, is small enough to fine-tune on a free T4, and is permissively licensed. Qwen's VL family and Llama's vision models are larger and stronger; both have LoRA recipes that fit on a single rented card.\n\nFor inference afterwards, `llama.cpp` supports several vision architectures on CPU, so the final artefact runs on a laptop.\n\n## The question to ask before any of it\n\nFor document work specifically (invoices, forms, scanned reports) a vision-language model is frequently the wrong tool, and it is the most common misapplication in this area.\n\nThe alternative is a pipeline: a dedicated OCR engine extracts text with layout, and a text model reads the result. Tesseract, PaddleOCR and the docling family are free. The pipeline is cheaper per page by a wide margin, its failures are inspectable (you can look at the OCR output and see exactly what went wrong) and on printed documents its accuracy is typically better, because OCR engines are specialists.\n\nThe vision model earns its cost when layout itself carries meaning that survives no text extraction: handwriting, photographs, charts to be interpreted, screenshots of interfaces, damage assessment.\n\nRun both on fifty pages before committing. The comparison takes an afternoon and it decides an architecture."},"check":{"question":"A team fine-tunes a vision-language model on 2,000 product photos and unfreezes everything including the vision encoder. Image understanding degrades noticeably on images unlike their training set. Why?","options":[{"text":"The projector cannot adapt while the encoder is also moving, so the two components fail to stay aligned.","correct":false,"tempting":"Points at an alignment dynamic that is real but secondary; the damage persists even where the projector tracks the encoder successfully."},{"text":"Unfreezing the encoder increases memory so much that the batch size collapsed, and small batches caused the instability.","correct":false,"tempting":"Names a genuine consequence of unfreezing, but batch size would degrade training broadly rather than specifically on unlike images."},{"text":"Vision encoders require a different learning rate from language models, and using one rate for both saturates the encoder's weights.","correct":false,"tempting":"Raises a tuning detail that mitigates the problem at best; a lower rate slows the overwriting rather than preventing it."},{"text":"The encoder's general visual representations were overwritten by gradients from a narrow set.","correct":true}]},"prev":{"slug":"speech-models-and-accents","title":"Fine-tuning a speech model","url":"https://addaly.com/learn/fine-tuning/speech-models-and-accents"},"next":{"slug":"diffusion-lora-and-likeness","title":"LoRA for image models, and the likeness question","url":"https://addaly.com/learn/fine-tuning/diffusion-lora-and-likeness"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Fine-tuning a vision-language model', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/vision-language-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Fine-tuning a vision-language model","courseTitle":"Fine-Tuning, and When Not To","module":{"index":7,"slug":"beyond-text-generation","title":"Fine-tuning beyond a chat model"},"answer":null,"keyIdea":"Freeze the vision encoder, train the projector and LoRA the language model, and check how many sequence positions one image actually consumes before planning memory. And for printed documents, an OCR pipeline into a text model is usually cheaper, more accurate and far easier to debug.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-parts-and-you-train-one-of-them","runs":[{"text":"Three parts, and you train one of them"}]},{"t":"p","runs":[{"text":"A vision-language model has a vision encoder that turns an image into a set of embeddings, a projector that maps those into the language model’s embedding space, and the language model itself."}]},{"t":"p","runs":[{"text":"The standard fine-tuning arrangement:"}]},{"t":"ul","items":[[{"text":"Freeze the vision encoder.","b":true},{"text":" It knows what images look like, it was trained on far more images than you have, and unfreezing it is the fastest way to damage the model."}],[{"text":"Train the projector","b":true},{"text":", fully. It is small, and it is the piece that aligns the two spaces for your kind of image."}],[{"text":"LoRA the language model.","b":true},{"text":" Attention and feed-forward projections, as usual."}]]},{"t":"code","lang":"python","label":"Python","text":"for p in model.vision_tower.parameters():\n    p.requires_grad = False"},{"t":"p","runs":[{"text":"That one loop is most of the difference between a run that works and one that does not."}]},{"t":"h2","id":"images-cost-tokens-and-it-is-more-than-you-think","runs":[{"text":"Images cost tokens, and it is more than you think"}]},{"t":"p","runs":[{"text":"An image does not occupy one position. It becomes a block of embeddings in the language model’s sequence, and the count depends on resolution and on whether the model tiles large images."}]},{"t":"p","runs":[{"text":"Depending on the architecture and resolution, a single image can occupy anywhere from about 64 to well over 1,000 positions, and models that split a high-resolution image into tiles multiply that by the tile count. A document page at full resolution can consume several thousand tokens on its own."}]},{"t":"p","runs":[{"text":"The consequences are immediate:"}]},{"t":"ul","items":[[{"text":"Activation memory scales with sequence length, so a batch of four images at high resolution can exceed what a batch of thirty-two text examples needs."}],[{"text":"Your effective batch size will be small: one or two per device, with heavy gradient accumulation."}],[{"text":"Lowering the input resolution is the most effective memory lever available, and it costs exactly the detail you lowered. For photographs of products this is usually free; for reading small print in a scanned document it is not."}]]},{"t":"p","runs":[{"text":"Check the number before you plan the run: encode one representative image and print how many positions it produced."}]},{"t":"h2","id":"the-data","runs":[{"text":"The data"}]},{"t":"p","runs":[{"text":"The messages format extends to images:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"messages\": [\n  {\"role\": \"user\", \"content\": [\n     {\"type\": \"image\"},\n     {\"type\": \"text\", \"text\": \"Which shelf is the damaged item on?\"}]},\n  {\"role\": \"assistant\", \"content\": \"The second shelf from the top, on the left.\"}\n]}"},{"t":"p","runs":[{"text":"The image itself is passed alongside via the processor. Masking works exactly as in text: loss on the assistant turn only, and the image positions are input."}]},{"t":"p","runs":[{"text":"Volume is modest for a specific task: 500 to 3,000 image-and-answer pairs changes behaviour markedly, because you are adapting an already-aligned model rather than teaching it to see."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"SmolVLM comes in sizes from a few hundred million parameters up, is small enough to fine-tune on a free T4, and is permissively licensed. Qwen’s VL family and Llama’s vision models are larger and stronger; both have LoRA recipes that fit on a single rented card."}]},{"t":"p","runs":[{"text":"For inference afterwards, "},{"text":"llama.cpp","code":true},{"text":" supports several vision architectures on CPU, so the final artefact runs on a laptop."}]},{"t":"h2","id":"the-question-to-ask-before-any-of-it","runs":[{"text":"The question to ask before any of it"}]},{"t":"p","runs":[{"text":"For document work specifically (invoices, forms, scanned reports) a vision-language model is frequently the wrong tool, and it is the most common misapplication in this area."}]},{"t":"p","runs":[{"text":"The alternative is a pipeline: a dedicated OCR engine extracts text with layout, and a text model reads the result. Tesseract, PaddleOCR and the docling family are free. The pipeline is cheaper per page by a wide margin, its failures are inspectable (you can look at the OCR output and see exactly what went wrong) and on printed documents its accuracy is typically better, because OCR engines are specialists."}]},{"t":"p","runs":[{"text":"The vision model earns its cost when layout itself carries meaning that survives no text extraction: handwriting, photographs, charts to be interpreted, screenshots of interfaces, damage assessment."}]},{"t":"p","runs":[{"text":"Run both on fifty pages before committing. The comparison takes an afternoon and it decides an architecture."}]}],"check":{"question":[{"text":"A team fine-tunes a vision-language model on 2,000 product photos and unfreezes everything including the vision encoder. Image understanding degrades noticeably on images unlike their training set. Why?"}],"options":[{"text":[{"text":"The projector cannot adapt while the encoder is also moving, so the two components fail to stay aligned."}],"correct":false,"tempting":[{"text":"Points at an alignment dynamic that is real but secondary; the damage persists even where the projector tracks the encoder successfully."}]},{"text":[{"text":"Unfreezing the encoder increases memory so much that the batch size collapsed, and small batches caused the instability."}],"correct":false,"tempting":[{"text":"Names a genuine consequence of unfreezing, but batch size would degrade training broadly rather than specifically on unlike images."}]},{"text":[{"text":"Vision encoders require a different learning rate from language models, and using one rate for both saturates the encoder’s weights."}],"correct":false,"tempting":[{"text":"Raises a tuning detail that mitigates the problem at best; a lower rate slows the overwriting rather than preventing it."}]},{"text":[{"text":"The encoder’s general visual representations were overwritten by gradients from a narrow set."}],"correct":true,"tempting":null}],"order":[0,3,1,2],"why":null},"prev":{"slug":"speech-models-and-accents","title":"Fine-tuning a speech model","minutes":9},"next":{"slug":"diffusion-lora-and-likeness","title":"LoRA for image models, and the likeness question","minutes":10}}}