{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"multimodal-open-models","title":"Models that can see","url":"https://addaly.com/learn/open-models/multimodal-open-models","position":28,"of":71,"minutes":9,"answer":"A vision-language model turns image patches into tokens through a projector, so images consume context and cost proportional to resolution, and the reliable failures are counting, dense layout and small text rather than description.","keyIdea":"A vision-language model turns image patches into tokens through a projector, so images consume context and cost proportional to resolution, and the reliable failures are counting, dense layout and small text rather than description."},"body":{"format":"markdown","text":"## The architecture, in one paragraph\n\nA vision-language model is three pieces glued together. A **vision encoder**, usually a SigLIP or CLIP-style transformer, turns an image into a grid of patch embeddings. A **projector**, often just a two-layer network, maps those into the language model's embedding space. The **language model** then treats them as if they were tokens, sitting in the sequence alongside the text.\n\nThat is the whole trick. Once the projector is trained, the language model attends to image patches exactly as it attends to words, which is why vision capability can be added to an existing model without rebuilding it.\n\n## Images cost tokens, and more than you expect\n\nBecause patches become tokens, images consume context. The arithmetic varies by model, but the shape is consistent: an image at moderate resolution is several hundred tokens, and a high-resolution image processed with tiling can be well over a thousand.\n\nModels that support dynamic resolution (Qwen-VL's approach, for instance) split a large image into tiles and process each, so a full-page scan can cost as much as several pages of text. Two consequences:\n\n- **Context fills fast.** Six document photographs can consume most of an 8k window before the question is asked.\n- **Cost is not intuitive.** On a rented endpoint, an image-heavy request can be more expensive than a long text one, and the price list only mentions tokens.\n\nIf a model exposes a resolution or tile limit, it is a real dial. Halving the resolution on a task that does not need fine detail can quarter the token cost with no measurable quality change, and you should test that rather than assume either way.\n\n## The families\n\n**Qwen-VL** has the widest size range and the strongest document and OCR results among the open models. **Gemma 3** adds vision from 4B upwards, which makes it the smallest useful open vision model for a laptop. **Llama 3.2 Vision** covers 11B and 90B. **InternVL** is a strong research line with many sizes. **Pixtral** comes from Mistral. **Molmo**, from AI2, is notable for releasing its training data as well as its weights, which makes it the family to reach for if you need to defend how the model was built.\n\nServing support lags behind releases here, more than for text models. A new vision model may work in `transformers` weeks before llama.cpp or your preferred server supports it, and half-supported vision in a serving stack produces confusing results rather than clean errors. Check support before you plan around a specific model.\n\n## What they are good at, and where they fail\n\nGood, and useful: describing an image, answering questions about a photograph, reading a screenshot, extracting fields from a reasonably clean document, transcribing handwriting at moderate quality, and describing a chart's overall shape.\n\nThe failures are consistent across the family and worth knowing before you build:\n\n- **Counting.** Ask how many people are in a photograph and expect errors above about five. This is a known and persistent weakness.\n- **Precise spatial relations.** Left of, above, third from the right: unreliable.\n- **Dense document layout.** A table with merged cells, a multi-column form, a scan at an angle. The model produces plausible, well-formatted, wrong output, which is worse than an error.\n- **Small text.** If a human squints, the model fails, and it will not tell you it failed.\n- **Reading exact numbers from charts.** It reads the trend, not the values.\n\nFor document work specifically, a dedicated OCR pipeline (Tesseract, PaddleOCR, or a specialised document model) followed by a text model is often more accurate and far cheaper than a vision-language model doing both jobs. Test the boring pipeline before assuming the impressive one wins.\n\n## Prompting a vision model\n\nTwo habits that measurably help. Ask for the evidence: *quote the line that contains the invoice number* rather than *what is the invoice number*, which makes the failure visible when it happens. And ask for a schema, so a missing field comes back as null rather than as an invented value.\n\n> Audio input is following the same architectural path (an audio encoder, a projector, the same language model) and the same warning applies: capability arrives in the weights well before it arrives in the serving stacks most people use."},"check":{"question":"A team extracts fields from photographed invoices with a vision-language model. Accuracy is high on clean invoices and poor on dense multi-column ones, with confident wrong values rather than errors. What is the best next step?","options":[{"text":"Increase the context window, since dense documents need more room for the model to reason.","correct":false,"tempting":"Context is not the binding constraint here; the model is misreading layout, not running out of space."},{"text":"Lower the temperature to zero, which will stop the model inventing values.","correct":false,"tempting":"Greedy decoding removes randomness but the model's most likely reading of an ambiguous layout is still wrong."},{"text":"Compare against a dedicated OCR step feeding a text model, and require the model to quote the source line for each field so wrong reads become visible.","correct":true},{"text":"Switch to a larger vision model, since dense layout understanding scales with parameters.","correct":false,"tempting":"Worth testing, but layout parsing is a known category weakness across sizes and the confident-wrong behaviour persists."}]},"prev":{"slug":"mixture-of-experts","title":"Mixture of experts, and the memory it still needs","url":"https://addaly.com/learn/open-models/mixture-of-experts"},"next":{"slug":"weight-formats-and-conversion","title":"Formats, and what conversion costs you","url":"https://addaly.com/learn/open-models/weight-formats-and-conversion"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Models that can see', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/multimodal-open-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Models that can see","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"A vision-language model turns image patches into tokens through a projector, so images consume context and cost proportional to resolution, and the reliable failures are counting, dense layout and small text rather than description.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-architecture-in-one-paragraph","runs":[{"text":"The architecture, in one paragraph"}]},{"t":"p","runs":[{"text":"A vision-language model is three pieces glued together. A "},{"text":"vision encoder","b":true},{"text":", usually a SigLIP or CLIP-style transformer, turns an image into a grid of patch embeddings. A "},{"text":"projector","b":true},{"text":", often just a two-layer network, maps those into the language model’s embedding space. The "},{"text":"language model","b":true},{"text":" then treats them as if they were tokens, sitting in the sequence alongside the text."}]},{"t":"p","runs":[{"text":"That is the whole trick. Once the projector is trained, the language model attends to image patches exactly as it attends to words, which is why vision capability can be added to an existing model without rebuilding it."}]},{"t":"h2","id":"images-cost-tokens-and-more-than-you-expect","runs":[{"text":"Images cost tokens, and more than you expect"}]},{"t":"p","runs":[{"text":"Because patches become tokens, images consume context. The arithmetic varies by model, but the shape is consistent: an image at moderate resolution is several hundred tokens, and a high-resolution image processed with tiling can be well over a thousand."}]},{"t":"p","runs":[{"text":"Models that support dynamic resolution (Qwen-VL’s approach, for instance) split a large image into tiles and process each, so a full-page scan can cost as much as several pages of text. Two consequences:"}]},{"t":"ul","items":[[{"text":"Context fills fast.","b":true},{"text":" Six document photographs can consume most of an 8k window before the question is asked."}],[{"text":"Cost is not intuitive.","b":true},{"text":" On a rented endpoint, an image-heavy request can be more expensive than a long text one, and the price list only mentions tokens."}]]},{"t":"p","runs":[{"text":"If a model exposes a resolution or tile limit, it is a real dial. Halving the resolution on a task that does not need fine detail can quarter the token cost with no measurable quality change, and you should test that rather than assume either way."}]},{"t":"h2","id":"the-families","runs":[{"text":"The families"}]},{"t":"p","runs":[{"text":"Qwen-VL","b":true},{"text":" has the widest size range and the strongest document and OCR results among the open models. "},{"text":"Gemma 3","b":true},{"text":" adds vision from 4B upwards, which makes it the smallest useful open vision model for a laptop. "},{"text":"Llama 3.2 Vision","b":true},{"text":" covers 11B and 90B. "},{"text":"InternVL","b":true},{"text":" is a strong research line with many sizes. "},{"text":"Pixtral","b":true},{"text":" comes from Mistral. "},{"text":"Molmo","b":true},{"text":", from AI2, is notable for releasing its training data as well as its weights, which makes it the family to reach for if you need to defend how the model was built."}]},{"t":"p","runs":[{"text":"Serving support lags behind releases here, more than for text models. A new vision model may work in "},{"text":"transformers","code":true},{"text":" weeks before llama.cpp or your preferred server supports it, and half-supported vision in a serving stack produces confusing results rather than clean errors. Check support before you plan around a specific model."}]},{"t":"h2","id":"what-they-are-good-at-and-where-they-fail","runs":[{"text":"What they are good at, and where they fail"}]},{"t":"p","runs":[{"text":"Good, and useful: describing an image, answering questions about a photograph, reading a screenshot, extracting fields from a reasonably clean document, transcribing handwriting at moderate quality, and describing a chart’s overall shape."}]},{"t":"p","runs":[{"text":"The failures are consistent across the family and worth knowing before you build:"}]},{"t":"ul","items":[[{"text":"Counting.","b":true},{"text":" Ask how many people are in a photograph and expect errors above about five. This is a known and persistent weakness."}],[{"text":"Precise spatial relations.","b":true},{"text":" Left of, above, third from the right: unreliable."}],[{"text":"Dense document layout.","b":true},{"text":" A table with merged cells, a multi-column form, a scan at an angle. The model produces plausible, well-formatted, wrong output, which is worse than an error."}],[{"text":"Small text.","b":true},{"text":" If a human squints, the model fails, and it will not tell you it failed."}],[{"text":"Reading exact numbers from charts.","b":true},{"text":" It reads the trend, not the values."}]]},{"t":"p","runs":[{"text":"For document work specifically, a dedicated OCR pipeline (Tesseract, PaddleOCR, or a specialised document model) followed by a text model is often more accurate and far cheaper than a vision-language model doing both jobs. Test the boring pipeline before assuming the impressive one wins."}]},{"t":"h2","id":"prompting-a-vision-model","runs":[{"text":"Prompting a vision model"}]},{"t":"p","runs":[{"text":"Two habits that measurably help. Ask for the evidence: "},{"text":"quote the line that contains the invoice number","i":true},{"text":" rather than "},{"text":"what is the invoice number","i":true},{"text":", which makes the failure visible when it happens. And ask for a schema, so a missing field comes back as null rather than as an invented value."}]},{"t":"quote","runs":[{"text":"Audio input is following the same architectural path (an audio encoder, a projector, the same language model) and the same warning applies: capability arrives in the weights well before it arrives in the serving stacks most people use."}]}],"check":{"question":[{"text":"A team extracts fields from photographed invoices with a vision-language model. Accuracy is high on clean invoices and poor on dense multi-column ones, with confident wrong values rather than errors. What is the best next step?"}],"options":[{"text":[{"text":"Increase the context window, since dense documents need more room for the model to reason."}],"correct":false,"tempting":[{"text":"Context is not the binding constraint here; the model is misreading layout, not running out of space."}]},{"text":[{"text":"Lower the temperature to zero, which will stop the model inventing values."}],"correct":false,"tempting":[{"text":"Greedy decoding removes randomness but the model’s most likely reading of an ambiguous layout is still wrong."}]},{"text":[{"text":"Compare against a dedicated OCR step feeding a text model, and require the model to quote the source line for each field so wrong reads become visible."}],"correct":true,"tempting":null},{"text":[{"text":"Switch to a larger vision model, since dense layout understanding scales with parameters."}],"correct":false,"tempting":[{"text":"Worth testing, but layout parsing is a known category weakness across sizes and the confident-wrong behaviour persists."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"mixture-of-experts","title":"Mixture of experts, and the memory it still needs","minutes":8},"next":{"slug":"weight-formats-and-conversion","title":"Formats, and what conversion costs you","minutes":8}}}