Models that can see

The architecture, in one paragraph

A vision-language model is three pieces glued together. A vision encoder, usually a SigLIP or CLIP-style transformer, turns an image into a grid of patch embeddings. A projector, often just a two-layer network, maps those into the language model’s embedding space. The language model then treats them as if they were tokens, sitting in the sequence alongside the text.

That is the whole trick. Once the projector is trained, the language model attends to image patches exactly as it attends to words, which is why vision capability can be added to an existing model without rebuilding it.

Images cost tokens, and more than you expect

Because patches become tokens, images consume context. The arithmetic varies by model, but the shape is consistent: an image at moderate resolution is several hundred tokens, and a high-resolution image processed with tiling can be well over a thousand.

Models that support dynamic resolution (Qwen-VL’s approach, for instance) split a large image into tiles and process each, so a full-page scan can cost as much as several pages of text. Two consequences:

• Context fills fast. Six document photographs can consume most of an 8k window before the question is asked.
• Cost is not intuitive. On a rented endpoint, an image-heavy request can be more expensive than a long text one, and the price list only mentions tokens.

If a model exposes a resolution or tile limit, it is a real dial. Halving the resolution on a task that does not need fine detail can quarter the token cost with no measurable quality change, and you should test that rather than assume either way.

The families

Qwen-VL has the widest size range and the strongest document and OCR results among the open models. Gemma 3 adds vision from 4B upwards, which makes it the smallest useful open vision model for a laptop. Llama 3.2 Vision covers 11B and 90B. InternVL is a strong research line with many sizes. Pixtral comes from Mistral. Molmo, from AI2, is notable for releasing its training data as well as its weights, which makes it the family to reach for if you need to defend how the model was built.

Serving support lags behind releases here, more than for text models. A new vision model may work in transformers weeks before llama.cpp or your preferred server supports it, and half-supported vision in a serving stack produces confusing results rather than clean errors. Check support before you plan around a specific model.

What they are good at, and where they fail

Good, and useful: describing an image, answering questions about a photograph, reading a screenshot, extracting fields from a reasonably clean document, transcribing handwriting at moderate quality, and describing a chart’s overall shape.

The failures are consistent across the family and worth knowing before you build:

• Counting. Ask how many people are in a photograph and expect errors above about five. This is a known and persistent weakness.
• Precise spatial relations. Left of, above, third from the right: unreliable.
• Dense document layout. A table with merged cells, a multi-column form, a scan at an angle. The model produces plausible, well-formatted, wrong output, which is worse than an error.
• Small text. If a human squints, the model fails, and it will not tell you it failed.
• Reading exact numbers from charts. It reads the trend, not the values.

For document work specifically, a dedicated OCR pipeline (Tesseract, PaddleOCR, or a specialised document model) followed by a text model is often more accurate and far cheaper than a vision-language model doing both jobs. Test the boring pipeline before assuming the impressive one wins.

Prompting a vision model

Two habits that measurably help. Ask for the evidence: quote the line that contains the invoice number rather than what is the invoice number, which makes the failure visible when it happens. And ask for a schema, so a missing field comes back as null rather than as an invented value.

Audio input is following the same architectural path (an audio encoder, a projector, the same language model) and the same warning applies: capability arrives in the weights well before it arrives in the serving stacks most people use.