Images and audio, as more tokens in the same stream

The same window, different contents

A vision-language model does not have a separate visual system. An image is processed by an encoder into a sequence of vectors, those vectors are projected into the same width as the text embeddings, and they are inserted into the context as if they were tokens. Attention then operates over text and image positions together, with no distinction.

That single architectural fact explains most of what these models do well and most of what they do badly.

How an image becomes tokens

The image is cut into fixed patches: 14 by 14 or 16 by 16 pixels is typical. Each patch becomes one vector. A 336 by 336 pixel image at 14-pixel patches gives 24 by 24, or 576 patches. Larger images are handled by tiling: split into several crops, encode each, and include a downscaled view of the whole for global context.

The count matters for your bill. A high-resolution image can consume one to three thousand tokens, which is comparable to several pages of text. Providers publish the formula; work it out for your typical image before designing a product around sending many of them.

What follows from patches

Fine detail below the patch size is lost. Small text, a thin line on a chart, a serial number in a corner. The information was averaged into a patch vector at the start and no later layer can recover it. Cropping and enlarging the region before sending is the fix, and it works because it changes what reaches the encoder.

Counting is unreliable. There is no counter, exactly as there is none for words. Counting twelve objects requires an operation the architecture does not have, and models approximate it from the overall pattern. Expect errors, especially above five or six items.

Precise spatial relations are approximate. “Which of these two labels is further left” is often right; “how many millimetres apart” is not. Position information survives in the patch ordering, but it is coarse.

Aspect ratio and resizing distort. An image squashed to a square before encoding changes proportions the model then reports. Check what your client library does before sending.

What it is good at

Description, classification, reading clear printed text, explaining a diagram, transcribing a moderately tidy table, identifying what is happening in a photograph, and answering questions that depend on the gist rather than the pixel. For document understanding on clean scans it is often better than traditional OCR pipelines because it uses context to resolve ambiguity, though it will also confidently invent a plausible value where OCR would have returned nothing, which is a different and more dangerous failure.

That contrast is the practical rule. Traditional OCR fails visibly. A model fails fluently. Where a wrong number is costly, use both and compare.

Audio, briefly

The same pattern. Audio is converted to a spectrogram, encoded into vectors, and inserted into the context. Speech-to-text models such as Whisper (free, open weights, runs on a laptop) are trained specifically for transcription. Newer audio-native models take the sound directly, which preserves tone and timing that a transcript discards.

The same limits apply, in audio form: precise timings are approximate, overlapping speakers confuse it, and it will produce fluent text for a passage that was actually inaudible.

What has not changed

Everything from the earlier modules still holds. Image tokens occupy the context window and compete with text. Position effects apply. The model does not “look again” at the image while generating: the encoding happened once, before the first output token, and generation attends to those fixed vectors.

That last point catches people out. Asking a follow-up question about an image does not cause a second, more careful look. It re-attends to the same vectors produced by the same encoder pass. If the detail was lost at encoding, asking more insistently will not retrieve it, and the confident answer you get is a plausible continuation rather than an observation.