{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"images-and-audio-as-tokens","title":"Images and audio, as more tokens in the same stream","url":"https://addaly.com/learn/how-llms-work/images-and-audio-as-tokens","position":38,"of":78,"minutes":8,"answer":"A multimodal model converts an image into a sequence of vectors placed in the same context as text, which explains both what it does well and its specific blindness to fine detail.","keyIdea":"A multimodal model converts an image into a sequence of vectors placed in the same context as text, which explains both what it does well and its specific blindness to fine detail."},"body":{"format":"markdown","text":"## The same window, different contents\n\nA vision-language model does not have a separate visual system. An image is processed by an encoder into a sequence of vectors, those vectors are projected into the same width as the text embeddings, and they are inserted into the context as if they were tokens. Attention then operates over text and image positions together, with no distinction.\n\nThat single architectural fact explains most of what these models do well and most of what they do badly.\n\n## How an image becomes tokens\n\nThe image is cut into fixed patches: 14 by 14 or 16 by 16 pixels is typical. Each patch becomes one vector. A 336 by 336 pixel image at 14-pixel patches gives 24 by 24, or 576 patches. Larger images are handled by tiling: split into several crops, encode each, and include a downscaled view of the whole for global context.\n\nThe count matters for your bill. A high-resolution image can consume one to three thousand tokens, which is comparable to several pages of text. Providers publish the formula; work it out for your typical image before designing a product around sending many of them.\n\n## What follows from patches\n\n**Fine detail below the patch size is lost.** Small text, a thin line on a chart, a serial number in a corner. The information was averaged into a patch vector at the start and no later layer can recover it. Cropping and enlarging the region before sending is the fix, and it works because it changes what reaches the encoder.\n\n**Counting is unreliable.** There is no counter, exactly as there is none for words. Counting twelve objects requires an operation the architecture does not have, and models approximate it from the overall pattern. Expect errors, especially above five or six items.\n\n**Precise spatial relations are approximate.** \"Which of these two labels is further left\" is often right; \"how many millimetres apart\" is not. Position information survives in the patch ordering, but it is coarse.\n\n**Aspect ratio and resizing distort.** An image squashed to a square before encoding changes proportions the model then reports. Check what your client library does before sending.\n\n## What it is good at\n\nDescription, classification, reading clear printed text, explaining a diagram, transcribing a moderately tidy table, identifying what is happening in a photograph, and answering questions that depend on the gist rather than the pixel. For document understanding on clean scans it is often better than traditional OCR pipelines because it uses context to resolve ambiguity, though it will also confidently *invent* a plausible value where OCR would have returned nothing, which is a different and more dangerous failure.\n\nThat contrast is the practical rule. Traditional OCR fails visibly. A model fails fluently. Where a wrong number is costly, use both and compare.\n\n## Audio, briefly\n\nThe same pattern. Audio is converted to a spectrogram, encoded into vectors, and inserted into the context. Speech-to-text models such as Whisper (free, open weights, runs on a laptop) are trained specifically for transcription. Newer audio-native models take the sound directly, which preserves tone and timing that a transcript discards.\n\nThe same limits apply, in audio form: precise timings are approximate, overlapping speakers confuse it, and it will produce fluent text for a passage that was actually inaudible.\n\n## What has not changed\n\nEverything from the earlier modules still holds. Image tokens occupy the context window and compete with text. Position effects apply. The model does not \"look again\" at the image while generating: the encoding happened once, before the first output token, and generation attends to those fixed vectors.\n\nThat last point catches people out. Asking a follow-up question about an image does not cause a second, more careful look. It re-attends to the same vectors produced by the same encoder pass. If the detail was lost at encoding, asking more insistently will not retrieve it, and the confident answer you get is a plausible continuation rather than an observation."},"check":{"question":"A model misreads a five-digit code in a photograph of a delivery label, and reads it differently when asked again. What is the mechanism, and what fixes it?","options":[{"text":"The model samples differently each time, so lowering the temperature will stabilise the reading.","correct":false,"tempting":"Temperature does control output variation, and a stable wrong answer would at least be consistent, which makes this feel like progress."},{"text":"The image was compressed in transit, losing detail that a lossless format would preserve.","correct":false,"tempting":"Compression artefacts genuinely do damage small text, so format choice is a real consideration in document pipelines."},{"text":"The digits occupy fewer pixels than a patch can resolve, so the detail was lost at encoding; cropping and enlarging that region before sending is what recovers it.","correct":true},{"text":"The model lacks OCR training and needs a specialised document model.","correct":false,"tempting":"Specialised document models do perform better on dense text, which makes a model-choice answer sound like the professional one."}]},"prev":{"slug":"the-system-prompt-is-just-tokens","title":"Why there is no boundary between instructions and data","url":"https://addaly.com/learn/how-llms-work/the-system-prompt-is-just-tokens"},"next":{"slug":"pretraining-finetuning-rlhf","title":"Pretraining, fine-tuning, and preference training","url":"https://addaly.com/learn/how-llms-work/pretraining-finetuning-rlhf"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Images and audio, as more tokens in the same stream', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/images-and-audio-as-tokens (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Images and audio, as more tokens in the same stream","courseTitle":"How a Language Model Actually Works","module":{"index":4,"slug":"the-context-window","title":"What the model can see"},"answer":null,"keyIdea":"A multimodal model converts an image into a sequence of vectors placed in the same context as text, which explains both what it does well and its specific blindness to fine detail.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-same-window-different-contents","runs":[{"text":"The same window, different contents"}]},{"t":"p","runs":[{"text":"A vision-language model does not have a separate visual system. An image is processed by an encoder into a sequence of vectors, those vectors are projected into the same width as the text embeddings, and they are inserted into the context as if they were tokens. Attention then operates over text and image positions together, with no distinction."}]},{"t":"p","runs":[{"text":"That single architectural fact explains most of what these models do well and most of what they do badly."}]},{"t":"h2","id":"how-an-image-becomes-tokens","runs":[{"text":"How an image becomes tokens"}]},{"t":"p","runs":[{"text":"The image is cut into fixed patches: 14 by 14 or 16 by 16 pixels is typical. Each patch becomes one vector. A 336 by 336 pixel image at 14-pixel patches gives 24 by 24, or 576 patches. Larger images are handled by tiling: split into several crops, encode each, and include a downscaled view of the whole for global context."}]},{"t":"p","runs":[{"text":"The count matters for your bill. A high-resolution image can consume one to three thousand tokens, which is comparable to several pages of text. Providers publish the formula; work it out for your typical image before designing a product around sending many of them."}]},{"t":"h2","id":"what-follows-from-patches","runs":[{"text":"What follows from patches"}]},{"t":"p","runs":[{"text":"Fine detail below the patch size is lost.","b":true},{"text":" Small text, a thin line on a chart, a serial number in a corner. The information was averaged into a patch vector at the start and no later layer can recover it. Cropping and enlarging the region before sending is the fix, and it works because it changes what reaches the encoder."}]},{"t":"p","runs":[{"text":"Counting is unreliable.","b":true},{"text":" There is no counter, exactly as there is none for words. Counting twelve objects requires an operation the architecture does not have, and models approximate it from the overall pattern. Expect errors, especially above five or six items."}]},{"t":"p","runs":[{"text":"Precise spatial relations are approximate.","b":true},{"text":" “Which of these two labels is further left” is often right; “how many millimetres apart” is not. Position information survives in the patch ordering, but it is coarse."}]},{"t":"p","runs":[{"text":"Aspect ratio and resizing distort.","b":true},{"text":" An image squashed to a square before encoding changes proportions the model then reports. Check what your client library does before sending."}]},{"t":"h2","id":"what-it-is-good-at","runs":[{"text":"What it is good at"}]},{"t":"p","runs":[{"text":"Description, classification, reading clear printed text, explaining a diagram, transcribing a moderately tidy table, identifying what is happening in a photograph, and answering questions that depend on the gist rather than the pixel. For document understanding on clean scans it is often better than traditional OCR pipelines because it uses context to resolve ambiguity, though it will also confidently "},{"text":"invent","i":true},{"text":" a plausible value where OCR would have returned nothing, which is a different and more dangerous failure."}]},{"t":"p","runs":[{"text":"That contrast is the practical rule. Traditional OCR fails visibly. A model fails fluently. Where a wrong number is costly, use both and compare."}]},{"t":"h2","id":"audio-briefly","runs":[{"text":"Audio, briefly"}]},{"t":"p","runs":[{"text":"The same pattern. Audio is converted to a spectrogram, encoded into vectors, and inserted into the context. Speech-to-text models such as Whisper (free, open weights, runs on a laptop) are trained specifically for transcription. Newer audio-native models take the sound directly, which preserves tone and timing that a transcript discards."}]},{"t":"p","runs":[{"text":"The same limits apply, in audio form: precise timings are approximate, overlapping speakers confuse it, and it will produce fluent text for a passage that was actually inaudible."}]},{"t":"h2","id":"what-has-not-changed","runs":[{"text":"What has not changed"}]},{"t":"p","runs":[{"text":"Everything from the earlier modules still holds. Image tokens occupy the context window and compete with text. Position effects apply. The model does not “look again” at the image while generating: the encoding happened once, before the first output token, and generation attends to those fixed vectors."}]},{"t":"p","runs":[{"text":"That last point catches people out. Asking a follow-up question about an image does not cause a second, more careful look. It re-attends to the same vectors produced by the same encoder pass. If the detail was lost at encoding, asking more insistently will not retrieve it, and the confident answer you get is a plausible continuation rather than an observation."}]}],"check":{"question":[{"text":"A model misreads a five-digit code in a photograph of a delivery label, and reads it differently when asked again. What is the mechanism, and what fixes it?"}],"options":[{"text":[{"text":"The model samples differently each time, so lowering the temperature will stabilise the reading."}],"correct":false,"tempting":[{"text":"Temperature does control output variation, and a stable wrong answer would at least be consistent, which makes this feel like progress."}]},{"text":[{"text":"The image was compressed in transit, losing detail that a lossless format would preserve."}],"correct":false,"tempting":[{"text":"Compression artefacts genuinely do damage small text, so format choice is a real consideration in document pipelines."}]},{"text":[{"text":"The digits occupy fewer pixels than a patch can resolve, so the detail was lost at encoding; cropping and enlarging that region before sending is what recovers it."}],"correct":true,"tempting":null},{"text":[{"text":"The model lacks OCR training and needs a specialised document model."}],"correct":false,"tempting":[{"text":"Specialised document models do perform better on dense text, which makes a model-choice answer sound like the professional one."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"the-system-prompt-is-just-tokens","title":"Why there is no boundary between instructions and data","minutes":9},"next":{"slug":"pretraining-finetuning-rlhf","title":"Pretraining, fine-tuning, and preference training","minutes":8}}}