{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"the-model-that-looks-at-pictures","title":"When the model looks at a picture","url":"https://addaly.com/learn/ai-actually-explained/the-model-that-looks-at-pictures","position":56,"of":92,"minutes":9,"answer":"A vision model cuts a photograph into a grid of patches and feeds them in as tokens beside the words, which is why it describes a scene fluently, costs hundreds of tokens per image.","keyIdea":"A vision model cuts a photograph into a grid of patches and feeds them in as tokens beside the words, which is why it describes a scene fluently, costs hundreds of tokens per image, and is reliably blind to counting, position and fine detail."},"body":{"format":"markdown","text":"## The other direction\n\nThe previous lessons made pictures from words. This one is about the reverse (a model that takes a photograph and answers questions about it) because it is now in every major chatbot, and because the way it works explains a set of failures that catch people out.\n\n## Patches as words\n\nRecall from the history module that in 2020 a Google team cut an image into a grid of small squares and fed the squares to a transformer as if they were words. That is still the mechanism. A photograph is divided into patches (sixteen pixels square is common) each patch is turned into a vector, and the vectors are placed into the model's context alongside the text tokens. From the model's point of view an image is a few hundred extra tokens that happen to have come from pixels. It attends across them the way it attends across a sentence.\n\nThat has a price you can see on an invoice. One large lab charges for images by tile: a fixed 85 tokens plus 170 for each 512-pixel square, so a photograph of 1024 by 1024 costs 765 tokens: about the same as a page and a half of text. Send ten photographs and you have spent the equivalent of a short chapter before asking anything.\n\n> Figure: What a photograph costs in the context window. Bars (tokens): Fixed overhead for any image 85; One 512-pixel tile 170; A page of English prose (about 500 words) 650; A 1024 by 1024 photograph 765. An image is cut into patches and enters as tokens beside the words. One large lab charges 85 tokens fixed plus 170 for each 512-pixel tile, so a 1024-square photograph is 765: a little more than a page of English prose. Ten photographs spend a short chapter before you have asked anything.\n\n## What it is good at\n\nDescribing. Given a photograph of a street, a modern model produces a paragraph that a sighted person would accept: the shops, the weather, the vehicle, the sign. It reads printed text in the image well, including handwriting, and it reads charts and diagrams well enough to answer questions about them. Since 2023 a free app used by blind and low-vision people has offered exactly this, and it is one of the least ambiguous benefits in this entire course.\n\nThe reason it is good at describing is the reason the chatbot is good at prose: the training pairs images with fluent captions, and fluent captions are what it learned to produce.\n\n## Where it is blind, and why\n\nThe failures cluster, and they cluster where patches-as-tokens breaks down.\n\n**Counting.** Ask how many people are in a crowd, or how many windows on a building, and the answer is wrong once the number passes four or five. Objects that span several patches, or share a patch, are not separate things to the model; they are regions of a texture, exactly as in the drawing lessons.\n\n**Position.** Left of, above, between. The patches carry position stamps, but the training captions rarely describe spatial relations precisely, so the model has thin support for them. A 2024 study gave the leading models tasks a child does easily (count how many times two lines cross, say whether two circles overlap) and found them near chance.\n\n**Small detail.** A patch that is sixteen pixels square holds one blurred vector. Fine print, a small dial, a mole on skin, the time on an analogue clock: anything that lives inside one patch is mostly lost. Analogue clock reading was a well-documented failure for years for this reason.\n\n**Anything that requires measuring.** Angles, lengths, whether a wall is straight. The model has no ruler, only the statistics of what captions say.\n\nEach of these is a mechanism, not a bug in this week's release, and each will improve as patches shrink and data improves, without being removed.\n\n## What this means for the medical photograph\n\nPeople send these models photographs of rashes, X-rays, moles and blood-test printouts, and the model answers fluently. Two things are true at once. It reads the printout accurately, because that is text. And its judgement of the rash is a caption-shaped guess: a plausible description of what such images tend to be captioned as, with no measurement, no history and no accountability. Systems that read medical images exist and are trained, validated and regulated separately, as the second module explained. The chatbot with vision is not one of them.\n\n## The free path\n\nOpen-weights vision models exist and run locally. The first, from a university group in 2023, showed the recipe could be followed at small scale; Alibaba's and Google's small multimodal models of 2024 and 2025 fit on a laptop, and a later module shows the tools that run them. The quality trails the closed models, but the mechanism, and the blind spots, are identical."},"check":{"question":"A vision model describes a crowded market scene accurately and then gives the wrong number when asked how many people are in it. Why?","options":[{"text":"The model was not trained on enough photographs of markets.","correct":false,"tempting":"Blames scarce data for a failure that appears in every kind of scene once the count passes a handful."},{"text":"Counting requires arithmetic, and language models cannot do arithmetic reliably beyond a few digits, so any count is a guess.","correct":false,"tempting":"Reaches for the arithmetic limit; the failure is upstream, in the model never having separate objects to count."},{"text":"The image was compressed before the model saw it, losing the people.","correct":false,"tempting":"Invents a preprocessing loss; the scene was described accurately from the same input."},{"text":"To the model the people are regions of texture, not separate objects, so a description is supported and a count is not.","correct":true}]},"prev":{"slug":"why-the-hands-were-wrong","title":"Why the hands were wrong","url":"https://addaly.com/learn/ai-actually-explained/why-the-hands-were-wrong"},"next":{"slug":"cloning-a-voice","title":"A voice from three seconds","url":"https://addaly.com/learn/ai-actually-explained/cloning-a-voice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When the model looks at a picture', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/the-model-that-looks-at-pictures (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When the model looks at a picture","courseTitle":"AI, Actually Explained","module":{"index":6,"slug":"beyond-text","title":"Beyond text: pictures, voices and video"},"answer":null,"keyIdea":"A vision model cuts a photograph into a grid of patches and feeds them in as tokens beside the words, which is why it describes a scene fluently, costs hundreds of tokens per image, and is reliably blind to counting, position and fine detail.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens","title":"What a photograph costs in the context window","series":[{"label":"Fixed overhead for any image","value":85},{"label":"One 512-pixel tile","value":170},{"label":"A page of English prose (about 500 words)","value":650},{"label":"A 1024 by 1024 photograph","value":765,"highlight":true}],"caption":"An image is cut into patches and enters as tokens beside the words. One large lab charges 85 tokens fixed plus 170 for each 512-pixel tile, so a 1024-square photograph is 765: a little more than a page of English prose. Ten photographs spend a short chapter before you have asked anything."},"alt":"Figure: What a photograph costs in the context window. Bars (tokens): Fixed overhead for any image 85; One 512-pixel tile 170; A page of English prose (about 500 words) 650; A 1024 by 1024 photograph 765. An image is cut into patches and enters as tokens beside the words. One large lab charges 85 tokens fixed plus 170 for each 512-pixel tile, so a 1024-square photograph is 765: a little more than a page of English prose. Ten photographs spend a short chapter before you have asked anything."}},"blocks":[{"t":"h2","id":"the-other-direction","runs":[{"text":"The other direction"}]},{"t":"p","runs":[{"text":"The previous lessons made pictures from words. This one is about the reverse (a model that takes a photograph and answers questions about it) because it is now in every major chatbot, and because the way it works explains a set of failures that catch people out."}]},{"t":"h2","id":"patches-as-words","runs":[{"text":"Patches as words"}]},{"t":"p","runs":[{"text":"Recall from the history module that in 2020 a Google team cut an image into a grid of small squares and fed the squares to a transformer as if they were words. That is still the mechanism. A photograph is divided into patches (sixteen pixels square is common) each patch is turned into a vector, and the vectors are placed into the model’s context alongside the text tokens. From the model’s point of view an image is a few hundred extra tokens that happen to have come from pixels. It attends across them the way it attends across a sentence."}]},{"t":"p","runs":[{"text":"That has a price you can see on an invoice. One large lab charges for images by tile: a fixed 85 tokens plus 170 for each 512-pixel square, so a photograph of 1024 by 1024 costs 765 tokens: about the same as a page and a half of text. Send ten photographs and you have spent the equivalent of a short chapter before asking anything."}]},{"t":"h2","id":"what-it-is-good-at","runs":[{"text":"What it is good at"}]},{"t":"p","runs":[{"text":"Describing. Given a photograph of a street, a modern model produces a paragraph that a sighted person would accept: the shops, the weather, the vehicle, the sign. It reads printed text in the image well, including handwriting, and it reads charts and diagrams well enough to answer questions about them. Since 2023 a free app used by blind and low-vision people has offered exactly this, and it is one of the least ambiguous benefits in this entire course."}]},{"t":"p","runs":[{"text":"The reason it is good at describing is the reason the chatbot is good at prose: the training pairs images with fluent captions, and fluent captions are what it learned to produce."}]},{"t":"h2","id":"where-it-is-blind-and-why","runs":[{"text":"Where it is blind, and why"}]},{"t":"p","runs":[{"text":"The failures cluster, and they cluster where patches-as-tokens breaks down."}]},{"t":"p","runs":[{"text":"Counting.","b":true},{"text":" Ask how many people are in a crowd, or how many windows on a building, and the answer is wrong once the number passes four or five. Objects that span several patches, or share a patch, are not separate things to the model; they are regions of a texture, exactly as in the drawing lessons."}]},{"t":"p","runs":[{"text":"Position.","b":true},{"text":" Left of, above, between. The patches carry position stamps, but the training captions rarely describe spatial relations precisely, so the model has thin support for them. A 2024 study gave the leading models tasks a child does easily (count how many times two lines cross, say whether two circles overlap) and found them near chance."}]},{"t":"p","runs":[{"text":"Small detail.","b":true},{"text":" A patch that is sixteen pixels square holds one blurred vector. Fine print, a small dial, a mole on skin, the time on an analogue clock: anything that lives inside one patch is mostly lost. Analogue clock reading was a well-documented failure for years for this reason."}]},{"t":"p","runs":[{"text":"Anything that requires measuring.","b":true},{"text":" Angles, lengths, whether a wall is straight. The model has no ruler, only the statistics of what captions say."}]},{"t":"p","runs":[{"text":"Each of these is a mechanism, not a bug in this week’s release, and each will improve as patches shrink and data improves, without being removed."}]},{"t":"h2","id":"what-this-means-for-the-medical-photograph","runs":[{"text":"What this means for the medical photograph"}]},{"t":"p","runs":[{"text":"People send these models photographs of rashes, X-rays, moles and blood-test printouts, and the model answers fluently. Two things are true at once. It reads the printout accurately, because that is text. And its judgement of the rash is a caption-shaped guess: a plausible description of what such images tend to be captioned as, with no measurement, no history and no accountability. Systems that read medical images exist and are trained, validated and regulated separately, as the second module explained. The chatbot with vision is not one of them."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"p","runs":[{"text":"Open-weights vision models exist and run locally. The first, from a university group in 2023, showed the recipe could be followed at small scale; Alibaba’s and Google’s small multimodal models of 2024 and 2025 fit on a laptop, and a later module shows the tools that run them. The quality trails the closed models, but the mechanism, and the blind spots, are identical."}]}],"check":{"question":[{"text":"A vision model describes a crowded market scene accurately and then gives the wrong number when asked how many people are in it. Why?"}],"options":[{"text":[{"text":"The model was not trained on enough photographs of markets."}],"correct":false,"tempting":[{"text":"Blames scarce data for a failure that appears in every kind of scene once the count passes a handful."}]},{"text":[{"text":"Counting requires arithmetic, and language models cannot do arithmetic reliably beyond a few digits, so any count is a guess."}],"correct":false,"tempting":[{"text":"Reaches for the arithmetic limit; the failure is upstream, in the model never having separate objects to count."}]},{"text":[{"text":"The image was compressed before the model saw it, losing the people."}],"correct":false,"tempting":[{"text":"Invents a preprocessing loss; the scene was described accurately from the same input."}]},{"text":[{"text":"To the model the people are regions of texture, not separate objects, so a description is supported and a count is not."}],"correct":true,"tempting":null}],"order":[0,2,1,3],"why":null},"prev":{"slug":"why-the-hands-were-wrong","title":"Why the hands were wrong","minutes":8},"next":{"slug":"cloning-a-voice","title":"A voice from three seconds","minutes":9}}}