{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"multimodal-in-practice","title":"'It understands images': what that buys","url":"https://addaly.com/learn/choosing-your-tools/multimodal-in-practice","position":8,"of":76,"minutes":8,"answer":"A vision model receives a downsampled, tokenised version of your image, so exact characters and counts should be extracted with OCR or a text layer first and only then handed to the model.","keyIdea":"A vision model receives a downsampled, tokenised version of your image, so exact characters and counts should be extracted with OCR or a text layer first and only then handed to the model."},"body":{"format":"markdown","text":"## What happens to your picture\n\nWhen you attach an image, it is not examined pixel by pixel at full resolution. It is resized to fit a budget, cut into tiles, and each tile converted into a fixed number of tokens: commonly a few hundred to a couple of thousand tokens for a whole image. A 4,000-pixel-wide scan of a dense page is compressed into roughly the same budget as a photograph of a cat.\n\nThat single fact predicts nearly every image failure you will hit. Small text is the first casualty, because after downsampling it is no longer legible in the tokens the model receives. The model does not report a blurry input; it reads what it can and fills the rest in.\n\n## What it is good at\n\n- **Describing layout and content.** What is in this screenshot, what does this interface do, where is the error message.\n- **Explaining a diagram.** Architecture drawings, flowcharts, circuit sketches, anatomical figures.\n- **Reading moderately sized printed text.** A slide, a poster, a page photographed straight-on at decent resolution.\n- **Turning a photograph of a whiteboard into notes.**\n- **Describing an image for accessibility**, which is a valuable everyday use.\n- **Judging design and composition**: spacing, alignment, contrast, whether something reads.\n\n## What it is unreliable at, and why\n\n**Exact numbers from a dense table.** Downsampling plus the model's tendency to produce a plausible digit means a figure can come back as 3.7 when the page says 8.7. It will not flag the uncertainty. Never let a number cross from an image into a spreadsheet unchecked.\n\n**Counting.** Ask how many people are in a photograph or how many bars are in a chart and you get an estimate presented as a count. Errors grow past about six objects.\n\n**Precise spatial relations.** Which of these two lines is longer, is this exactly centred, does this bar reach the axis. Approximate answers, confidently phrased.\n\n**Handwriting.** Highly variable. Neat printing often works; joined-up cursive, medical notes and anything faint often does not, and it will produce fluent text that is not what the page says.\n\n**Small print in a photograph.** Terms on a package, an invoice line, an ingredient list at an angle. The fix is a straight-on photo, cropped, at the highest resolution the tool will accept.\n\n## The reliable technique for documents\n\nFor anything where the exact characters matter (invoices, receipts, forms, tables of figures) do not ask a vision model to read the page. Extract the text with a tool built for it, then hand the *text* to the model.\n\nFree paths, all capable:\n\n- **Tesseract**: the long-standing free OCR engine, available on every platform.\n- **PaddleOCR** and **docTR**: free, more accurate on messy layouts than Tesseract, and better at non-Latin scripts.\n- **pdftotext** (from Poppler): for PDFs that already contain a text layer, which is most PDFs produced by a computer, and it is exact rather than recognised.\n\n```bash\npdftotext -layout invoice.pdf -   # exact, if the PDF has a text layer\ntesseract scan.png - --psm 6      # OCR, when it is a picture of a page\n```\n\nCheck first whether the PDF already has text. If it does, using a vision model to read it is strictly worse: slower, more expensive, and capable of getting a digit wrong where the exact characters were sitting right there.\n\n## Audio and video\n\nThe same principle. Products that \"understand video\" usually sample a frame every few seconds and transcribe the audio, then reason over frames plus transcript. So anything that happens between the sampled frames is invisible, and any question that depends on continuous motion is being answered from stills.\n\nFor audio the free path is strong: **Whisper**, released openly by OpenAI, runs on an ordinary laptop and transcribes dozens of languages well. **whisper.cpp** runs it fast on a CPU, and **faster-whisper** on modest GPUs. Transcribe locally, then send the transcript. It is cheaper, more private, more accurate, and you keep a text record you can search.\n\n## The habit\n\nAsk what the model is actually being given. Text extracted exactly, or pixels squeezed into a token budget? A continuous recording, or eleven frames? Once you know that, the failures stop being mysterious and start being predictable, which is the only kind you can work around."},"check":{"question":"Someone photographs a computer-generated PDF invoice and asks a vision model to read the totals, and one figure comes back wrong. What was the avoidable mistake?","options":[{"text":"Using a general-purpose assistant rather than one of the document-specialised vision models trained specifically on invoices and receipts.","correct":false,"tempting":"Assumes a specialised model removes the constraint; the loss happens when pixels are downsampled into a token budget, which applies to specialised vision models too."},{"text":"Photographing a PDF that already contained an exact text layer pdftotext could read.","correct":true},{"text":"Not asking the model to state its confidence in each number it read.","correct":false,"tempting":"Requests a self-report the model cannot ground, since it has no signal distinguishing a digit it read from a digit it reconstructed."},{"text":"Attaching only one page instead of the whole document for context.","correct":false,"tempting":"Adds context to a problem of resolution; more pages would consume more of the same token budget and make small print less legible, not more."}]},"prev":{"slug":"what-tool-use-means","title":"What 'it can browse the web' actually means","url":"https://addaly.com/learn/choosing-your-tools/what-tool-use-means"},"next":{"slug":"under-new-management","title":"When the engine gets swapped","url":"https://addaly.com/learn/choosing-your-tools/under-new-management"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) ''It understands images': what that buys', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/multimodal-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"‘It understands images’: what that buys","courseTitle":"Choosing and Using the Tools","module":{"index":1,"slug":"anatomy-of-a-tool","title":"What a tool is made of"},"answer":null,"keyIdea":"A vision model receives a downsampled, tokenised version of your image, so exact characters and counts should be extracted with OCR or a text layer first and only then handed to the model.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-happens-to-your-picture","runs":[{"text":"What happens to your picture"}]},{"t":"p","runs":[{"text":"When you attach an image, it is not examined pixel by pixel at full resolution. It is resized to fit a budget, cut into tiles, and each tile converted into a fixed number of tokens: commonly a few hundred to a couple of thousand tokens for a whole image. A 4,000-pixel-wide scan of a dense page is compressed into roughly the same budget as a photograph of a cat."}]},{"t":"p","runs":[{"text":"That single fact predicts nearly every image failure you will hit. Small text is the first casualty, because after downsampling it is no longer legible in the tokens the model receives. The model does not report a blurry input; it reads what it can and fills the rest in."}]},{"t":"h2","id":"what-it-is-good-at","runs":[{"text":"What it is good at"}]},{"t":"ul","items":[[{"text":"Describing layout and content.","b":true},{"text":" What is in this screenshot, what does this interface do, where is the error message."}],[{"text":"Explaining a diagram.","b":true},{"text":" Architecture drawings, flowcharts, circuit sketches, anatomical figures."}],[{"text":"Reading moderately sized printed text.","b":true},{"text":" A slide, a poster, a page photographed straight-on at decent resolution."}],[{"text":"Turning a photograph of a whiteboard into notes.","b":true}],[{"text":"Describing an image for accessibility","b":true},{"text":", which is a valuable everyday use."}],[{"text":"Judging design and composition","b":true},{"text":": spacing, alignment, contrast, whether something reads."}]]},{"t":"h2","id":"what-it-is-unreliable-at-and-why","runs":[{"text":"What it is unreliable at, and why"}]},{"t":"p","runs":[{"text":"Exact numbers from a dense table.","b":true},{"text":" Downsampling plus the model’s tendency to produce a plausible digit means a figure can come back as 3.7 when the page says 8.7. It will not flag the uncertainty. Never let a number cross from an image into a spreadsheet unchecked."}]},{"t":"p","runs":[{"text":"Counting.","b":true},{"text":" Ask how many people are in a photograph or how many bars are in a chart and you get an estimate presented as a count. Errors grow past about six objects."}]},{"t":"p","runs":[{"text":"Precise spatial relations.","b":true},{"text":" Which of these two lines is longer, is this exactly centred, does this bar reach the axis. Approximate answers, confidently phrased."}]},{"t":"p","runs":[{"text":"Handwriting.","b":true},{"text":" Highly variable. Neat printing often works; joined-up cursive, medical notes and anything faint often does not, and it will produce fluent text that is not what the page says."}]},{"t":"p","runs":[{"text":"Small print in a photograph.","b":true},{"text":" Terms on a package, an invoice line, an ingredient list at an angle. The fix is a straight-on photo, cropped, at the highest resolution the tool will accept."}]},{"t":"h2","id":"the-reliable-technique-for-documents","runs":[{"text":"The reliable technique for documents"}]},{"t":"p","runs":[{"text":"For anything where the exact characters matter (invoices, receipts, forms, tables of figures) do not ask a vision model to read the page. Extract the text with a tool built for it, then hand the "},{"text":"text","i":true},{"text":" to the model."}]},{"t":"p","runs":[{"text":"Free paths, all capable:"}]},{"t":"ul","items":[[{"text":"Tesseract","b":true},{"text":": the long-standing free OCR engine, available on every platform."}],[{"text":"PaddleOCR","b":true},{"text":" and "},{"text":"docTR","b":true},{"text":": free, more accurate on messy layouts than Tesseract, and better at non-Latin scripts."}],[{"text":"pdftotext","b":true},{"text":" (from Poppler): for PDFs that already contain a text layer, which is most PDFs produced by a computer, and it is exact rather than recognised."}]]},{"t":"code","lang":"bash","label":"Bash","text":"pdftotext -layout invoice.pdf -   # exact, if the PDF has a text layer\ntesseract scan.png - --psm 6      # OCR, when it is a picture of a page"},{"t":"p","runs":[{"text":"Check first whether the PDF already has text. If it does, using a vision model to read it is strictly worse: slower, more expensive, and capable of getting a digit wrong where the exact characters were sitting right there."}]},{"t":"h2","id":"audio-and-video","runs":[{"text":"Audio and video"}]},{"t":"p","runs":[{"text":"The same principle. Products that “understand video” usually sample a frame every few seconds and transcribe the audio, then reason over frames plus transcript. So anything that happens between the sampled frames is invisible, and any question that depends on continuous motion is being answered from stills."}]},{"t":"p","runs":[{"text":"For audio the free path is strong: "},{"text":"Whisper","b":true},{"text":", released openly by OpenAI, runs on an ordinary laptop and transcribes dozens of languages well. "},{"text":"whisper.cpp","b":true},{"text":" runs it fast on a CPU, and "},{"text":"faster-whisper","b":true},{"text":" on modest GPUs. Transcribe locally, then send the transcript. It is cheaper, more private, more accurate, and you keep a text record you can search."}]},{"t":"h2","id":"the-habit","runs":[{"text":"The habit"}]},{"t":"p","runs":[{"text":"Ask what the model is actually being given. Text extracted exactly, or pixels squeezed into a token budget? A continuous recording, or eleven frames? Once you know that, the failures stop being mysterious and start being predictable, which is the only kind you can work around."}]}],"check":{"question":[{"text":"Someone photographs a computer-generated PDF invoice and asks a vision model to read the totals, and one figure comes back wrong. What was the avoidable mistake?"}],"options":[{"text":[{"text":"Using a general-purpose assistant rather than one of the document-specialised vision models trained specifically on invoices and receipts."}],"correct":false,"tempting":[{"text":"Assumes a specialised model removes the constraint; the loss happens when pixels are downsampled into a token budget, which applies to specialised vision models too."}]},{"text":[{"text":"Photographing a PDF that already contained an exact text layer pdftotext could read."}],"correct":true,"tempting":null},{"text":[{"text":"Not asking the model to state its confidence in each number it read."}],"correct":false,"tempting":[{"text":"Requests a self-report the model cannot ground, since it has no signal distinguishing a digit it read from a digit it reconstructed."}]},{"text":[{"text":"Attaching only one page instead of the whole document for context."}],"correct":false,"tempting":[{"text":"Adds context to a problem of resolution; more pages would consume more of the same token budget and make small print less legible, not more."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"what-tool-use-means","title":"What ‘it can browse the web’ actually means","minutes":9},"next":{"slug":"under-new-management","title":"When the engine gets swapped","minutes":8}}}