‘It understands images’: what that buys

What happens to your picture

When you attach an image, it is not examined pixel by pixel at full resolution. It is resized to fit a budget, cut into tiles, and each tile converted into a fixed number of tokens: commonly a few hundred to a couple of thousand tokens for a whole image. A 4,000-pixel-wide scan of a dense page is compressed into roughly the same budget as a photograph of a cat.

That single fact predicts nearly every image failure you will hit. Small text is the first casualty, because after downsampling it is no longer legible in the tokens the model receives. The model does not report a blurry input; it reads what it can and fills the rest in.

What it is good at

• Describing layout and content. What is in this screenshot, what does this interface do, where is the error message.
• Explaining a diagram. Architecture drawings, flowcharts, circuit sketches, anatomical figures.
• Reading moderately sized printed text. A slide, a poster, a page photographed straight-on at decent resolution.
• Turning a photograph of a whiteboard into notes.
• Describing an image for accessibility, which is a valuable everyday use.
• Judging design and composition: spacing, alignment, contrast, whether something reads.

What it is unreliable at, and why

Exact numbers from a dense table. Downsampling plus the model’s tendency to produce a plausible digit means a figure can come back as 3.7 when the page says 8.7. It will not flag the uncertainty. Never let a number cross from an image into a spreadsheet unchecked.

Counting. Ask how many people are in a photograph or how many bars are in a chart and you get an estimate presented as a count. Errors grow past about six objects.

Precise spatial relations. Which of these two lines is longer, is this exactly centred, does this bar reach the axis. Approximate answers, confidently phrased.

Handwriting. Highly variable. Neat printing often works; joined-up cursive, medical notes and anything faint often does not, and it will produce fluent text that is not what the page says.

Small print in a photograph. Terms on a package, an invoice line, an ingredient list at an angle. The fix is a straight-on photo, cropped, at the highest resolution the tool will accept.

The reliable technique for documents

For anything where the exact characters matter (invoices, receipts, forms, tables of figures) do not ask a vision model to read the page. Extract the text with a tool built for it, then hand the text to the model.

Free paths, all capable:

• Tesseract: the long-standing free OCR engine, available on every platform.
• PaddleOCR and docTR: free, more accurate on messy layouts than Tesseract, and better at non-Latin scripts.
• pdftotext (from Poppler): for PDFs that already contain a text layer, which is most PDFs produced by a computer, and it is exact rather than recognised.

pdftotext -layout invoice.pdf -   # exact, if the PDF has a text layer
tesseract scan.png - --psm 6      # OCR, when it is a picture of a page

Check first whether the PDF already has text. If it does, using a vision model to read it is strictly worse: slower, more expensive, and capable of getting a digit wrong where the exact characters were sitting right there.

Audio and video

The same principle. Products that “understand video” usually sample a frame every few seconds and transcribe the audio, then reason over frames plus transcript. So anything that happens between the sampled frames is invisible, and any question that depends on continuous motion is being answered from stills.

For audio the free path is strong: Whisper, released openly by OpenAI, runs on an ordinary laptop and transcribes dozens of languages well. whisper.cpp runs it fast on a CPU, and faster-whisper on modest GPUs. Transcribe locally, then send the transcript. It is cheaper, more private, more accurate, and you keep a text record you can search.

The habit

Ask what the model is actually being given. Text extracted exactly, or pixels squeezed into a token budget? A continuous recording, or eleven frames? Once you know that, the failures stop being mysterious and start being predictable, which is the only kind you can work around.