{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"embedding-images-and-audio","title":"One space for pictures and words","url":"https://addaly.com/learn/hugging-face/embedding-images-and-audio","position":55,"of":75,"minutes":8,"answer":"CLIP-style models place images and their captions in one shared space, so photographs become searchable by typed sentences and classifiable with no training, while the same shape does not mean the same space.","keyIdea":"CLIP-style models place images and their captions in one shared space, so photographs become searchable by typed sentences and classifiable with no training, while the same shape does not mean the same space, and mixing vectors from different models in one index produces confident nonsense."},"body":{"format":"markdown","text":"## CLIP, and why it is interesting\n\nCLIP was trained on hundreds of millions of image–caption pairs with one objective: put an image and its caption close together, and push mismatched pairs apart. The result is a **shared** space where a photograph and the sentence describing it are neighbours.\n\n```python\nfrom sentence_transformers import SentenceTransformer\nfrom PIL import Image\n\nm = SentenceTransformer(\"clip-ViT-B-32\")\nimg = m.encode([Image.open(\"kitchen.jpg\")], normalize_embeddings=True)\ntxt = m.encode([\"a photo of a kitchen\", \"a photo of a motorbike\"],\n               normalize_embeddings=True)\nprint(img @ txt.T)     # [[0.31 0.09]]\n```\n\nText and image vectors are directly comparable, which gives you two capabilities that used to require training:\n\n**Search photos with words.** Embed 50,000 images once, then search them with a typed sentence. No tags, no labels, no training. This runs on a laptop and it is the single most immediately useful thing in this lesson.\n\n**Zero-shot classification.** Write the labels as sentences, embed them, and classify each image by which label sentence is nearest. Accuracy depends heavily on how you phrase the labels: \"a photo of a cat\" typically beats bare \"cat\", because the training captions were sentences. That sensitivity to phrasing is a real limitation and an honest part of the method.\n\n## The limits, stated plainly\n\n- **It is not an OCR system.** CLIP reads text in images unreliably. Use a dedicated OCR model.\n- **It has no fine spatial sense.** \"A red cup to the left of a blue plate\" retrieves images with red cups and blue plates in any arrangement.\n- **It counts badly.** Three of something and five of something are barely distinguished.\n- **It inherits the biases of internet captions**, including the associations between people's appearance and occupations that those captions carry. For anything involving photographs of people, this is not a footnote; it is a reason to evaluate carefully before deploying, and sometimes a reason not to.\n- **The base model's resolution is low**: 224 pixels square. Fine detail is resized away before the model sees it.\n\nNewer families (SigLIP, and the multilingual variants) improve accuracy and, importantly, extend the text side beyond English. `clip-ViT-B-32-multilingual-v1` handles many languages on the text side while sharing the same image space.\n\n## Audio embeddings\n\nThe same idea exists for sound. CLAP places audio clips and text descriptions in a shared space, so you can search a sound library for \"dog barking\" without tags. Speaker-embedding models such as the ECAPA and x-vector families place recordings of the same speaker close together, which is how diarisation and speaker verification work.\n\n```python\nfrom transformers import AutoProcessor, ClapModel\n```\n\nThe practical caution is the same as module five's: sampling rate must match what the model expects, and a mismatch degrades results with no error.\n\n## Combining modalities in one search\n\nBecause vectors from a shared-space model are comparable, one index can hold both. A product catalogue can be searched by photograph or by description against the same set of vectors, with no separate pipeline.\n\nWhat you must not do is mix vectors from **different** models in one index. A 512-dimensional CLIP vector and a 512-dimensional text-embedding vector have the same shape and no relationship whatsoever; the arithmetic runs and returns confident nonsense. Store the model name and revision with every index, and refuse to load an index whose recorded model does not match the one you are querying with. That check is four lines and prevents a bug that is almost impossible to diagnose from its symptoms.\n\n## Document images\n\nA newer approach embeds page images directly rather than extracting text first: useful for scanned documents, forms and anything where layout carries meaning that plain text loses. It is more expensive per page and it skips a fragile OCR step, which on poor scans is frequently a net gain. Worth knowing it exists when a text-extraction pipeline is producing rubbish.\n\n## Do this now\n\nEmbed a folder of your own photographs with CLIP and search them by typing a sentence. It takes fifteen lines and it is the demonstration that makes embeddings click for most people, because you did not label anything and it works."},"check":{"question":"Two models both output 512-dimensional vectors. Storing both in one index and searching produces plausible but wrong results. Why does nothing fail loudly?","options":[{"text":"Dot products between equal-length vectors are always defined, so the arithmetic succeeds regardless of whether the spaces relate.","correct":true},{"text":"Normalisation maps every model's output onto the same unit sphere, which makes vectors from different models genuinely comparable.","correct":false,"tempting":"Normalisation equalises length without aligning the underlying dimensions' meanings."},{"text":"Both models were trained on overlapping web data, so their spaces are approximately aligned and results degrade only slightly.","correct":false,"tempting":"Shared training data does not produce aligned coordinate systems; alignment must be trained for explicitly."},{"text":"The index validates dimensionality at insert time, so a mismatch would have been rejected if the spaces were incompatible.","correct":false,"tempting":"Dimensionality is the only thing checked, which is precisely why the incompatibility passes silently."}]},"prev":{"slug":"reranking-with-cross-encoders","title":"The second pass that fixes the first","url":"https://addaly.com/learn/hugging-face/reranking-with-cross-encoders"},"next":{"slug":"measuring-your-retrieval","title":"Thirty questions, and the numbers they give you","url":"https://addaly.com/learn/hugging-face/measuring-your-retrieval"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'One space for pictures and words', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/embedding-images-and-audio (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"One space for pictures and words","courseTitle":"Hugging Face, End to End","module":{"index":6,"slug":"embeddings-and-retrieval","title":"Embeddings, and search that understands"},"answer":null,"keyIdea":"CLIP-style models place images and their captions in one shared space, so photographs become searchable by typed sentences and classifiable with no training, while the same shape does not mean the same space, and mixing vectors from different models in one index produces confident nonsense.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"clip-and-why-it-is-interesting","runs":[{"text":"CLIP, and why it is interesting"}]},{"t":"p","runs":[{"text":"CLIP was trained on hundreds of millions of image–caption pairs with one objective: put an image and its caption close together, and push mismatched pairs apart. The result is a "},{"text":"shared","b":true},{"text":" space where a photograph and the sentence describing it are neighbours."}]},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import SentenceTransformer\nfrom PIL import Image\n\nm = SentenceTransformer(\"clip-ViT-B-32\")\nimg = m.encode([Image.open(\"kitchen.jpg\")], normalize_embeddings=True)\ntxt = m.encode([\"a photo of a kitchen\", \"a photo of a motorbike\"],\n               normalize_embeddings=True)\nprint(img @ txt.T)     # [[0.31 0.09]]"},{"t":"p","runs":[{"text":"Text and image vectors are directly comparable, which gives you two capabilities that used to require training:"}]},{"t":"p","runs":[{"text":"Search photos with words.","b":true},{"text":" Embed 50,000 images once, then search them with a typed sentence. No tags, no labels, no training. This runs on a laptop and it is the single most immediately useful thing in this lesson."}]},{"t":"p","runs":[{"text":"Zero-shot classification.","b":true},{"text":" Write the labels as sentences, embed them, and classify each image by which label sentence is nearest. Accuracy depends heavily on how you phrase the labels: “a photo of a cat” typically beats bare “cat”, because the training captions were sentences. That sensitivity to phrasing is a real limitation and an honest part of the method."}]},{"t":"h2","id":"the-limits-stated-plainly","runs":[{"text":"The limits, stated plainly"}],"role":"limit"},{"t":"ul","items":[[{"text":"It is not an OCR system.","b":true},{"text":" CLIP reads text in images unreliably. Use a dedicated OCR model."}],[{"text":"It has no fine spatial sense.","b":true},{"text":" “A red cup to the left of a blue plate” retrieves images with red cups and blue plates in any arrangement."}],[{"text":"It counts badly.","b":true},{"text":" Three of something and five of something are barely distinguished."}],[{"text":"It inherits the biases of internet captions","b":true},{"text":", including the associations between people’s appearance and occupations that those captions carry. For anything involving photographs of people, this is not a footnote; it is a reason to evaluate carefully before deploying, and sometimes a reason not to."}],[{"text":"The base model’s resolution is low","b":true},{"text":": 224 pixels square. Fine detail is resized away before the model sees it."}]]},{"t":"p","runs":[{"text":"Newer families (SigLIP, and the multilingual variants) improve accuracy and, importantly, extend the text side beyond English. "},{"text":"clip-ViT-B-32-multilingual-v1","code":true},{"text":" handles many languages on the text side while sharing the same image space."}]},{"t":"h2","id":"audio-embeddings","runs":[{"text":"Audio embeddings"}]},{"t":"p","runs":[{"text":"The same idea exists for sound. CLAP places audio clips and text descriptions in a shared space, so you can search a sound library for “dog barking” without tags. Speaker-embedding models such as the ECAPA and x-vector families place recordings of the same speaker close together, which is how diarisation and speaker verification work."}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import AutoProcessor, ClapModel"},{"t":"p","runs":[{"text":"The practical caution is the same as module five’s: sampling rate must match what the model expects, and a mismatch degrades results with no error."}]},{"t":"h2","id":"combining-modalities-in-one-search","runs":[{"text":"Combining modalities in one search"}]},{"t":"p","runs":[{"text":"Because vectors from a shared-space model are comparable, one index can hold both. A product catalogue can be searched by photograph or by description against the same set of vectors, with no separate pipeline."}]},{"t":"p","runs":[{"text":"What you must not do is mix vectors from "},{"text":"different","b":true},{"text":" models in one index. A 512-dimensional CLIP vector and a 512-dimensional text-embedding vector have the same shape and no relationship whatsoever; the arithmetic runs and returns confident nonsense. Store the model name and revision with every index, and refuse to load an index whose recorded model does not match the one you are querying with. That check is four lines and prevents a bug that is almost impossible to diagnose from its symptoms."}]},{"t":"h2","id":"document-images","runs":[{"text":"Document images"}]},{"t":"p","runs":[{"text":"A newer approach embeds page images directly rather than extracting text first: useful for scanned documents, forms and anything where layout carries meaning that plain text loses. It is more expensive per page and it skips a fragile OCR step, which on poor scans is frequently a net gain. Worth knowing it exists when a text-extraction pipeline is producing rubbish."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Embed a folder of your own photographs with CLIP and search them by typing a sentence. It takes fifteen lines and it is the demonstration that makes embeddings click for most people, because you did not label anything and it works."}]}],"check":{"question":[{"text":"Two models both output 512-dimensional vectors. Storing both in one index and searching produces plausible but wrong results. Why does nothing fail loudly?"}],"options":[{"text":[{"text":"Dot products between equal-length vectors are always defined, so the arithmetic succeeds regardless of whether the spaces relate."}],"correct":true,"tempting":null},{"text":[{"text":"Normalisation maps every model’s output onto the same unit sphere, which makes vectors from different models genuinely comparable."}],"correct":false,"tempting":[{"text":"Normalisation equalises length without aligning the underlying dimensions’ meanings."}]},{"text":[{"text":"Both models were trained on overlapping web data, so their spaces are approximately aligned and results degrade only slightly."}],"correct":false,"tempting":[{"text":"Shared training data does not produce aligned coordinate systems; alignment must be trained for explicitly."}]},{"text":[{"text":"The index validates dimensionality at insert time, so a mismatch would have been rejected if the spaces were incompatible."}],"correct":false,"tempting":[{"text":"Dimensionality is the only thing checked, which is precisely why the incompatibility passes silently."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"reranking-with-cross-encoders","title":"The second pass that fixes the first","minutes":8},"next":{"slug":"measuring-your-retrieval","title":"Thirty questions, and the numbers they give you","minutes":9}}}