One space for pictures and words

CLIP, and why it is interesting

CLIP was trained on hundreds of millions of image–caption pairs with one objective: put an image and its caption close together, and push mismatched pairs apart. The result is a shared space where a photograph and the sentence describing it are neighbours.

from sentence_transformers import SentenceTransformer
from PIL import Image

m = SentenceTransformer("clip-ViT-B-32")
img = m.encode([Image.open("kitchen.jpg")], normalize_embeddings=True)
txt = m.encode(["a photo of a kitchen", "a photo of a motorbike"],
               normalize_embeddings=True)
print(img @ txt.T)     # [[0.31 0.09]]

Text and image vectors are directly comparable, which gives you two capabilities that used to require training:

Search photos with words. Embed 50,000 images once, then search them with a typed sentence. No tags, no labels, no training. This runs on a laptop and it is the single most immediately useful thing in this lesson.

Zero-shot classification. Write the labels as sentences, embed them, and classify each image by which label sentence is nearest. Accuracy depends heavily on how you phrase the labels: “a photo of a cat” typically beats bare “cat”, because the training captions were sentences. That sensitivity to phrasing is a real limitation and an honest part of the method.

The limits, stated plainly

• It is not an OCR system. CLIP reads text in images unreliably. Use a dedicated OCR model.
• It has no fine spatial sense. “A red cup to the left of a blue plate” retrieves images with red cups and blue plates in any arrangement.
• It counts badly. Three of something and five of something are barely distinguished.
• It inherits the biases of internet captions, including the associations between people’s appearance and occupations that those captions carry. For anything involving photographs of people, this is not a footnote; it is a reason to evaluate carefully before deploying, and sometimes a reason not to.
• The base model’s resolution is low: 224 pixels square. Fine detail is resized away before the model sees it.

Newer families (SigLIP, and the multilingual variants) improve accuracy and, importantly, extend the text side beyond English. clip-ViT-B-32-multilingual-v1 handles many languages on the text side while sharing the same image space.

Audio embeddings

The same idea exists for sound. CLAP places audio clips and text descriptions in a shared space, so you can search a sound library for “dog barking” without tags. Speaker-embedding models such as the ECAPA and x-vector families place recordings of the same speaker close together, which is how diarisation and speaker verification work.

from transformers import AutoProcessor, ClapModel

The practical caution is the same as module five’s: sampling rate must match what the model expects, and a mismatch degrades results with no error.

Combining modalities in one search

Because vectors from a shared-space model are comparable, one index can hold both. A product catalogue can be searched by photograph or by description against the same set of vectors, with no separate pipeline.

What you must not do is mix vectors from different models in one index. A 512-dimensional CLIP vector and a 512-dimensional text-embedding vector have the same shape and no relationship whatsoever; the arithmetic runs and returns confident nonsense. Store the model name and revision with every index, and refuse to load an index whose recorded model does not match the one you are querying with. That check is four lines and prevents a bug that is almost impossible to diagnose from its symptoms.

Document images

A newer approach embeds page images directly rather than extracting text first: useful for scanned documents, forms and anything where layout carries meaning that plain text loses. It is more expensive per page and it skips a fragile OCR step, which on poor scans is frequently a net gain. Worth knowing it exists when a text-extraction pipeline is producing rubbish.

Do this now

Embed a folder of your own photographs with CLIP and search them by typing a sentence. It takes fifteen lines and it is the demonstration that makes embeddings click for most people, because you did not label anything and it works.