{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"sentence-transformers-in-practice","title":"The library, and the four calls you need","url":"https://addaly.com/learn/hugging-face/sentence-transformers-in-practice","position":49,"of":75,"minutes":8,"answer":"SentenceTransformer.encode with normalize_embeddings=True plus one matrix multiply is a complete search engine over a hundred thousand documents.","keyIdea":"`SentenceTransformer.encode` with `normalize_embeddings=True` plus one matrix multiply is a complete search engine over a hundred thousand documents, and the two details that silently cost accuracy are forgetting a family's query prefix and letting a vectors array drift out of order with its metadata."},"body":{"format":"markdown","text":"## Install and encode\n\n```bash\npip install sentence-transformers\n```\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"BAAI/bge-small-en-v1.5\")\n\nvectors = model.encode(\n    [\"first document\", \"second document\", \"third document\"],\n    batch_size=64,\n    normalize_embeddings=True,\n    show_progress_bar=True,\n)\nprint(vectors.shape)     # (3, 384)\n```\n\nFour arguments worth setting every time:\n\n- **`batch_size`**: 32 to 128 on a CPU, larger on a GPU. Encoding a list is far faster than encoding in a loop, for the same reasons as module two.\n- **`normalize_embeddings=True`**: scales every vector to unit length, which makes dot product and cosine similarity identical and lets you use the faster one. Set it and then be consistent; mixing normalised and unnormalised vectors in one index gives you nonsense.\n- **`show_progress_bar`**: for anything over a few thousand items, so you know whether to make tea.\n- **`convert_to_tensor=True`** when you want torch tensors on the GPU rather than NumPy arrays.\n\n## Similarity, in one call\n\n```python\nsim = model.similarity(vectors[:1], vectors)\nprint(sim)      # tensor([[1.0000, 0.6421, 0.3117]])\n```\n\n`model.similarity` uses whatever the model was trained for (cosine for most, dot product for a few) which removes a decision you would otherwise get wrong. With normalised vectors you can also do it by hand:\n\n```python\nimport numpy as np\nscores = vectors @ query_vec          # a plain matrix multiply\ntop = np.argsort(-scores)[:5]\n```\n\nOne matrix multiply is a complete search engine over a hundred thousand documents, and it runs in milliseconds. Do not reach for a vector database until you have measured that this is too slow.\n\n## The prompt prefixes almost nobody reads about\n\nSeveral of the best models were trained with an instruction prefix on the query but not on the documents. BGE, E5, GTE and Nomic families all do some version of this. Omit it and you lose several points of accuracy for no visible reason.\n\n```python\n# E5 family\ndocs = model.encode([f\"passage: {d}\" for d in documents])\nq = model.encode(\"query: where is my parcel\")\n\n# BGE family, via the library's own prompt support\nq = model.encode(\"where is my parcel\", prompt_name=\"query\")\n```\n\nThe model card says which. Read it before benchmarking anything: a large share of \"this model scored worse than advertised\" reports are this.\n\n## Matryoshka: shorter vectors on purpose\n\nSome recent models are trained so that the first *n* dimensions are usable on their own. You can truncate 768 down to 256 and keep most of the quality:\n\n```python\nmodel = SentenceTransformer(\"nomic-ai/nomic-embed-text-v1.5\", trust_remote_code=True)\nv = model.encode(texts, truncate_dim=256)\n```\n\nThat is a third of the memory and roughly a third of the search time, for a small accuracy cost. On a million documents it is the difference between fitting in RAM and not. It only works for models trained this way: truncating an ordinary model's vectors destroys them, because nothing made the early dimensions more important than the late ones.\n\n## Re-encoding is the cost that bites later\n\nEvery vector in your index was produced by one model at one revision. Change the model and every stored vector is worthless: you must re-encode the whole corpus. On a million chunks that is hours of compute and, if you are calling a hosted API, a real bill.\n\nTwo habits follow. Pick the model with the thirty-question test **before** encoding at scale rather than after. And keep the chunk text stored alongside the vectors, not only the vectors, so re-encoding is a local job rather than a re-extraction from source PDFs you may no longer have.\n\n## Saving and reloading vectors\n\n```python\nimport numpy as np\nnp.save(\"vectors.npy\", vectors.astype(\"float16\"))\nvectors = np.load(\"vectors.npy\").astype(\"float32\")\n```\n\nStore at float16 and cast back for arithmetic: half the disk, no measurable retrieval loss. Store the document ids and texts alongside in a separate file, in the same order, and never sort one without the other: a vectors array whose order has drifted from its metadata is the single most annoying bug in this area, because every result is plausible and wrong.\n\nWrite the model name and revision into the file too. Vectors from two different models are not comparable and there is nothing in the array to tell you which produced it.\n\n## Do this now\n\nEncode 1,000 sentences of your own, save them as float16, reload, and search with a matrix multiply. Time the search. It will be a few milliseconds, and that number is why most projects never need a vector database."},"check":{"question":"A team truncates 768-dimensional vectors from an ordinary embedding model to 256 dimensions to save memory, and retrieval quality collapses. Why did it work for a colleague using a different model?","options":[{"text":"The colleague normalised vectors after truncating, which restores the geometry that truncation distorts.","correct":false,"tempting":"Normalisation rescales but cannot recover information discarded with the dropped dimensions."},{"text":"The colleague's model has fewer dimensions to begin with, so proportionally less information was lost.","correct":false,"tempting":"Assumes the loss scales with the proportion cut rather than with how the model was trained."},{"text":"The colleague's model was trained so its early dimensions carry most of the information.","correct":true},{"text":"Truncation requires re-encoding the corpus at the shorter length, which the colleague did and the team did not.","correct":false,"tempting":"Matryoshka truncation is applied to existing vectors; re-encoding is not what makes it work."}]},"prev":{"slug":"what-an-embedding-is","title":"A list of numbers where near means similar","url":"https://addaly.com/learn/hugging-face/what-an-embedding-is"},"next":{"slug":"choosing-an-embedding-model","title":"Picking the model, and reading MTEB without being fooled","url":"https://addaly.com/learn/hugging-face/choosing-an-embedding-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The library, and the four calls you need', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/sentence-transformers-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The library, and the four calls you need","courseTitle":"Hugging Face, End to End","module":{"index":6,"slug":"embeddings-and-retrieval","title":"Embeddings, and search that understands"},"answer":null,"keyIdea":"`SentenceTransformer.encode` with `normalize_embeddings=True` plus one matrix multiply is a complete search engine over a hundred thousand documents, and the two details that silently cost accuracy are forgetting a family’s query prefix and letting a vectors array drift out of order with its metadata.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"install-and-encode","runs":[{"text":"Install and encode"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install sentence-transformers"},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"BAAI/bge-small-en-v1.5\")\n\nvectors = model.encode(\n    [\"first document\", \"second document\", \"third document\"],\n    batch_size=64,\n    normalize_embeddings=True,\n    show_progress_bar=True,\n)\nprint(vectors.shape)     # (3, 384)"},{"t":"p","runs":[{"text":"Four arguments worth setting every time:"}]},{"t":"ul","items":[[{"text":"batch_size","b":true,"code":true},{"text":": 32 to 128 on a CPU, larger on a GPU. Encoding a list is far faster than encoding in a loop, for the same reasons as module two."}],[{"text":"normalize_embeddings=True","b":true,"code":true},{"text":": scales every vector to unit length, which makes dot product and cosine similarity identical and lets you use the faster one. Set it and then be consistent; mixing normalised and unnormalised vectors in one index gives you nonsense."}],[{"text":"show_progress_bar","b":true,"code":true},{"text":": for anything over a few thousand items, so you know whether to make tea."}],[{"text":"convert_to_tensor=True","b":true,"code":true},{"text":" when you want torch tensors on the GPU rather than NumPy arrays."}]]},{"t":"h2","id":"similarity-in-one-call","runs":[{"text":"Similarity, in one call"}]},{"t":"code","lang":"python","label":"Python","text":"sim = model.similarity(vectors[:1], vectors)\nprint(sim)      # tensor([[1.0000, 0.6421, 0.3117]])"},{"t":"p","runs":[{"text":"model.similarity","code":true},{"text":" uses whatever the model was trained for (cosine for most, dot product for a few) which removes a decision you would otherwise get wrong. With normalised vectors you can also do it by hand:"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nscores = vectors @ query_vec          # a plain matrix multiply\ntop = np.argsort(-scores)[:5]"},{"t":"p","runs":[{"text":"One matrix multiply is a complete search engine over a hundred thousand documents, and it runs in milliseconds. Do not reach for a vector database until you have measured that this is too slow."}]},{"t":"h2","id":"the-prompt-prefixes-almost-nobody-reads-about","runs":[{"text":"The prompt prefixes almost nobody reads about"}]},{"t":"p","runs":[{"text":"Several of the best models were trained with an instruction prefix on the query but not on the documents. BGE, E5, GTE and Nomic families all do some version of this. Omit it and you lose several points of accuracy for no visible reason."}]},{"t":"code","lang":"python","label":"Python","text":"# E5 family\ndocs = model.encode([f\"passage: {d}\" for d in documents])\nq = model.encode(\"query: where is my parcel\")\n\n# BGE family, via the library's own prompt support\nq = model.encode(\"where is my parcel\", prompt_name=\"query\")"},{"t":"p","runs":[{"text":"The model card says which. Read it before benchmarking anything: a large share of “this model scored worse than advertised” reports are this."}]},{"t":"h2","id":"matryoshka-shorter-vectors-on-purpose","runs":[{"text":"Matryoshka: shorter vectors on purpose"}]},{"t":"p","runs":[{"text":"Some recent models are trained so that the first "},{"text":"n","i":true},{"text":" dimensions are usable on their own. You can truncate 768 down to 256 and keep most of the quality:"}]},{"t":"code","lang":"python","label":"Python","text":"model = SentenceTransformer(\"nomic-ai/nomic-embed-text-v1.5\", trust_remote_code=True)\nv = model.encode(texts, truncate_dim=256)"},{"t":"p","runs":[{"text":"That is a third of the memory and roughly a third of the search time, for a small accuracy cost. On a million documents it is the difference between fitting in RAM and not. It only works for models trained this way: truncating an ordinary model’s vectors destroys them, because nothing made the early dimensions more important than the late ones."}]},{"t":"h2","id":"re-encoding-is-the-cost-that-bites-later","runs":[{"text":"Re-encoding is the cost that bites later"}]},{"t":"p","runs":[{"text":"Every vector in your index was produced by one model at one revision. Change the model and every stored vector is worthless: you must re-encode the whole corpus. On a million chunks that is hours of compute and, if you are calling a hosted API, a real bill."}]},{"t":"p","runs":[{"text":"Two habits follow. Pick the model with the thirty-question test "},{"text":"before","b":true},{"text":" encoding at scale rather than after. And keep the chunk text stored alongside the vectors, not only the vectors, so re-encoding is a local job rather than a re-extraction from source PDFs you may no longer have."}]},{"t":"h2","id":"saving-and-reloading-vectors","runs":[{"text":"Saving and reloading vectors"}]},{"t":"code","lang":"python","label":"Python","text":"import numpy as np\nnp.save(\"vectors.npy\", vectors.astype(\"float16\"))\nvectors = np.load(\"vectors.npy\").astype(\"float32\")"},{"t":"p","runs":[{"text":"Store at float16 and cast back for arithmetic: half the disk, no measurable retrieval loss. Store the document ids and texts alongside in a separate file, in the same order, and never sort one without the other: a vectors array whose order has drifted from its metadata is the single most annoying bug in this area, because every result is plausible and wrong."}]},{"t":"p","runs":[{"text":"Write the model name and revision into the file too. Vectors from two different models are not comparable and there is nothing in the array to tell you which produced it."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Encode 1,000 sentences of your own, save them as float16, reload, and search with a matrix multiply. Time the search. It will be a few milliseconds, and that number is why most projects never need a vector database."}]}],"check":{"question":[{"text":"A team truncates 768-dimensional vectors from an ordinary embedding model to 256 dimensions to save memory, and retrieval quality collapses. Why did it work for a colleague using a different model?"}],"options":[{"text":[{"text":"The colleague normalised vectors after truncating, which restores the geometry that truncation distorts."}],"correct":false,"tempting":[{"text":"Normalisation rescales but cannot recover information discarded with the dropped dimensions."}]},{"text":[{"text":"The colleague’s model has fewer dimensions to begin with, so proportionally less information was lost."}],"correct":false,"tempting":[{"text":"Assumes the loss scales with the proportion cut rather than with how the model was trained."}]},{"text":[{"text":"The colleague’s model was trained so its early dimensions carry most of the information."}],"correct":true,"tempting":null},{"text":[{"text":"Truncation requires re-encoding the corpus at the shorter length, which the colleague did and the team did not."}],"correct":false,"tempting":[{"text":"Matryoshka truncation is applied to existing vectors; re-encoding is not what makes it work."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"what-an-embedding-is","title":"A list of numbers where near means similar","minutes":9},"next":{"slug":"choosing-an-embedding-model","title":"Picking the model, and reading MTEB without being fooled","minutes":9}}}