{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"embedding-space-in-practice","title":"Working in embedding space, with free tools","url":"https://addaly.com/learn/how-llms-work/embedding-space-in-practice","position":6,"of":78,"minutes":10,"answer":"Semantic search is three operations (embed, normalise, take the dot product) and building it once teaches you more about what embeddings can and cannot do than any explanation.","keyIdea":"Semantic search is three operations (embed, normalise, take the dot product) and building it once teaches you more about what embeddings can and cannot do than any explanation."},"body":{"format":"markdown","text":"## Build the thing, then argue about it\n\nYou have the idea from the previous lesson: meaning is a direction. This lesson is the practical half, because the limits of embeddings only become obvious when you have watched them fail on your own data.\n\nEverything here runs offline on a laptop with no GPU. `sentence-transformers` is free and open source, the small models are a few hundred megabytes, and a corpus of a thousand documents embeds in under a minute on a CPU.\n\n```bash\npip install sentence-transformers numpy\n```\n\n```python\nfrom sentence_transformers import SentenceTransformer\nimport numpy as np\n\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")   # 384 dimensions, ~80 MB\n\ndocs = [\n    \"Refunds are issued within 7 working days of approval.\",\n    \"Our delivery partner covers Pune, Nashik and Aurangabad.\",\n    \"To reset your password, use the link on the sign-in page.\",\n    \"Orders above 2,000 rupees ship free.\",\n]\nD = model.encode(docs, normalize_embeddings=True)  # shape (4, 384)\n\ndef search(q, k=2):\n    v = model.encode([q], normalize_embeddings=True)[0]\n    scores = D @ v                      # cosine similarity, because rows are unit length\n    order = np.argsort(-scores)[:k]\n    return [(round(float(scores[i]), 3), docs[i]) for i in order]\n\nprint(search(\"my money hasn't come back yet\"))\n```\n\nThe query shares no words with the refund line and still retrieves it. That is the entire trick behind semantic search, and it is four lines once the vectors exist.\n\n## Normalise, then dot product\n\nNote `normalize_embeddings=True`. Once every vector has length 1, cosine similarity *is* the dot product, so a search over a million documents becomes one matrix multiplication. This is why vector databases are fast: they are not doing anything clever with meaning, they are doing linear algebra with good indexing.\n\nFor up to a few hundred thousand documents you do not need a vector database at all. A NumPy array in memory and one `@` is faster than most people expect and has no operational cost.\n\n## Four failure modes to see for yourself\n\n**Negation is nearly invisible.** Embed \"the medicine is safe for children\" and \"the medicine is not safe for children\". They will typically score above 0.85. Cosine similarity measures what the sentence is *about*, not what it asserts. Retrieval systems built on embeddings alone will happily hand a model the opposite of what it needed, and the model, having no other source, will use it.\n\n**Length distorts.** Very short queries and very long documents compare badly. Chunking documents into passages of a few hundred tokens is not a stylistic choice; it is how you keep the comparison meaningful.\n\n**Domain shift is brutal.** A general model has never seen your internal product codes. \"ERR_4412 on the DX2 unit\" is nearly meaningless to it, and its neighbours will be arbitrary. This is where keyword search still wins, and why serious systems combine both.\n\n**Spaces are not comparable.** Vectors from `all-MiniLM-L6-v2` mean nothing to another provider's model. Changing embedding model means re-embedding every document you have stored. Budget for it.\n\n## The fix that actually works: hybrid and rerank\n\nThe standard production shape, and it is not complicated:\n\n1. Retrieve 50 candidates by embedding similarity.\n2. Retrieve 50 more by keyword search: BM25, available free in `rank_bm25` or any search engine.\n3. Merge the two lists.\n4. Score the merged set with a **cross-encoder**, a small model that reads the query and the passage *together* rather than comparing two independent vectors.\n5. Keep the top 5.\n\nThe cross-encoder is the step that recovers negation and fine distinctions, because it does not compress each side into one vector before comparing. It is slower (it must run once per candidate), which is exactly why you use it on 100 candidates rather than a million documents. `cross-encoder/ms-marco-MiniLM-L-6-v2` is free and runs on CPU.\n\n## What the vector is not\n\nAn embedding is not a summary, not a compression you can decode, and not evidence. You cannot read a vector to recover the text. You cannot compare a similarity score across two different corpora and conclude anything. And a score of 0.83 has no absolute meaning at all: only the ranking matters, which is why thresholds tuned on one dataset break on the next.\n\nBuild the small version first, on fifty of your own documents, and look at the ten worst results by hand. That habit is worth more than any published benchmark, and it is the same discipline the evaluation course teaches for everything else."},"check":{"question":"A retrieval system returns the passage \"This treatment is not recommended for patients under 12\" for the query \"treatments recommended for children\". Why does embedding search do this?","options":[{"text":"The embedding model was trained mainly on general web text and has weak medical coverage.","correct":false,"tempting":"Domain shift is a genuine failure mode, so blaming coverage usually pays off, but this sentence is ordinary English that any model represents well."},{"text":"Cosine similarity scores what a sentence is about, and the negation changes almost nothing about the topic direction.","correct":true},{"text":"The passage was chunked badly, splitting the negation away from its subject.","correct":false,"tempting":"Chunking errors do cause retrieval nonsense, and the fix is easy, which makes this an attractive first guess."},{"text":"The query was too short to produce a well-formed vector.","correct":false,"tempting":"Short queries really do compare poorly against long documents, so length is a reasonable suspicion here."}]},"prev":{"slug":"meaning-as-direction","title":"Embeddings: meaning as a direction","url":"https://addaly.com/learn/how-llms-work/meaning-as-direction"},"next":{"slug":"position-and-order","title":"How the model knows what order the words were in","url":"https://addaly.com/learn/how-llms-work/position-and-order"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Working in embedding space, with free tools', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/embedding-space-in-practice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Working in embedding space, with free tools","courseTitle":"How a Language Model Actually Works","module":{"index":1,"slug":"from-characters-to-vectors","title":"From characters to vectors"},"answer":null,"keyIdea":"Semantic search is three operations (embed, normalise, take the dot product) and building it once teaches you more about what embeddings can and cannot do than any explanation.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"build-the-thing-then-argue-about-it","runs":[{"text":"Build the thing, then argue about it"}]},{"t":"p","runs":[{"text":"You have the idea from the previous lesson: meaning is a direction. This lesson is the practical half, because the limits of embeddings only become obvious when you have watched them fail on your own data."}]},{"t":"p","runs":[{"text":"Everything here runs offline on a laptop with no GPU. "},{"text":"sentence-transformers","code":true},{"text":" is free and open source, the small models are a few hundred megabytes, and a corpus of a thousand documents embeds in under a minute on a CPU."}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install sentence-transformers numpy"},{"t":"code","lang":"python","label":"Python","text":"from sentence_transformers import SentenceTransformer\nimport numpy as np\n\nmodel = SentenceTransformer(\"all-MiniLM-L6-v2\")   # 384 dimensions, ~80 MB\n\ndocs = [\n    \"Refunds are issued within 7 working days of approval.\",\n    \"Our delivery partner covers Pune, Nashik and Aurangabad.\",\n    \"To reset your password, use the link on the sign-in page.\",\n    \"Orders above 2,000 rupees ship free.\",\n]\nD = model.encode(docs, normalize_embeddings=True)  # shape (4, 384)\n\ndef search(q, k=2):\n    v = model.encode([q], normalize_embeddings=True)[0]\n    scores = D @ v                      # cosine similarity, because rows are unit length\n    order = np.argsort(-scores)[:k]\n    return [(round(float(scores[i]), 3), docs[i]) for i in order]\n\nprint(search(\"my money hasn't come back yet\"))"},{"t":"p","runs":[{"text":"The query shares no words with the refund line and still retrieves it. That is the entire trick behind semantic search, and it is four lines once the vectors exist."}]},{"t":"h2","id":"normalise-then-dot-product","runs":[{"text":"Normalise, then dot product"}]},{"t":"p","runs":[{"text":"Note "},{"text":"normalize_embeddings=True","code":true},{"text":". Once every vector has length 1, cosine similarity "},{"text":"is","i":true},{"text":" the dot product, so a search over a million documents becomes one matrix multiplication. This is why vector databases are fast: they are not doing anything clever with meaning, they are doing linear algebra with good indexing."}]},{"t":"p","runs":[{"text":"For up to a few hundred thousand documents you do not need a vector database at all. A NumPy array in memory and one "},{"text":"@","code":true},{"text":" is faster than most people expect and has no operational cost."}]},{"t":"h2","id":"four-failure-modes-to-see-for-yourself","runs":[{"text":"Four failure modes to see for yourself"}]},{"t":"p","runs":[{"text":"Negation is nearly invisible.","b":true},{"text":" Embed “the medicine is safe for children” and “the medicine is not safe for children”. They will typically score above 0.85. Cosine similarity measures what the sentence is "},{"text":"about","i":true},{"text":", not what it asserts. Retrieval systems built on embeddings alone will happily hand a model the opposite of what it needed, and the model, having no other source, will use it."}]},{"t":"p","runs":[{"text":"Length distorts.","b":true},{"text":" Very short queries and very long documents compare badly. Chunking documents into passages of a few hundred tokens is not a stylistic choice; it is how you keep the comparison meaningful."}]},{"t":"p","runs":[{"text":"Domain shift is brutal.","b":true},{"text":" A general model has never seen your internal product codes. “ERR_4412 on the DX2 unit” is nearly meaningless to it, and its neighbours will be arbitrary. This is where keyword search still wins, and why serious systems combine both."}]},{"t":"p","runs":[{"text":"Spaces are not comparable.","b":true},{"text":" Vectors from "},{"text":"all-MiniLM-L6-v2","code":true},{"text":" mean nothing to another provider’s model. Changing embedding model means re-embedding every document you have stored. Budget for it."}]},{"t":"h2","id":"the-fix-that-actually-works-hybrid-and-rerank","runs":[{"text":"The fix that actually works: hybrid and rerank"}]},{"t":"p","runs":[{"text":"The standard production shape, and it is not complicated:"}]},{"t":"ol","start":1,"items":[[{"text":"Retrieve 50 candidates by embedding similarity."}],[{"text":"Retrieve 50 more by keyword search: BM25, available free in "},{"text":"rank_bm25","code":true},{"text":" or any search engine."}],[{"text":"Merge the two lists."}],[{"text":"Score the merged set with a "},{"text":"cross-encoder","b":true},{"text":", a small model that reads the query and the passage "},{"text":"together","i":true},{"text":" rather than comparing two independent vectors."}],[{"text":"Keep the top 5."}]]},{"t":"p","runs":[{"text":"The cross-encoder is the step that recovers negation and fine distinctions, because it does not compress each side into one vector before comparing. It is slower (it must run once per candidate), which is exactly why you use it on 100 candidates rather than a million documents. "},{"text":"cross-encoder/ms-marco-MiniLM-L-6-v2","code":true},{"text":" is free and runs on CPU."}]},{"t":"h2","id":"what-the-vector-is-not","runs":[{"text":"What the vector is not"}]},{"t":"p","runs":[{"text":"An embedding is not a summary, not a compression you can decode, and not evidence. You cannot read a vector to recover the text. You cannot compare a similarity score across two different corpora and conclude anything. And a score of 0.83 has no absolute meaning at all: only the ranking matters, which is why thresholds tuned on one dataset break on the next."}]},{"t":"p","runs":[{"text":"Build the small version first, on fifty of your own documents, and look at the ten worst results by hand. That habit is worth more than any published benchmark, and it is the same discipline the evaluation course teaches for everything else."}]}],"check":{"question":[{"text":"A retrieval system returns the passage “This treatment is not recommended for patients under 12” for the query “treatments recommended for children”. Why does embedding search do this?"}],"options":[{"text":[{"text":"The embedding model was trained mainly on general web text and has weak medical coverage."}],"correct":false,"tempting":[{"text":"Domain shift is a genuine failure mode, so blaming coverage usually pays off, but this sentence is ordinary English that any model represents well."}]},{"text":[{"text":"Cosine similarity scores what a sentence is about, and the negation changes almost nothing about the topic direction."}],"correct":true,"tempting":null},{"text":[{"text":"The passage was chunked badly, splitting the negation away from its subject."}],"correct":false,"tempting":[{"text":"Chunking errors do cause retrieval nonsense, and the fix is easy, which makes this an attractive first guess."}]},{"text":[{"text":"The query was too short to produce a well-formed vector."}],"correct":false,"tempting":[{"text":"Short queries really do compare poorly against long documents, so length is a reasonable suspicion here."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"meaning-as-direction","title":"Embeddings: meaning as a direction","minutes":7},"next":{"slug":"position-and-order","title":"How the model knows what order the words were in","minutes":8}}}