{"course":{"slug":"maths-you-need","title":"The Maths You Actually Need","url":"https://addaly.com/learn/maths-you-need","level":"intermediate","track":"deep"},"lesson":{"slug":"dot-products","title":"What a dot product measures","url":"https://addaly.com/learn/maths-you-need/dot-products","position":3,"of":76,"minutes":7,"answer":"A dot product scores how much two lists agree, but it also grows with their length, so normalise before you compare.","keyIdea":"A dot product scores how much two lists agree, but it also grows with their length, so normalise before you compare."},"body":{"format":"markdown","text":"## Multiply the matching slots, add them up\n\nTwo lists of the same length. Multiply slot 1 by slot 1, slot 2 by slot 2, and so on, then add every result into one number.\n\n```python\na = [2, 0, 3]\nb = [1, 5, -1]\ndot = 2*1 + 0*5 + 3*-1   # 2 + 0 - 3 = -1\n```\n\nOne number out. That number is a score of how much the two lists agree.\n\nRead the sign first. Positive means the two vectors mostly lean the same way. Zero means they are unrelated in the specific sense of being at right angles. Negative means they lean opposite. In the example above, `a` and `b` disagree slightly.\n\n## Where the number comes from\n\nThe dot product equals `|a| × |b| × cos(θ)`: the length of the first, times the length of the second, times the cosine of the angle between them.\n\nYou do not need to work with that formula. You need one consequence of it: a dot product mixes two different things together. It is large when the vectors agree, and it is also large when the vectors are long. Length can drown out agreement.\n\nDivide the dot product by both lengths and the lengths cancel out. What remains is `cos(θ)`, a number from -1 to 1 that depends on angle alone. That is cosine similarity, and it is a dot product with the size effect removed.\n\n> Figure: The same two documents, ranked twice against one query. Ranked by raw dot product: Product manual, 6,000 words: score 9.6; Help page, 180 words: score 4.1; The manual wins because its vector is long, not because it answers; The top result mentions everything and settles nothing. Ranked by cosine, both lengths divided out: Help page, 180 words: score 0.71; Product manual, 6,000 words: score 0.34; Only the angle survives the division; The top result is the page that answers the question. Dividing by both lengths turns a dot product into a cosine and removes the size effect. Normalising the stored vectors but forgetting the query is the commonest bug in a first search system, and it never raises an error.\n\n```python\ndef cosine(a, b):\n    dot = sum(x*y for x, y in zip(a, b))\n    na = sum(x*x for x in a) ** 0.5\n    nb = sum(y*y for y in b) ** 0.5\n    return dot / (na * nb)\n```\n\n## Where models use it\n\nAlmost every number in a model that answers \"how relevant is this to that\" is a dot product.\n\n**Attention.** Each token produces a query vector, each token produces a key vector. The attention score between two tokens is the dot product of one query and one key. A big score means \"this token should look at that one\". When people say attention is expensive, this is why: for 4000 tokens, that is 16 million dot products, per head, per layer.\n\n**Retrieval.** Your question becomes a vector, every document is already a vector, and the search is a dot product against each one. Vector databases exist to do that fast.\n\n**The final layer.** A language model ends with a hidden state, one vector. To score the next token, it dot-products that state against one row per vocabulary item. 50,000 dot products, 50,000 scores, softmax over them, and you have the next-token distribution.\n\n## The honest limits\n\nCosine similarity measures whether two pieces of text get used in similar contexts. That is not the same as meaning the same thing, and people forget it constantly.\n\nAntonyms score high. \"Hot\" and \"cold\" appear in near-identical sentences, so their vectors are close. \"I approved the loan\" and \"I rejected the loan\" are near neighbours. If you are building a retrieval system where the difference between approved and rejected matters, and in banking, health, or law it always matters, cosine similarity alone will hurt you, and it will do it quietly, with high confidence scores.\n\nThe second limit is the one from the formula. If you skip normalisation and rank by raw dot product, long documents win because long documents have long vectors. Your top result is then a 6000-word page that mentions everything, not the 200-word page that answers the question. This is one of the most common bugs in a first search system, and it never throws an error."},"check":{"question":"A search over unnormalised embeddings keeps putting very long documents at the top, no matter what the query is. What is going on?","options":[{"text":"Long documents contain more of the query's words, so they genuinely are the better matches.","correct":false,"tempting":"Assumes the score reflects real relevance rather than an artefact of vector length."},{"text":"A dot product counts how many terms the query and document share, and longer text shares more terms.","correct":false,"tempting":"Thinks a dot product is term overlap, so more words mechanically means a higher score."},{"text":"A dot product grows with vector length, and long documents produce longer vectors, so size is outvoting angle.","correct":true},{"text":"The embedding model was trained mostly on long text and is biased, so it needs retraining.","correct":false,"tempting":"Blames the model when the scoring function, not the model, is doing the damage."}]},"prev":{"slug":"vector-is-a-list","title":"A vector is just a list","url":"https://addaly.com/learn/maths-you-need/vector-is-a-list"},"next":{"slug":"norms-and-distance","title":"How long, and how far","url":"https://addaly.com/learn/maths-you-need/norms-and-distance"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What a dot product measures', in The Maths You Actually Need. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/maths-you-need/dot-products (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What a dot product measures","courseTitle":"The Maths You Actually Need","module":{"index":1,"slug":"vectors-and-space","title":"Vectors, and the space they live in"},"answer":null,"keyIdea":"A dot product scores how much two lists agree, but it also grows with their length, so normalise before you compare.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Ranked by raw dot product","items":["Product manual, 6,000 words: score 9.6","Help page, 180 words: score 4.1","The manual wins because its vector is long, not because it answers","The top result mentions everything and settles nothing"]},"type":"compare","right":{"title":"Ranked by cosine, both lengths divided out","items":["Help page, 180 words: score 0.71","Product manual, 6,000 words: score 0.34","Only the angle survives the division","The top result is the page that answers the question"]},"title":"The same two documents, ranked twice against one query","caption":"Dividing by both lengths turns a dot product into a cosine and removes the size effect. Normalising the stored vectors but forgetting the query is the commonest bug in a first search system, and it never raises an error."},"alt":"Figure: The same two documents, ranked twice against one query. Ranked by raw dot product: Product manual, 6,000 words: score 9.6; Help page, 180 words: score 4.1; The manual wins because its vector is long, not because it answers; The top result mentions everything and settles nothing. Ranked by cosine, both lengths divided out: Help page, 180 words: score 0.71; Product manual, 6,000 words: score 0.34; Only the angle survives the division; The top result is the page that answers the question. Dividing by both lengths turns a dot product into a cosine and removes the size effect. Normalising the stored vectors but forgetting the query is the commonest bug in a first search system, and it never raises an error."}},"blocks":[{"t":"h2","id":"multiply-the-matching-slots-add-them-up","runs":[{"text":"Multiply the matching slots, add them up"}]},{"t":"p","runs":[{"text":"Two lists of the same length. Multiply slot 1 by slot 1, slot 2 by slot 2, and so on, then add every result into one number."}]},{"t":"code","lang":"python","label":"Python","text":"a = [2, 0, 3]\nb = [1, 5, -1]\ndot = 2*1 + 0*5 + 3*-1   # 2 + 0 - 3 = -1"},{"t":"p","runs":[{"text":"One number out. That number is a score of how much the two lists agree."}]},{"t":"p","runs":[{"text":"Read the sign first. Positive means the two vectors mostly lean the same way. Zero means they are unrelated in the specific sense of being at right angles. Negative means they lean opposite. In the example above, "},{"text":"a","code":true},{"text":" and "},{"text":"b","code":true},{"text":" disagree slightly."}]},{"t":"h2","id":"where-the-number-comes-from","runs":[{"text":"Where the number comes from"}]},{"t":"p","runs":[{"text":"The dot product equals "},{"text":"|a| × |b| × cos(θ)","code":true},{"text":": the length of the first, times the length of the second, times the cosine of the angle between them."}]},{"t":"p","runs":[{"text":"You do not need to work with that formula. You need one consequence of it: a dot product mixes two different things together. It is large when the vectors agree, and it is also large when the vectors are long. Length can drown out agreement."}]},{"t":"p","runs":[{"text":"Divide the dot product by both lengths and the lengths cancel out. What remains is "},{"text":"cos(θ)","code":true},{"text":", a number from -1 to 1 that depends on angle alone. That is cosine similarity, and it is a dot product with the size effect removed."}]},{"t":"code","lang":"python","label":"Python","text":"def cosine(a, b):\n    dot = sum(x*y for x, y in zip(a, b))\n    na = sum(x*x for x in a) ** 0.5\n    nb = sum(y*y for y in b) ** 0.5\n    return dot / (na * nb)"},{"t":"h2","id":"where-models-use-it","runs":[{"text":"Where models use it"}]},{"t":"p","runs":[{"text":"Almost every number in a model that answers “how relevant is this to that” is a dot product."}]},{"t":"p","runs":[{"text":"Attention.","b":true},{"text":" Each token produces a query vector, each token produces a key vector. The attention score between two tokens is the dot product of one query and one key. A big score means “this token should look at that one”. When people say attention is expensive, this is why: for 4000 tokens, that is 16 million dot products, per head, per layer."}]},{"t":"p","runs":[{"text":"Retrieval.","b":true},{"text":" Your question becomes a vector, every document is already a vector, and the search is a dot product against each one. Vector databases exist to do that fast."}]},{"t":"p","runs":[{"text":"The final layer.","b":true},{"text":" A language model ends with a hidden state, one vector. To score the next token, it dot-products that state against one row per vocabulary item. 50,000 dot products, 50,000 scores, softmax over them, and you have the next-token distribution."}]},{"t":"h2","id":"the-honest-limits","runs":[{"text":"The honest limits"}],"role":"limit"},{"t":"p","runs":[{"text":"Cosine similarity measures whether two pieces of text get used in similar contexts. That is not the same as meaning the same thing, and people forget it constantly."}]},{"t":"p","runs":[{"text":"Antonyms score high. “Hot” and “cold” appear in near-identical sentences, so their vectors are close. “I approved the loan” and “I rejected the loan” are near neighbours. If you are building a retrieval system where the difference between approved and rejected matters, and in banking, health, or law it always matters, cosine similarity alone will hurt you, and it will do it quietly, with high confidence scores."}]},{"t":"p","runs":[{"text":"The second limit is the one from the formula. If you skip normalisation and rank by raw dot product, long documents win because long documents have long vectors. Your top result is then a 6000-word page that mentions everything, not the 200-word page that answers the question. This is one of the most common bugs in a first search system, and it never throws an error."}]}],"check":{"question":[{"text":"A search over unnormalised embeddings keeps putting very long documents at the top, no matter what the query is. What is going on?"}],"options":[{"text":[{"text":"Long documents contain more of the query’s words, so they genuinely are the better matches."}],"correct":false,"tempting":[{"text":"Assumes the score reflects real relevance rather than an artefact of vector length."}]},{"text":[{"text":"A dot product counts how many terms the query and document share, and longer text shares more terms."}],"correct":false,"tempting":[{"text":"Thinks a dot product is term overlap, so more words mechanically means a higher score."}]},{"text":[{"text":"A dot product grows with vector length, and long documents produce longer vectors, so size is outvoting angle."}],"correct":true,"tempting":null},{"text":[{"text":"The embedding model was trained mostly on long text and is biased, so it needs retraining."}],"correct":false,"tempting":[{"text":"Blames the model when the scoring function, not the model, is doing the damage."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"vector-is-a-list","title":"A vector is just a list","minutes":7},"next":{"slug":"norms-and-distance","title":"How long, and how far","minutes":8}}}