{"course":{"slug":"python-for-ai","title":"Python, from Zero, for AI","url":"https://addaly.com/learn/python-for-ai","level":"beginner","track":"build"},"lesson":{"slug":"vectors-and-cosine-similarity","title":"Dot products and cosine similarity: one vector against a hundred thousand in a single line","url":"https://addaly.com/learn/python-for-ai/vectors-and-cosine-similarity","position":74,"of":89,"minutes":9,"answer":"Cosine similarity is the dot product of two unit-length vectors, so normalise the matrix once and every query becomes one matrix-vector product; at a hundred thousand rows that is milliseconds.","keyIdea":"Cosine similarity is the dot product of two unit-length vectors, so normalise the matrix once and every query becomes one matrix-vector product; at a hundred thousand rows that is milliseconds, and the point at which it stops being milliseconds is the point at which an index becomes worth its complexity."},"body":{"format":"markdown","text":"## An embedding is a row of floats\n\nAsk an embedding model for a vector and you get back a list of numbers: 384 for a small open model, 1,536 or 3,072 for the hosted ones. What the numbers *mean* is the business of the machine learning course; here they are a `(768,)` float32 array, and a corpus of a hundred thousand documents is a `(100000, 768)` matrix. Everything below is arithmetic on that matrix.\n\n```python\nq = np.asarray(embed(\"how do I reset my password\"), dtype=np.float32)   # (768,)\nM = np.load(\"embeddings.npy\")                                            # (100000, 768)\n```\n\n`np.save`/`np.load` write the raw array with a small header: the fastest way to persist one, and much smaller than JSON.\n\n## Dot product\n\nThe dot product of two vectors multiplies them elementwise and sums:\n\n```python\na @ b                    # same as np.dot(a, b), same as (a * b).sum()\n```\n\n`@` is the matrix-multiplication operator, and for a matrix against a vector it computes the dot product of every row with the vector at once:\n\n```python\nM @ q                    # (100000, 768) @ (768,) → (100000,)\n```\n\nOne line, one compiled loop, one hundred thousand dot products. Compare:\n\n```python\n[np.dot(row, q) for row in M]     # ~400 ms\nM @ q                             # ~3 ms\n```\n\nSame numbers, same arithmetic, different plumbing. The loop pays Python overhead per row; the matrix product hands the whole block to a compiled routine that reads memory in order and uses the CPU's vector units: the previous lessons in one measurement.\n\n> Figure: Scoring one query against every document. Line chart of Milliseconds per query against Documents in the index: Python loop, np.dot per row (10000, 62) (50000, 310) (100000, 620) (200000, 1240) (400000, 2480); M_unit @ q, one line (10000, 0.8) (50000, 4) (100000, 8) (200000, 16) (400000, 32). Both lines are straight: the arithmetic is rows times dimensions either way. What differs is the constant. The loop pays Python's per-row overhead a hundred thousand times; the matrix product hands the whole block to a compiled routine once. Brute force stays honest to a few hundred thousand rows, and an index earns its complexity after that.\n\n## Cosine similarity\n\nThe dot product grows with the lengths of the vectors as well as their alignment. Cosine similarity removes the length:\n\n```python\ndef cosine(a, b):\n    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))\n```\n\nIt is the cosine of the angle between them: 1 for the same direction, 0 for perpendicular, −1 for opposite. Embedding models are used with cosine because a document's vector length carries little meaning while its direction carries the content.\n\nDoing the division per query is wasteful. Normalise each row of `M` to unit length **once**, when you build it:\n\n```python\nnorms = np.linalg.norm(M, axis=1, keepdims=True)    # (100000, 1)\nM_unit = M / norms                                  # every row now has length 1\n```\n\nThen every query is:\n\n```python\nq_unit = q / np.linalg.norm(q)\nsims = M_unit @ q_unit                              # (100000,) of cosines\n```\n\nA dot product of two unit vectors *is* their cosine. The `keepdims=True` is the broadcasting lesson paying off: `(100000, 768) / (100000, 1)` divides each row by its own norm.\n\nGuard the zero vector: a document that embedded to all zeros has norm 0 and produces `nan` everywhere. `norms[norms == 0] = 1` before dividing, or filter such rows out and log them.\n\n## Top k\n\n```python\nk = 5\ntop = np.argpartition(sims, -k)[-k:]         # indices of the k largest, unordered\ntop = top[np.argsort(sims[top])[::-1]]       # sort just those k\nfor i in top:\n    print(f\"{sims[i]:.3f}  {docs[i][:60]}\")\n```\n\n`argpartition` finds the k largest in linear time without sorting all hundred thousand; the second line orders only the five. For a hundred thousand rows `argsort` on everything is fine too (it is a few milliseconds) but the habit matters once the corpus is tens of millions.\n\n## Many queries at once\n\nA batch of queries is a matrix too, and the matrix-matrix product gives every pairwise similarity:\n\n```python\nQ_unit = Q / np.linalg.norm(Q, axis=1, keepdims=True)   # (50, 768)\nS = Q_unit @ M_unit.T                                   # (50, 100000)\n```\n\nRow `i` of `S` is the similarity of query `i` to every document. The `.T` is a view, so nothing is copied. Fifty queries cost little more than one, because the compiled routine is doing the same memory pass with more arithmetic per element loaded.\n\n## Where brute force stops\n\nThe arithmetic is `rows × dimensions` multiply-adds per query: 100,000 × 768 ≈ 77 million, a few milliseconds on one core. At ten million rows it is 7.7 billion, around a second, and the matrix is 30 GB in float32: more than most machines hold. That is the point where an approximate index (FAISS, HNSW via `hnswlib`, or a vector database) earns its complexity by trading a little recall for a thousand-fold speed-up. Below a million rows, `M_unit @ q` in NumPy is simpler, exact, and fast enough, and that covers nearly every project a person builds alone.\n\nA cheap intermediate step: store `M_unit` as float16 to halve memory, and upcast the query result. The precision lesson said float16 keeps three figures; for ranking similarities that is usually enough, and it is a measurable decision rather than a guess.\n\n## Try this now\n\nMake a random `(100000, 768)` float32 matrix with `default_rng`, normalise its rows, and time `M_unit @ q` against the Python loop. Then plant a known vector at row 4,321, query with a slightly noisy copy of it, and confirm that row comes out on top."},"check":{"question":"A developer has a `(100000, 768)` float32 matrix of embeddings and computes cosine similarity for each query with a Python loop over rows calling `np.dot` and two `np.linalg.norm` calls per row. It takes about 400 ms per query. What single change brings it to a few milliseconds?","options":[{"text":"Convert the matrix to float64 so the dot products can use the CPU's wider registers.","correct":false,"tempting":"Wider floats double the memory traffic and do not remove the Python loop, which is where the time goes."},{"text":"Store the matrix as a list of arrays so that each row is a separate contiguous block that `np.dot` can read faster, since a two-dimensional array interleaves rows in memory.","correct":false,"tempting":"Rows of a C-contiguous matrix are already contiguous; splitting into a list adds Python overhead rather than removing it."},{"text":"Sort the rows by norm before looping so the loop can terminate early once similarities start to fall.","correct":false,"tempting":"Similarity does not decrease with norm, so no early stop is valid, and the loop would still be a loop."},{"text":"Normalise the rows once, normalise the query, and compute `M @ q` as one matrix-vector product instead of a hundred thousand Python-level calls.","correct":true}]},"prev":{"slug":"masks-views-and-copies","title":"Masks, where, and the slice that is not a copy","url":"https://addaly.com/learn/python-for-ai/masks-views-and-copies"},"next":{"slug":"pandas-dataframes","title":"pandas: a table with named columns, and the four things to look at first","url":"https://addaly.com/learn/python-for-ai/pandas-dataframes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Source and assessment review continues. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Dot products and cosine similarity: one vector against a hundred thousand in a single line', in Python, from Zero, for AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/python-for-ai/vectors-and-cosine-similarity (Accessed: 21 November 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Dot products and cosine similarity: one vector against a hundred thousand in a single line","courseTitle":"Python, from Zero, for AI","module":{"index":8,"slug":"numbers-in-bulk","title":"Numbers in bulk: NumPy, pandas and the vector under every model"},"answer":null,"keyIdea":"Cosine similarity is the dot product of two unit-length vectors, so normalise the matrix once and every query becomes one matrix-vector product; at a hundred thousand rows that is milliseconds, and the point at which it stops being milliseconds is the point at which an index becomes worth its complexity.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"caption":"Both lines are straight: the arithmetic is rows times dimensions either way. What differs is the constant. The loop pays Python’s per-row overhead a hundred thousand times; the matrix product hands the whole block to a compiled routine once. Brute force stays honest to a few hundred thousand rows, and an index earns its complexity after that.","series":[{"label":"Python loop, np.dot per row","points":[[10000,62],[50000,310],[100000,620],[200000,1240],[400000,2480]]},{"label":"M_unit @ q, one line","points":[[10000,0.8],[50000,4],[100000,8],[200000,16],[400000,32]]}],"title":"Scoring one query against every document","type":"line","xLabel":"Documents in the index","yLabel":"Milliseconds per query"},"alt":"Figure: Scoring one query against every document. Line chart of Milliseconds per query against Documents in the index: Python loop, np.dot per row (10000, 62) (50000, 310) (100000, 620) (200000, 1240) (400000, 2480); M_unit @ q, one line (10000, 0.8) (50000, 4) (100000, 8) (200000, 16) (400000, 32). Both lines are straight: the arithmetic is rows times dimensions either way. What differs is the constant. The loop pays Python’s per-row overhead a hundred thousand times; the matrix product hands the whole block to a compiled routine once. Brute force stays honest to a few hundred thousand rows, and an index earns its complexity after that."}},"blocks":[{"t":"h2","id":"an-embedding-is-a-row-of-floats","runs":[{"text":"An embedding is a row of floats"}]},{"t":"p","runs":[{"text":"Ask an embedding model for a vector and you get back a list of numbers: 384 for a small open model, 1,536 or 3,072 for the hosted ones. What the numbers "},{"text":"mean","i":true},{"text":" is the business of the machine learning course; here they are a "},{"text":"(768,)","code":true},{"text":" float32 array, and a corpus of a hundred thousand documents is a "},{"text":"(100000, 768)","code":true},{"text":" matrix. Everything below is arithmetic on that matrix."}]},{"t":"code","lang":"python","label":"Python","text":"q = np.asarray(embed(\"how do I reset my password\"), dtype=np.float32)   # (768,)\nM = np.load(\"embeddings.npy\")                                            # (100000, 768)"},{"t":"p","runs":[{"text":"np.save","code":true},{"text":"/"},{"text":"np.load","code":true},{"text":" write the raw array with a small header: the fastest way to persist one, and much smaller than JSON."}]},{"t":"h2","id":"dot-product","runs":[{"text":"Dot product"}]},{"t":"p","runs":[{"text":"The dot product of two vectors multiplies them elementwise and sums:"}]},{"t":"code","lang":"python","label":"Python","text":"a @ b                    # same as np.dot(a, b), same as (a * b).sum()"},{"t":"p","runs":[{"text":"@","code":true},{"text":" is the matrix-multiplication operator, and for a matrix against a vector it computes the dot product of every row with the vector at once:"}]},{"t":"code","lang":"python","label":"Python","text":"M @ q                    # (100000, 768) @ (768,) → (100000,)"},{"t":"p","runs":[{"text":"One line, one compiled loop, one hundred thousand dot products. Compare:"}]},{"t":"code","lang":"python","label":"Python","text":"[np.dot(row, q) for row in M]     # ~400 ms\nM @ q                             # ~3 ms"},{"t":"p","runs":[{"text":"Same numbers, same arithmetic, different plumbing. The loop pays Python overhead per row; the matrix product hands the whole block to a compiled routine that reads memory in order and uses the CPU’s vector units: the previous lessons in one measurement."}]},{"t":"h2","id":"cosine-similarity","runs":[{"text":"Cosine similarity"}]},{"t":"p","runs":[{"text":"The dot product grows with the lengths of the vectors as well as their alignment. Cosine similarity removes the length:"}]},{"t":"code","lang":"python","label":"Python","text":"def cosine(a, b):\n    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))"},{"t":"p","runs":[{"text":"It is the cosine of the angle between them: 1 for the same direction, 0 for perpendicular, −1 for opposite. Embedding models are used with cosine because a document’s vector length carries little meaning while its direction carries the content."}]},{"t":"p","runs":[{"text":"Doing the division per query is wasteful. Normalise each row of "},{"text":"M","code":true},{"text":" to unit length "},{"text":"once","b":true},{"text":", when you build it:"}]},{"t":"code","lang":"python","label":"Python","text":"norms = np.linalg.norm(M, axis=1, keepdims=True)    # (100000, 1)\nM_unit = M / norms                                  # every row now has length 1"},{"t":"p","runs":[{"text":"Then every query is:"}]},{"t":"code","lang":"python","label":"Python","text":"q_unit = q / np.linalg.norm(q)\nsims = M_unit @ q_unit                              # (100000,) of cosines"},{"t":"p","runs":[{"text":"A dot product of two unit vectors "},{"text":"is","i":true},{"text":" their cosine. The "},{"text":"keepdims=True","code":true},{"text":" is the broadcasting lesson paying off: "},{"text":"(100000, 768) / (100000, 1)","code":true},{"text":" divides each row by its own norm."}]},{"t":"p","runs":[{"text":"Guard the zero vector: a document that embedded to all zeros has norm 0 and produces "},{"text":"nan","code":true},{"text":" everywhere. "},{"text":"norms[norms == 0] = 1","code":true},{"text":" before dividing, or filter such rows out and log them."}]},{"t":"h2","id":"top-k","runs":[{"text":"Top k"}]},{"t":"code","lang":"python","label":"Python","text":"k = 5\ntop = np.argpartition(sims, -k)[-k:]         # indices of the k largest, unordered\ntop = top[np.argsort(sims[top])[::-1]]       # sort just those k\nfor i in top:\n    print(f\"{sims[i]:.3f}  {docs[i][:60]}\")"},{"t":"p","runs":[{"text":"argpartition","code":true},{"text":" finds the k largest in linear time without sorting all hundred thousand; the second line orders only the five. For a hundred thousand rows "},{"text":"argsort","code":true},{"text":" on everything is fine too (it is a few milliseconds) but the habit matters once the corpus is tens of millions."}]},{"t":"h2","id":"many-queries-at-once","runs":[{"text":"Many queries at once"}]},{"t":"p","runs":[{"text":"A batch of queries is a matrix too, and the matrix-matrix product gives every pairwise similarity:"}]},{"t":"code","lang":"python","label":"Python","text":"Q_unit = Q / np.linalg.norm(Q, axis=1, keepdims=True)   # (50, 768)\nS = Q_unit @ M_unit.T                                   # (50, 100000)"},{"t":"p","runs":[{"text":"Row "},{"text":"i","code":true},{"text":" of "},{"text":"S","code":true},{"text":" is the similarity of query "},{"text":"i","code":true},{"text":" to every document. The "},{"text":".T","code":true},{"text":" is a view, so nothing is copied. Fifty queries cost little more than one, because the compiled routine is doing the same memory pass with more arithmetic per element loaded."}]},{"t":"h2","id":"where-brute-force-stops","runs":[{"text":"Where brute force stops"}]},{"t":"p","runs":[{"text":"The arithmetic is "},{"text":"rows × dimensions","code":true},{"text":" multiply-adds per query: 100,000 × 768 ≈ 77 million, a few milliseconds on one core. At ten million rows it is 7.7 billion, around a second, and the matrix is 30 GB in float32: more than most machines hold. That is the point where an approximate index (FAISS, HNSW via "},{"text":"hnswlib","code":true},{"text":", or a vector database) earns its complexity by trading a little recall for a thousand-fold speed-up. Below a million rows, "},{"text":"M_unit @ q","code":true},{"text":" in NumPy is simpler, exact, and fast enough, and that covers nearly every project a person builds alone."}]},{"t":"p","runs":[{"text":"A cheap intermediate step: store "},{"text":"M_unit","code":true},{"text":" as float16 to halve memory, and upcast the query result. The precision lesson said float16 keeps three figures; for ranking similarities that is usually enough, and it is a measurable decision rather than a guess."}]},{"t":"h2","id":"try-this-now","runs":[{"text":"Try this now"}],"role":"try"},{"t":"p","runs":[{"text":"Make a random "},{"text":"(100000, 768)","code":true},{"text":" float32 matrix with "},{"text":"default_rng","code":true},{"text":", normalise its rows, and time "},{"text":"M_unit @ q","code":true},{"text":" against the Python loop. Then plant a known vector at row 4,321, query with a slightly noisy copy of it, and confirm that row comes out on top."}]}],"check":{"question":[{"text":"A developer has a "},{"text":"(100000, 768)","code":true},{"text":" float32 matrix of embeddings and computes cosine similarity for each query with a Python loop over rows calling "},{"text":"np.dot","code":true},{"text":" and two "},{"text":"np.linalg.norm","code":true},{"text":" calls per row. It takes about 400 ms per query. What single change brings it to a few milliseconds?"}],"options":[{"text":[{"text":"Convert the matrix to float64 so the dot products can use the CPU’s wider registers."}],"correct":false,"tempting":[{"text":"Wider floats double the memory traffic and do not remove the Python loop, which is where the time goes."}]},{"text":[{"text":"Store the matrix as a list of arrays so that each row is a separate contiguous block that "},{"text":"np.dot","code":true},{"text":" can read faster, since a two-dimensional array interleaves rows in memory."}],"correct":false,"tempting":[{"text":"Rows of a C-contiguous matrix are already contiguous; splitting into a list adds Python overhead rather than removing it."}]},{"text":[{"text":"Sort the rows by norm before looping so the loop can terminate early once similarities start to fall."}],"correct":false,"tempting":[{"text":"Similarity does not decrease with norm, so no early stop is valid, and the loop would still be a loop."}]},{"text":[{"text":"Normalise the rows once, normalise the query, and compute "},{"text":"M @ q","code":true},{"text":" as one matrix-vector product instead of a hundred thousand Python-level calls."}],"correct":true,"tempting":null}],"order":[2,1,3,0],"why":null},"prev":{"slug":"masks-views-and-copies","title":"Masks, where, and the slice that is not a copy","minutes":8},"next":{"slug":"pandas-dataframes","title":"pandas: a table with named columns, and the four things to look at first","minutes":9}}}