{"course":{"slug":"agents-and-automation","title":"Agents and Automation","url":"https://addaly.com/learn/agents-and-automation","level":"intermediate","track":"build"},"lesson":{"slug":"deduplication-and-matching","title":"The same customer, three times","url":"https://addaly.com/learn/agents-and-automation/deduplication-and-matching","position":25,"of":72,"minutes":9,"answer":"Most duplicate problems are solved by normalising a key rather than by fuzzy matching, and a wrong merge is far more expensive than a surviving duplicate because merges are hard to undo.","keyIdea":"Most duplicate problems are solved by normalising a key rather than by fuzzy matching, and a wrong merge is far more expensive than a surviving duplicate because merges are hard to undo."},"body":{"format":"markdown","text":"## Where duplicates come from\n\nThree sources, and they need different answers. A retry that timed out after the write succeeded creates an exact duplicate, and the fix is idempotency at the write, covered later in this block. Two systems describing the same real thing (the CRM's \"Acme Ltd\" and the accounts system's \"ACME LIMITED\") need matching. And a human typing the same customer in twice on a Tuesday needs a constraint at the point of entry, which is a product decision rather than a data one.\n\nKnow which you have before choosing a technique, because a fuzzy matcher pointed at a retry bug is an expensive way to not fix the bug.\n\n## Normalise a key before you reach for similarity\n\nThe great majority of real deduplication is solved without any matching algorithm at all. Pick a field that identifies the thing, normalise it hard, and group on it.\n\n- Email: lowercase, trim. For Gmail specifically, dots in the local part are ignored and `+tag` suffixes are aliases: worth handling if consumer emails matter to you, and worth *not* generalising to other providers, where those rules do not hold.\n- Phone: normalise to E.164 with the country code (`libphonenumber` does this properly and free; hand-rolled regex does not survive contact with real data).\n- Company: strip legal suffixes, punctuation and case: `acme` from `Acme Ltd.`, `ACME LIMITED`, `Acme Limited`.\n- Tax identifier, registration number, IBAN, national ID where you have one and are allowed to use it. These are the strongest keys available and they end the discussion.\n\nDo this first. It is cheap, exact, and explains itself in an audit, which fuzzy matching never quite does.\n\n## When there is no key: block, then compare\n\nComparing every record with every other is quadratic. Fifty thousand records is about 1.25 billion pairs, which is a job that runs overnight and gets skipped.\n\n**Blocking** fixes it. Only compare records that share a cheap coarse key: same postcode, same first three letters of the surname, same first token of the company name, same year of birth. Fifty thousand records split into two thousand blocks of twenty-five is roughly six hundred thousand comparisons: seconds instead of hours.\n\nBlocking trades recall for speed: a pair that disagrees on the blocking key is never compared, so choose a key that is unlikely to be wrong on both records. Run two or three different blocking passes and take the union, which recovers most of what any single pass misses.\n\nWithin a block, score with a string similarity. **Jaro-Winkler** is well suited to names because it weights agreement at the start of the string. **Levenshtein** distance suits short codes and typos. A token-set ratio handles reordering, so that \"Kumar Rajesh\" and \"Rajesh Kumar\" score high. **RapidFuzz** in Python is fast and free; the `recordlinkage` and `dedupe` libraries build the whole pipeline if you want blocking and scoring together.\n\n## Choose the threshold by asymmetry, not by the F1 score\n\nTwo records left unmerged means somebody sees a customer twice: irritating, visible, fixable in ten seconds. Two different people merged into one record means one person can see the other's orders, invoices and address. That is a data-protection incident, it is discovered late, and unpicking a merge after downstream systems have copied the merged record is hard.\n\n> Figure: The two ways a matcher is wrong, and what each costs. A two by two grid, rows Merged and Left apart, columns Same person and Different people: Merged and Same person: What you wanted; Merged and Different people: A data-protection incident; Left apart and Same person: A duplicate survives; Left apart and Different people: What you wanted. The costs are not symmetric, so the answer is not one threshold. Merge automatically only above a high score or on an exact key, send the middle band to a person, and leave the rest alone.\n\nSo the policy is three bands, not one threshold:\n\n1. Above a high score, or an exact key match: merge automatically.\n2. In the middle band: create a review task with both records side by side. A person decides in seconds.\n3. Below: leave them alone and do not record a suspicion.\n\nAnd make every merge reversible. Never delete the losing record: write `merged_into: <id>` on it, keep it, and have your reads follow the pointer. An undo that is a single UPDATE is worth building on the first day.\n\n## Embeddings are the wrong tool for identity\n\nThis is worth stating plainly because it is a common and expensive mistake. An embedding places text near other text that means something similar. \"Acme Ltd, Bristol\" and \"Acme Ltd, Leeds\" are almost the same sentence, so their vectors sit almost on top of each other, and they are two different companies. Meanwhile \"Ranjit Singh\" and \"R. Singh\" may sit further apart than either does from \"Rajesh Singh\".\n\nThe mechanism is straightforward: identity is decided by a handful of discriminating tokens (a branch, a house number, a middle initial) and those tokens contribute very little to a vector that is summarising overall meaning. Semantic similarity is designed to be robust to exactly the small differences that identity depends on.\n\nUse embeddings to *generate candidates* for messy free-text names, then decide with exact fields: postcode, registration number, email domain. Never let a cosine score alone authorise a merge.\n\n## Measure it like anything else\n\nLabel two hundred pairs by hand: same or not. Report precision and recall separately, and treat precision on auto-merges as the number that must be near perfect. Re-run when the source data changes. A matcher tuned on last year's records will drift as a new source arrives with different formatting conventions, and nobody will notice until two accounts are joined that should never have met."},"check":{"question":"A team uses cosine similarity over embeddings of full company records to decide which entries describe the same company. It merges \"Acme Ltd, Unit 4, Bristol\" with \"Acme Ltd, Unit 9, Leeds\". Why does this approach fail at identity in particular?","options":[{"text":"The embedding model was not trained on company names, so a domain-specific model would resolve it.","correct":false,"tempting":"Assumes a training-data gap, when the same failure appears with domain-tuned models because the objective, not the corpus, is the issue."},{"text":"The two records are too short for a stable embedding; longer text would separate them.","correct":false,"tempting":"Length affects stability, but adding more shared boilerplate would push these two records closer together, not further apart."},{"text":"Identity turns on a few discriminating tokens (the branch, the unit number) that contribute little to a vector summarising overall meaning.","correct":true},{"text":"The similarity threshold was set too low and should be raised until such pairs fall below it.","correct":false,"tempting":"Raising the threshold suppresses this pair but also rejects genuine matches, because the two cases are not separable on this score at any cut point."}]},"prev":{"slug":"dates-names-and-numbers","title":"The boring bugs that eat automations","url":"https://addaly.com/learn/agents-and-automation/dates-names-and-numbers"},"next":{"slug":"retrieval-for-agents","title":"Grep first, embeddings later","url":"https://addaly.com/learn/agents-and-automation/retrieval-for-agents"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The same customer, three times', in Agents and Automation. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/agents-and-automation/deduplication-and-matching (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The same customer, three times","courseTitle":"Agents and Automation","module":{"index":3,"slug":"data-plumbing","title":"The data underneath: sources, documents, records"},"answer":null,"keyIdea":"Most duplicate problems are solved by normalising a key rather than by fuzzy matching, and a wrong merge is far more expensive than a surviving duplicate because merges are hard to undo.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["Same person","Different people"],"rows":["Merged","Left apart"],"type":"matrix","cells":[["What you wanted","A data-protection incident"],["A duplicate survives","What you wanted"]],"title":"The two ways a matcher is wrong, and what each costs","caption":"The costs are not symmetric, so the answer is not one threshold. Merge automatically only above a high score or on an exact key, send the middle band to a person, and leave the rest alone.","cellLabels":[["the merge you built the matcher for","one person sees another’s orders and address; hard to unpick"],["somebody sees a customer twice; fixed in ten seconds","two records that were never the same thing"]]},"alt":"Figure: The two ways a matcher is wrong, and what each costs. A two by two grid, rows Merged and Left apart, columns Same person and Different people: Merged and Same person: What you wanted; Merged and Different people: A data-protection incident; Left apart and Same person: A duplicate survives; Left apart and Different people: What you wanted. The costs are not symmetric, so the answer is not one threshold. Merge automatically only above a high score or on an exact key, send the middle band to a person, and leave the rest alone."}},"blocks":[{"t":"h2","id":"where-duplicates-come-from","runs":[{"text":"Where duplicates come from"}]},{"t":"p","runs":[{"text":"Three sources, and they need different answers. A retry that timed out after the write succeeded creates an exact duplicate, and the fix is idempotency at the write, covered later in this block. Two systems describing the same real thing (the CRM’s “Acme Ltd” and the accounts system’s “ACME LIMITED”) need matching. And a human typing the same customer in twice on a Tuesday needs a constraint at the point of entry, which is a product decision rather than a data one."}]},{"t":"p","runs":[{"text":"Know which you have before choosing a technique, because a fuzzy matcher pointed at a retry bug is an expensive way to not fix the bug."}]},{"t":"h2","id":"normalise-a-key-before-you-reach-for-similarity","runs":[{"text":"Normalise a key before you reach for similarity"}]},{"t":"p","runs":[{"text":"The great majority of real deduplication is solved without any matching algorithm at all. Pick a field that identifies the thing, normalise it hard, and group on it."}]},{"t":"ul","items":[[{"text":"Email: lowercase, trim. For Gmail specifically, dots in the local part are ignored and "},{"text":"+tag","code":true},{"text":" suffixes are aliases: worth handling if consumer emails matter to you, and worth "},{"text":"not","i":true},{"text":" generalising to other providers, where those rules do not hold."}],[{"text":"Phone: normalise to E.164 with the country code ("},{"text":"libphonenumber","code":true},{"text":" does this properly and free; hand-rolled regex does not survive contact with real data)."}],[{"text":"Company: strip legal suffixes, punctuation and case: "},{"text":"acme","code":true},{"text":" from "},{"text":"Acme Ltd.","code":true},{"text":", "},{"text":"ACME LIMITED","code":true},{"text":", "},{"text":"Acme Limited","code":true},{"text":"."}],[{"text":"Tax identifier, registration number, IBAN, national ID where you have one and are allowed to use it. These are the strongest keys available and they end the discussion."}]]},{"t":"p","runs":[{"text":"Do this first. It is cheap, exact, and explains itself in an audit, which fuzzy matching never quite does."}]},{"t":"h2","id":"when-there-is-no-key-block-then-compare","runs":[{"text":"When there is no key: block, then compare"}]},{"t":"p","runs":[{"text":"Comparing every record with every other is quadratic. Fifty thousand records is about 1.25 billion pairs, which is a job that runs overnight and gets skipped."}]},{"t":"p","runs":[{"text":"Blocking","b":true},{"text":" fixes it. Only compare records that share a cheap coarse key: same postcode, same first three letters of the surname, same first token of the company name, same year of birth. Fifty thousand records split into two thousand blocks of twenty-five is roughly six hundred thousand comparisons: seconds instead of hours."}]},{"t":"p","runs":[{"text":"Blocking trades recall for speed: a pair that disagrees on the blocking key is never compared, so choose a key that is unlikely to be wrong on both records. Run two or three different blocking passes and take the union, which recovers most of what any single pass misses."}]},{"t":"p","runs":[{"text":"Within a block, score with a string similarity. "},{"text":"Jaro-Winkler","b":true},{"text":" is well suited to names because it weights agreement at the start of the string. "},{"text":"Levenshtein","b":true},{"text":" distance suits short codes and typos. A token-set ratio handles reordering, so that “Kumar Rajesh” and “Rajesh Kumar” score high. "},{"text":"RapidFuzz","b":true},{"text":" in Python is fast and free; the "},{"text":"recordlinkage","code":true},{"text":" and "},{"text":"dedupe","code":true},{"text":" libraries build the whole pipeline if you want blocking and scoring together."}]},{"t":"h2","id":"choose-the-threshold-by-asymmetry-not-by-the-f1-score","runs":[{"text":"Choose the threshold by asymmetry, not by the F1 score"}]},{"t":"p","runs":[{"text":"Two records left unmerged means somebody sees a customer twice: irritating, visible, fixable in ten seconds. Two different people merged into one record means one person can see the other’s orders, invoices and address. That is a data-protection incident, it is discovered late, and unpicking a merge after downstream systems have copied the merged record is hard."}]},{"t":"p","runs":[{"text":"So the policy is three bands, not one threshold:"}]},{"t":"ol","start":1,"items":[[{"text":"Above a high score, or an exact key match: merge automatically."}],[{"text":"In the middle band: create a review task with both records side by side. A person decides in seconds."}],[{"text":"Below: leave them alone and do not record a suspicion."}]]},{"t":"p","runs":[{"text":"And make every merge reversible. Never delete the losing record: write "},{"text":"merged_into: <id>","code":true},{"text":" on it, keep it, and have your reads follow the pointer. An undo that is a single UPDATE is worth building on the first day."}]},{"t":"h2","id":"embeddings-are-the-wrong-tool-for-identity","runs":[{"text":"Embeddings are the wrong tool for identity"}]},{"t":"p","runs":[{"text":"This is worth stating plainly because it is a common and expensive mistake. An embedding places text near other text that means something similar. “Acme Ltd, Bristol” and “Acme Ltd, Leeds” are almost the same sentence, so their vectors sit almost on top of each other, and they are two different companies. Meanwhile “Ranjit Singh” and “R. Singh” may sit further apart than either does from “Rajesh Singh”."}]},{"t":"p","runs":[{"text":"The mechanism is straightforward: identity is decided by a handful of discriminating tokens (a branch, a house number, a middle initial) and those tokens contribute very little to a vector that is summarising overall meaning. Semantic similarity is designed to be robust to exactly the small differences that identity depends on."}]},{"t":"p","runs":[{"text":"Use embeddings to "},{"text":"generate candidates","i":true},{"text":" for messy free-text names, then decide with exact fields: postcode, registration number, email domain. Never let a cosine score alone authorise a merge."}]},{"t":"h2","id":"measure-it-like-anything-else","runs":[{"text":"Measure it like anything else"}]},{"t":"p","runs":[{"text":"Label two hundred pairs by hand: same or not. Report precision and recall separately, and treat precision on auto-merges as the number that must be near perfect. Re-run when the source data changes. A matcher tuned on last year’s records will drift as a new source arrives with different formatting conventions, and nobody will notice until two accounts are joined that should never have met."}]}],"check":{"question":[{"text":"A team uses cosine similarity over embeddings of full company records to decide which entries describe the same company. It merges “Acme Ltd, Unit 4, Bristol” with “Acme Ltd, Unit 9, Leeds”. Why does this approach fail at identity in particular?"}],"options":[{"text":[{"text":"The embedding model was not trained on company names, so a domain-specific model would resolve it."}],"correct":false,"tempting":[{"text":"Assumes a training-data gap, when the same failure appears with domain-tuned models because the objective, not the corpus, is the issue."}]},{"text":[{"text":"The two records are too short for a stable embedding; longer text would separate them."}],"correct":false,"tempting":[{"text":"Length affects stability, but adding more shared boilerplate would push these two records closer together, not further apart."}]},{"text":[{"text":"Identity turns on a few discriminating tokens (the branch, the unit number) that contribute little to a vector summarising overall meaning."}],"correct":true,"tempting":null},{"text":[{"text":"The similarity threshold was set too low and should be raised until such pairs fall below it."}],"correct":false,"tempting":[{"text":"Raising the threshold suppresses this pair but also rejects genuine matches, because the two cases are not separable on this score at any cut point."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"dates-names-and-numbers","title":"The boring bugs that eat automations","minutes":9},"next":{"slug":"retrieval-for-agents","title":"Grep first, embeddings later","minutes":9}}}