{"course":{"slug":"motion-and-audio","title":"Motion, Sound and Music","url":"https://addaly.com/learn/motion-and-audio","level":"intermediate","track":"create"},"lesson":{"slug":"how-a-voice-is-cloned","title":"The vector that is supposed to be you","url":"https://addaly.com/learn/motion-and-audio/how-a-voice-is-cloned","position":65,"of":72,"minutes":11,"answer":"A speaker embedding captures timbre (the resonances and average pitch that make a voice recognisable in a word) and not idiolect, the timing and stress habits of how a particular person speaks.","keyIdea":"A speaker embedding captures timbre (the resonances and average pitch that make a voice recognisable in a word) and not idiolect, the timing and stress habits of how a particular person speaks, which is why a clone convinces for three seconds and not for thirty."},"body":{"format":"markdown","text":"## A vector that is supposed to be you\n\nVoice cloning works by separating *what is being said* from *who is saying it*.\n\nA **speaker encoder** is trained on many thousands of voices with an objective that pushes recordings of the same person close together in a vector space and different people apart. Given a few seconds of speech, it produces an embedding (commonly a few hundred numbers) that is meant to capture speaker identity and nothing else.\n\nA synthesiser is then trained to produce speech conditioned on both the text and that embedding. At inference, you supply a short sample of a target voice, the encoder turns it into a vector, and the synthesiser speaks your text in something resembling that voice.\n\nThis is **zero-shot cloning**: no training on the target, seconds of reference audio, a result in seconds. **Fine-tuning** (actually training on ten minutes to an hour of the target) produces a closer match and takes compute and time.\n\n## What the embedding captures, and what it does not\n\nIt captures **timbre**: the resonances of a particular vocal tract, the average pitch, the general quality of the voice. That is most of what makes a voice recognisable in a single word.\n\nIt does not capture **idiolect**: the way a specific person actually speaks. Their characteristic pauses. Whether they run sentences together or break them. Which syllable they lean on. Their filler words. The rhythm of their thinking.\n\n> Figure: What a speaker embedding carries. Timbre, which it captures: The resonances of one vocal tract; Average pitch and general quality; Enough to convince in three seconds. Idiolect, which it does not: Characteristic pauses and phrasing; Which syllable this person leans on; Filler words and the rhythm of thinking; Which is why thirty seconds sounds wrong. A clone is the right timbre performing somebody else's habits: the generic behaviour the synthesiser learned from its training distribution. Play three seconds to a friend of the speaker and they say yes, that is them; play thirty and they say something is off and cannot name it. Fine-tuning on longer material narrows the gap and does not close it.\n\nThis is precisely why a good clone sounds like the person and not like them. Play three seconds to somebody who knows them and they will say yes, that is them. Play thirty seconds and they will say something is off, and struggle to name it. The timbre is right and the behaviour is somebody else's: the generic behaviour the synthesiser learned from its training distribution.\n\nFine-tuning on longer material narrows the gap because the model picks up some timing and stress habits along with the timbre. It does not close it.\n\n## Where clones degrade predictably\n\n- **Under-represented languages and accents.** The encoder was trained on a distribution; voices far from it get embeddings that are less distinctive, and the output drifts toward the training average. In practice this means a clone of a speaker with a strong regional accent in a well-represented language often comes out noticeably softened, and clones in low-resource languages are much weaker.\n- **Emotional range.** Reference audio of somebody speaking calmly gives you a calm clone. Asking it to shout produces a shout in something that no longer sounds like the person, because the mapping between calm and loud in a real voice is specific to that person's anatomy.\n- **Poor reference audio.** The encoder cannot separate the speaker from the room. A reference recorded in a bathroom produces a clone with a hint of bathroom in it, because the embedding absorbed some of it.\n- **Long output.** Consistency drifts across a long read, which is why production use chunks and re-conditions.\n\n## Detection, and why it is not a solution\n\nTwo families of defence are discussed, and both have real limits.\n\n**Anti-spoofing classifiers** are trained to distinguish synthetic from genuine speech. They perform well on the generation methods they were trained against and degrade on new ones, which is the ordinary arms-race problem: the classifier is always fitted to yesterday's generators.\n\n**Watermarking** embeds an imperceptible signal in generated audio. It works when the audio reaches the detector broadly intact. It is weakened or removed by the ordinary things audio undergoes (heavy lossy compression, re-recording through a speaker and microphone, pitch shifting, deliberate noise addition) and a determined adversary can strip it. Vendors publish robustness figures under specific transformations; those figures are not a claim about an adversary who is trying.\n\nThe honest summary: watermarking helps with accidental and casual misuse and with platform-side filtering. It is not evidence of authenticity, and the absence of a watermark proves nothing at all.\n\n## The technical does not settle the ethical\n\nThe next lesson covers consent in detail, and it is worth stating here why it cannot be an afterthought. Everything above describes how easy this is: a few seconds of audio, freely available for anybody who has ever spoken publicly, and a free model.\n\nThe capability is available to everyone. The restraint has to come from the practice, because it is not coming from the difficulty.\n\n## The free path\n\n- **Coqui XTTS**: multilingual zero-shot cloning; the models remain available with the maintenance caveat from the previous lesson, and note that the XTTS weights carry a licence that restricts commercial use, so read it rather than assuming.\n- **OpenVoice**: open-source tone-colour cloning with separate control of style.\n- **Piper**: can be fine-tuned on a target speaker for a self-hosted voice.\n- **RVC** and similar voice-conversion tools convert an existing performance into a target timbre, which preserves the original speaker's timing and often sounds better than text-to-speech cloning for exactly the idiolect reason above.\n\nLicences differ sharply across these projects and several restrict commercial use. Read the licence for each model, not just the repository.\n\n## The limitation\n\nThe most important one is not technical. A cloned voice is convincing enough for fraud, and the documented cases (relatives called by a synthesised voice asking for money, an employee instructed by a synthesised executive) turn on the fact that three seconds is enough to convince somebody who is not expecting it.\n\nThere is no technical countermeasure that an ordinary person can deploy. The practical defence that security guidance consistently recommends is procedural rather than technical: an agreed phrase within a family, a call back on a known number, a second channel for any instruction involving money. That is an unsatisfying answer and it is the one that works."},"check":{"question":"A colleague says a clone of their voice 'sounds like me but is not me', and cannot say what is wrong. What does the architecture predict they are hearing?","options":[{"text":"The vocoder is introducing artefacts in the high frequencies that are below conscious detection but sufficient to make the voice feel unfamiliar.","correct":false,"tempting":"Modern vocoders are clean, and the described effect appears over time rather than as a timbral flaw."},{"text":"The embedding was taken from too short a reference, so it has captured an average of the speaker rather than their voice; a longer reference would resolve it.","correct":false,"tempting":"Longer references improve the match but do not supply the behavioural information the embedding is not designed to carry."},{"text":"The reference recording carried the room, so the clone has a trace of that acoustic on it which reads as subtly wrong.","correct":false,"tempting":"A real contamination effect, but it would sound like a space rather than like the wrong person's behaviour."},{"text":"The timbre is right and the speech behaviour (pauses, stress placement, rhythm) is the synthesiser's generic default rather than theirs.","correct":true}]},"prev":{"slug":"tts-and-prosody","title":"What a text-to-speech system cannot read","url":"https://addaly.com/learn/motion-and-audio/tts-and-prosody"},"next":{"slug":"a-cloned-voice-needs-consent","title":"A cloned voice needs a yes you can point to","url":"https://addaly.com/learn/motion-and-audio/a-cloned-voice-needs-consent"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The vector that is supposed to be you', in Motion, Sound and Music. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/motion-and-audio/how-a-voice-is-cloned (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The vector that is supposed to be you","courseTitle":"Motion, Sound and Music","module":{"index":8,"slug":"machines-that-make-sound-and-motion","title":"Machines that make sound and motion"},"answer":null,"keyIdea":"A speaker embedding captures timbre (the resonances and average pitch that make a voice recognisable in a word) and not idiolect, the timing and stress habits of how a particular person speaks, which is why a clone convinces for three seconds and not for thirty.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"Timbre, which it captures","items":["The resonances of one vocal tract","Average pitch and general quality","Enough to convince in three seconds"]},"type":"compare","right":{"title":"Idiolect, which it does not","items":["Characteristic pauses and phrasing","Which syllable this person leans on","Filler words and the rhythm of thinking","Which is why thirty seconds sounds wrong"]},"title":"What a speaker embedding carries","caption":"A clone is the right timbre performing somebody else’s habits: the generic behaviour the synthesiser learned from its training distribution. Play three seconds to a friend of the speaker and they say yes, that is them; play thirty and they say something is off and cannot name it. Fine-tuning on longer material narrows the gap and does not close it."},"alt":"Figure: What a speaker embedding carries. Timbre, which it captures: The resonances of one vocal tract; Average pitch and general quality; Enough to convince in three seconds. Idiolect, which it does not: Characteristic pauses and phrasing; Which syllable this person leans on; Filler words and the rhythm of thinking; Which is why thirty seconds sounds wrong. A clone is the right timbre performing somebody else’s habits: the generic behaviour the synthesiser learned from its training distribution. Play three seconds to a friend of the speaker and they say yes, that is them; play thirty and they say something is off and cannot name it. Fine-tuning on longer material narrows the gap and does not close it."}},"blocks":[{"t":"h2","id":"a-vector-that-is-supposed-to-be-you","runs":[{"text":"A vector that is supposed to be you"}]},{"t":"p","runs":[{"text":"Voice cloning works by separating "},{"text":"what is being said","i":true},{"text":" from "},{"text":"who is saying it","i":true},{"text":"."}]},{"t":"p","runs":[{"text":"A "},{"text":"speaker encoder","b":true},{"text":" is trained on many thousands of voices with an objective that pushes recordings of the same person close together in a vector space and different people apart. Given a few seconds of speech, it produces an embedding (commonly a few hundred numbers) that is meant to capture speaker identity and nothing else."}]},{"t":"p","runs":[{"text":"A synthesiser is then trained to produce speech conditioned on both the text and that embedding. At inference, you supply a short sample of a target voice, the encoder turns it into a vector, and the synthesiser speaks your text in something resembling that voice."}]},{"t":"p","runs":[{"text":"This is "},{"text":"zero-shot cloning","b":true},{"text":": no training on the target, seconds of reference audio, a result in seconds. "},{"text":"Fine-tuning","b":true},{"text":" (actually training on ten minutes to an hour of the target) produces a closer match and takes compute and time."}]},{"t":"h2","id":"what-the-embedding-captures-and-what-it-does-not","runs":[{"text":"What the embedding captures, and what it does not"}]},{"t":"p","runs":[{"text":"It captures "},{"text":"timbre","b":true},{"text":": the resonances of a particular vocal tract, the average pitch, the general quality of the voice. That is most of what makes a voice recognisable in a single word."}]},{"t":"p","runs":[{"text":"It does not capture "},{"text":"idiolect","b":true},{"text":": the way a specific person actually speaks. Their characteristic pauses. Whether they run sentences together or break them. Which syllable they lean on. Their filler words. The rhythm of their thinking."}]},{"t":"p","runs":[{"text":"This is precisely why a good clone sounds like the person and not like them. Play three seconds to somebody who knows them and they will say yes, that is them. Play thirty seconds and they will say something is off, and struggle to name it. The timbre is right and the behaviour is somebody else’s: the generic behaviour the synthesiser learned from its training distribution."}]},{"t":"p","runs":[{"text":"Fine-tuning on longer material narrows the gap because the model picks up some timing and stress habits along with the timbre. It does not close it."}]},{"t":"h2","id":"where-clones-degrade-predictably","runs":[{"text":"Where clones degrade predictably"}]},{"t":"ul","items":[[{"text":"Under-represented languages and accents.","b":true},{"text":" The encoder was trained on a distribution; voices far from it get embeddings that are less distinctive, and the output drifts toward the training average. In practice this means a clone of a speaker with a strong regional accent in a well-represented language often comes out noticeably softened, and clones in low-resource languages are much weaker."}],[{"text":"Emotional range.","b":true},{"text":" Reference audio of somebody speaking calmly gives you a calm clone. Asking it to shout produces a shout in something that no longer sounds like the person, because the mapping between calm and loud in a real voice is specific to that person’s anatomy."}],[{"text":"Poor reference audio.","b":true},{"text":" The encoder cannot separate the speaker from the room. A reference recorded in a bathroom produces a clone with a hint of bathroom in it, because the embedding absorbed some of it."}],[{"text":"Long output.","b":true},{"text":" Consistency drifts across a long read, which is why production use chunks and re-conditions."}]]},{"t":"h2","id":"detection-and-why-it-is-not-a-solution","runs":[{"text":"Detection, and why it is not a solution"}]},{"t":"p","runs":[{"text":"Two families of defence are discussed, and both have real limits."}]},{"t":"p","runs":[{"text":"Anti-spoofing classifiers","b":true},{"text":" are trained to distinguish synthetic from genuine speech. They perform well on the generation methods they were trained against and degrade on new ones, which is the ordinary arms-race problem: the classifier is always fitted to yesterday’s generators."}]},{"t":"p","runs":[{"text":"Watermarking","b":true},{"text":" embeds an imperceptible signal in generated audio. It works when the audio reaches the detector broadly intact. It is weakened or removed by the ordinary things audio undergoes (heavy lossy compression, re-recording through a speaker and microphone, pitch shifting, deliberate noise addition) and a determined adversary can strip it. Vendors publish robustness figures under specific transformations; those figures are not a claim about an adversary who is trying."}]},{"t":"p","runs":[{"text":"The honest summary: watermarking helps with accidental and casual misuse and with platform-side filtering. It is not evidence of authenticity, and the absence of a watermark proves nothing at all."}]},{"t":"h2","id":"the-technical-does-not-settle-the-ethical","runs":[{"text":"The technical does not settle the ethical"}]},{"t":"p","runs":[{"text":"The next lesson covers consent in detail, and it is worth stating here why it cannot be an afterthought. Everything above describes how easy this is: a few seconds of audio, freely available for anybody who has ever spoken publicly, and a free model."}]},{"t":"p","runs":[{"text":"The capability is available to everyone. The restraint has to come from the practice, because it is not coming from the difficulty."}]},{"t":"h2","id":"the-free-path","runs":[{"text":"The free path"}],"role":"free"},{"t":"ul","items":[[{"text":"Coqui XTTS","b":true},{"text":": multilingual zero-shot cloning; the models remain available with the maintenance caveat from the previous lesson, and note that the XTTS weights carry a licence that restricts commercial use, so read it rather than assuming."}],[{"text":"OpenVoice","b":true},{"text":": open-source tone-colour cloning with separate control of style."}],[{"text":"Piper","b":true},{"text":": can be fine-tuned on a target speaker for a self-hosted voice."}],[{"text":"RVC","b":true},{"text":" and similar voice-conversion tools convert an existing performance into a target timbre, which preserves the original speaker’s timing and often sounds better than text-to-speech cloning for exactly the idiolect reason above."}]]},{"t":"p","runs":[{"text":"Licences differ sharply across these projects and several restrict commercial use. Read the licence for each model, not just the repository."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"The most important one is not technical. A cloned voice is convincing enough for fraud, and the documented cases (relatives called by a synthesised voice asking for money, an employee instructed by a synthesised executive) turn on the fact that three seconds is enough to convince somebody who is not expecting it."}]},{"t":"p","runs":[{"text":"There is no technical countermeasure that an ordinary person can deploy. The practical defence that security guidance consistently recommends is procedural rather than technical: an agreed phrase within a family, a call back on a known number, a second channel for any instruction involving money. That is an unsatisfying answer and it is the one that works."}]}],"check":{"question":[{"text":"A colleague says a clone of their voice ‘sounds like me but is not me’, and cannot say what is wrong. What does the architecture predict they are hearing?"}],"options":[{"text":[{"text":"The vocoder is introducing artefacts in the high frequencies that are below conscious detection but sufficient to make the voice feel unfamiliar."}],"correct":false,"tempting":[{"text":"Modern vocoders are clean, and the described effect appears over time rather than as a timbral flaw."}]},{"text":[{"text":"The embedding was taken from too short a reference, so it has captured an average of the speaker rather than their voice; a longer reference would resolve it."}],"correct":false,"tempting":[{"text":"Longer references improve the match but do not supply the behavioural information the embedding is not designed to carry."}]},{"text":[{"text":"The reference recording carried the room, so the clone has a trace of that acoustic on it which reads as subtly wrong."}],"correct":false,"tempting":[{"text":"A real contamination effect, but it would sound like a space rather than like the wrong person’s behaviour."}]},{"text":[{"text":"The timbre is right and the speech behaviour (pauses, stress placement, rhythm) is the synthesiser’s generic default rather than theirs."}],"correct":true,"tempting":null}],"order":[0,1,2,3],"why":null},"prev":{"slug":"tts-and-prosody","title":"What a text-to-speech system cannot read","minutes":10},"next":{"slug":"a-cloned-voice-needs-consent","title":"A cloned voice needs a yes you can point to","minutes":11}}}