{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"cloning-a-voice","title":"A voice from three seconds","url":"https://addaly.com/learn/ai-actually-explained/cloning-a-voice","position":57,"of":92,"minutes":9,"answer":"A voice model separates who is speaking from what is said into a handful of numbers, so a few seconds of audio is enough to place a new speaker in a space learned from thousands.","keyIdea":"A voice model separates who is speaking from what is said into a handful of numbers, so a few seconds of audio is enough to place a new speaker in a space learned from thousands, and that is why the scam calls work."},"body":{"format":"markdown","text":"## Two different products\n\nText-to-speech has existed for decades and until recently sounded like it. What changed around 2023 was not that machines could read aloud but that they could read aloud *as you*, from a sample shorter than a sentence. The two are sold separately and confused constantly, so start with the split.\n\n**Text-to-speech** turns writing into a voice: one of a set of stock voices, or a voice the company built. **Voice cloning** turns writing into a *particular* person's voice, from a recording of them. The first is a reading tool. The second is an identity tool, and the interesting mechanism is in it.\n\n## Separating who from what\n\nSpeech carries two things at once: the words, and the speaker. The same sentence in your voice and mine differs in pitch, timbre, pace, accent, the way the consonants land. A voice model is trained on thousands of speakers saying millions of sentences, and it learns to represent the speaker as a short list of numbers (a speaker embedding, the same idea as the embeddings for words in the chatbot module) separate from the words being said.\n\nOnce that separation is learned, cloning is cheap. Play the model three seconds of a new voice, and it computes where in the speaker-space that voice sits. Then it generates any sentence you type from that position. It does not need to hear you say the words; it needs to know where you are in a space it learned from everyone else. A Microsoft research model in January 2023 demonstrated this from a three-second sample, and the commercial products followed within the year.\n\nThat is why three seconds is enough. The model is not learning your voice. It is locating it.\n\n## What the recordings are used for\n\nThe honest uses are real: a person losing their voice to illness recording it while they still can; audiobooks in a narrator's voice at a fraction of studio cost; dubbing a film into another language in the original actor's voice, lips adjusted to match; a language learner hearing their own voice speak correctly.\n\nThe dishonest ones are the reason this lesson exists. In January 2024 voters in an American state received a call in the president's voice telling them not to vote. It was generated with a commercial cloning tool by a political consultant, who was later fined six million dollars. In February 2024 an engineering firm's Hong Kong office paid out about twenty-five million dollars after a finance employee joined a video call in which every other participant, including the chief financial officer, was a synthetic likeness. And the commonest case by far is the family emergency: a call in a relative's voice, in distress, asking for money now. Consumer regulators in several countries issued warnings about it in 2023, and the sample needed can be lifted from a social-media video.\n\n## Why you cannot tell by listening\n\nFor the same reason you cannot detect generated text: the output is optimised to be the likeliest continuation of a real voice, and the artefacts that once gave it away (a metallic edge, flat intonation) were the first things the training removed. Detectors exist and, like image detectors, work best on the generator they were built against and degrade on the next one. Over a phone line, compressed, with background noise, nobody reliably tells.\n\n## What works instead\n\nThe defences are old and cheap, and they are the same shape as the ones for fake images. Agree a word with your family that no recording would contain, and ask for it. Hang up and call the person back on the number you already have. Treat urgency as the signature of the attack rather than a reason to act. Institutions have started adding a second channel for any payment instruction that arrives by voice or video, because the voice is no longer evidence of the person.\n\n## The free path, and the paid one\n\nThe cloning tool named in most job advertisements is a commercial service that also sells the dubbing and audiobook products. Open models do the same thing: several released in 2024 clone from a six-second sample and run on a laptop, and a lightweight open text-to-speech engine runs on a single-board computer for reading tools where no cloning is wanted. Most open cloning models carry licences that forbid commercial use, and every one of them will clone any voice you give it, including one you have no right to. That last fact is the limitation nobody markets, and it is the whole of the problem."},"check":{"question":"Why does a modern voice-cloning model need only a few seconds of someone's speech, when learning to imitate a voice used to take hours of recordings?","options":[{"text":"Because the model records every sound the speaker makes in the sample and replays those sounds in new orders to form new words.","correct":false,"tempting":"Imagines splicing; the model generates sentences the speaker never said, in sounds it never heard from them."},{"text":"Because it already learned a space of speakers from thousands of voices, and the sample only locates a new one in it.","correct":true},{"text":"Because a few seconds contains every phoneme in the language, which is all the model needs.","correct":false,"tempting":"Assumes the sample must be complete; three seconds contains a handful of phonemes and the model fills in the rest."},{"text":"Because modern microphones capture enough detail that a short clip is as good as an hour.","correct":false,"tempting":"Puts the change in the hardware; the clips come from social-media videos and phone calls."}]},"prev":{"slug":"the-model-that-looks-at-pictures","title":"When the model looks at a picture","url":"https://addaly.com/learn/ai-actually-explained/the-model-that-looks-at-pictures"},"next":{"slug":"music-from-a-prompt","title":"Music from a prompt","url":"https://addaly.com/learn/ai-actually-explained/music-from-a-prompt"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A voice from three seconds', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/cloning-a-voice (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A voice from three seconds","courseTitle":"AI, Actually Explained","module":{"index":6,"slug":"beyond-text","title":"Beyond text: pictures, voices and video"},"answer":null,"keyIdea":"A voice model separates who is speaking from what is said into a handful of numbers, so a few seconds of audio is enough to place a new speaker in a space learned from thousands, and that is why the scam calls work.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-different-products","runs":[{"text":"Two different products"}]},{"t":"p","runs":[{"text":"Text-to-speech has existed for decades and until recently sounded like it. What changed around 2023 was not that machines could read aloud but that they could read aloud "},{"text":"as you","i":true},{"text":", from a sample shorter than a sentence. The two are sold separately and confused constantly, so start with the split."}]},{"t":"p","runs":[{"text":"Text-to-speech","b":true},{"text":" turns writing into a voice: one of a set of stock voices, or a voice the company built. "},{"text":"Voice cloning","b":true},{"text":" turns writing into a "},{"text":"particular","i":true},{"text":" person’s voice, from a recording of them. The first is a reading tool. The second is an identity tool, and the interesting mechanism is in it."}]},{"t":"h2","id":"separating-who-from-what","runs":[{"text":"Separating who from what"}]},{"t":"p","runs":[{"text":"Speech carries two things at once: the words, and the speaker. The same sentence in your voice and mine differs in pitch, timbre, pace, accent, the way the consonants land. A voice model is trained on thousands of speakers saying millions of sentences, and it learns to represent the speaker as a short list of numbers (a speaker embedding, the same idea as the embeddings for words in the chatbot module) separate from the words being said."}]},{"t":"p","runs":[{"text":"Once that separation is learned, cloning is cheap. Play the model three seconds of a new voice, and it computes where in the speaker-space that voice sits. Then it generates any sentence you type from that position. It does not need to hear you say the words; it needs to know where you are in a space it learned from everyone else. A Microsoft research model in January 2023 demonstrated this from a three-second sample, and the commercial products followed within the year."}]},{"t":"p","runs":[{"text":"That is why three seconds is enough. The model is not learning your voice. It is locating it."}]},{"t":"h2","id":"what-the-recordings-are-used-for","runs":[{"text":"What the recordings are used for"}]},{"t":"p","runs":[{"text":"The honest uses are real: a person losing their voice to illness recording it while they still can; audiobooks in a narrator’s voice at a fraction of studio cost; dubbing a film into another language in the original actor’s voice, lips adjusted to match; a language learner hearing their own voice speak correctly."}]},{"t":"p","runs":[{"text":"The dishonest ones are the reason this lesson exists. In January 2024 voters in an American state received a call in the president’s voice telling them not to vote. It was generated with a commercial cloning tool by a political consultant, who was later fined six million dollars. In February 2024 an engineering firm’s Hong Kong office paid out about twenty-five million dollars after a finance employee joined a video call in which every other participant, including the chief financial officer, was a synthetic likeness. And the commonest case by far is the family emergency: a call in a relative’s voice, in distress, asking for money now. Consumer regulators in several countries issued warnings about it in 2023, and the sample needed can be lifted from a social-media video."}]},{"t":"h2","id":"why-you-cannot-tell-by-listening","runs":[{"text":"Why you cannot tell by listening"}]},{"t":"p","runs":[{"text":"For the same reason you cannot detect generated text: the output is optimised to be the likeliest continuation of a real voice, and the artefacts that once gave it away (a metallic edge, flat intonation) were the first things the training removed. Detectors exist and, like image detectors, work best on the generator they were built against and degrade on the next one. Over a phone line, compressed, with background noise, nobody reliably tells."}]},{"t":"h2","id":"what-works-instead","runs":[{"text":"What works instead"}]},{"t":"p","runs":[{"text":"The defences are old and cheap, and they are the same shape as the ones for fake images. Agree a word with your family that no recording would contain, and ask for it. Hang up and call the person back on the number you already have. Treat urgency as the signature of the attack rather than a reason to act. Institutions have started adding a second channel for any payment instruction that arrives by voice or video, because the voice is no longer evidence of the person."}]},{"t":"h2","id":"the-free-path-and-the-paid-one","runs":[{"text":"The free path, and the paid one"}],"role":"free"},{"t":"p","runs":[{"text":"The cloning tool named in most job advertisements is a commercial service that also sells the dubbing and audiobook products. Open models do the same thing: several released in 2024 clone from a six-second sample and run on a laptop, and a lightweight open text-to-speech engine runs on a single-board computer for reading tools where no cloning is wanted. Most open cloning models carry licences that forbid commercial use, and every one of them will clone any voice you give it, including one you have no right to. That last fact is the limitation nobody markets, and it is the whole of the problem."}]}],"check":{"question":[{"text":"Why does a modern voice-cloning model need only a few seconds of someone’s speech, when learning to imitate a voice used to take hours of recordings?"}],"options":[{"text":[{"text":"Because the model records every sound the speaker makes in the sample and replays those sounds in new orders to form new words."}],"correct":false,"tempting":[{"text":"Imagines splicing; the model generates sentences the speaker never said, in sounds it never heard from them."}]},{"text":[{"text":"Because it already learned a space of speakers from thousands of voices, and the sample only locates a new one in it."}],"correct":true,"tempting":null},{"text":[{"text":"Because a few seconds contains every phoneme in the language, which is all the model needs."}],"correct":false,"tempting":[{"text":"Assumes the sample must be complete; three seconds contains a handful of phonemes and the model fills in the rest."}]},{"text":[{"text":"Because modern microphones capture enough detail that a short clip is as good as an hour."}],"correct":false,"tempting":[{"text":"Puts the change in the hardware; the clips come from social-media videos and phone calls."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"the-model-that-looks-at-pictures","title":"When the model looks at a picture","minutes":9},"next":{"slug":"music-from-a-prompt","title":"Music from a prompt","minutes":8}}}