{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"open-speech-models","title":"Speech, in both directions","url":"https://addaly.com/learn/open-models/open-speech-models","position":17,"of":71,"minutes":9,"answer":"Whisper invents text during silence unless voice activity detection gates it, so a production transcription pipeline is VAD plus a fast reimplementation, and text-to-speech is where the open licences get genuinely restrictive.","keyIdea":"Whisper invents text during silence unless voice activity detection gates it, so a production transcription pipeline is VAD plus a fast reimplementation, and text-to-speech is where the open licences get genuinely restrictive."},"body":{"format":"markdown","text":"## Recognition: Whisper and the things built on it\n\nOpenAI's Whisper is MIT-licensed, which makes it one of the most permissively released useful models in existence. The large models are around 1.5 billion parameters, and the family runs down to `tiny` at 39 million. It was trained on around 680,000 hours of multilingual audio and it transcribes, translates into English, and detects language.\n\nYou will rarely run the original implementation, because faster reimplementations exist and are equivalent in output:\n\n- **faster-whisper** uses CTranslate2 and is roughly four times quicker with lower memory. This is the default choice on a server.\n- **whisper.cpp** is the C++ port with the same relationship to Whisper that llama.cpp has to language models: it runs on a laptop CPU, a phone, or a Raspberry Pi.\n- **distil-whisper** is a distilled English model, about six times faster than large with a small accuracy cost.\n\n```bash\npip install faster-whisper\npython - <<'EOF'\nfrom faster_whisper import WhisperModel\nm = WhisperModel(\"large-v3\", device=\"cpu\", compute_type=\"int8\")\nsegments, info = m.transcribe(\"call.wav\", language=\"hi\", vad_filter=True)\nfor s in segments:\n    print(f\"[{s.start:.1f}-{s.end:.1f}] {s.text}\")\nEOF\n```\n\n## The failure everybody meets\n\nWhisper hallucinates on silence. Given a long pause, background noise, or a segment with no speech, it will confidently produce a sentence: very often a piece of text that was common in its training subtitles, such as a thank-you line or a channel sign-off. Studies of medical and legal transcription have found this in a meaningful fraction of segments, and it is worse on recordings with long silences, which is exactly what a phone call or an interview contains.\n\nThe fix is voice activity detection: run a VAD to find speech regions and transcribe only those. `vad_filter=True` above does this. It is not optional for production work, and it is the difference between a transcript you can trust and one that invents sentences.\n\nTwo smaller cautions. Word-level timestamps are approximate, derived from attention alignment rather than measured; treat them as within a couple of hundred milliseconds, not exact. And accuracy varies enormously by language: word error rates in the well-resourced European languages are in the single digits, while many Indian and African languages sit far higher. For Indian languages specifically, AI4Bharat's IndicWhisper and related work are usually better than the general model, for the same data-and-tokenizer reason as in the language-model case.\n\n**Diarisation** (who spoke when) is a separate model. `pyannote.audio` is the common open choice; it is free but its models are gated and carry their own terms, so read them before shipping.\n\n## Synthesis: text to speech\n\nThe open TTS landscape is more fragmented and the licences are worse. Four worth knowing:\n\n- **Piper.** MIT, tiny, runs in real time on a Raspberry Pi, dozens of languages. It sounds like a good 2015 voice, and for accessibility, announcements, IVR and anything on a low-power device it is the correct answer.\n- **Kokoro.** Around 82 million parameters, Apache 2.0, and much better sounding than its size suggests. The current default for good quality at low cost.\n- **XTTS (Coqui).** Voice cloning from a few seconds of audio, multilingual, and released under a licence that is not commercially free. Widely used anyway, which is a compliance problem waiting for someone.\n- **F5-TTS**, **Orpheus** and others in the newer generation, generally permissive, higher quality, heavier.\n\nThe quality ladder here costs compute in a way that recognition does not: a good neural voice is several times more expensive per second of audio than transcribing the same second.\n\n## The consent question, stated plainly\n\nVoice cloning from a short sample is now easy, cheap and open. Cloning a specific person's voice without their explicit consent is the harmful use of this technology, it is the basis of a large and growing fraud category, and several jurisdictions have started legislating on it. If you build with cloning, get recorded consent from the person whose voice it is, keep it, and label synthetic audio. This is not a legal opinion; it is the minimum that makes the work defensible.\n\n## A working pipeline\n\nAudio in, transcript out, on a laptop, free: VAD to find speech, faster-whisper at `int8` for transcription, pyannote for speaker labels if you need them, then a small language model for summarising. Every piece of that runs offline, which is often the actual requirement: recordings of conversations are among the most sensitive data most organisations hold, and the reason to self-host is usually not cost."},"check":{"question":"A transcript of a one-hour interview contains several plausible sentences at points where the recording is silent. What is happening and what fixes it?","options":[{"text":"The audio has clipping artefacts the model is misreading as speech; re-encoding at a higher bitrate removes them.","correct":false,"tempting":"Encoding quality affects accuracy on real speech but does not cause fabricated text in genuine silence."},{"text":"The temperature fallback is producing random output; setting temperature to zero eliminates it.","correct":false,"tempting":"Lowering temperature reduces variety but the model still emits its most likely completion for a segment it should not have transcribed at all."},{"text":"The model is transcribing beyond the audio length; trimming the file to its true duration fixes it.","correct":false,"tempting":"Mislocates the cause: the silences are inside the recording, not appended to it."},{"text":"The model always produces text for whatever segment it is given, so silent segments get its most likely training phrase; gating with voice activity detection stops those segments reaching it.","correct":true}]},"prev":{"slug":"rerankers-and-the-retrieval-stack","title":"Rerankers, and why retrieval is two stages","url":"https://addaly.com/learn/open-models/rerankers-and-the-retrieval-stack"},"next":{"slug":"open-image-and-video-models","title":"Image and video weights, and their licence traps","url":"https://addaly.com/learn/open-models/open-image-and-video-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Speech, in both directions', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/open-speech-models (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Speech, in both directions","courseTitle":"The Open Model Ecosystem","module":{"index":2,"slug":"the-map-of-the-ecosystem","title":"The map, and how to read it"},"answer":null,"keyIdea":"Whisper invents text during silence unless voice activity detection gates it, so a production transcription pipeline is VAD plus a fast reimplementation, and text-to-speech is where the open licences get genuinely restrictive.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"recognition-whisper-and-the-things-built-on-it","runs":[{"text":"Recognition: Whisper and the things built on it"}]},{"t":"p","runs":[{"text":"OpenAI’s Whisper is MIT-licensed, which makes it one of the most permissively released useful models in existence. The large models are around 1.5 billion parameters, and the family runs down to "},{"text":"tiny","code":true},{"text":" at 39 million. It was trained on around 680,000 hours of multilingual audio and it transcribes, translates into English, and detects language."}]},{"t":"p","runs":[{"text":"You will rarely run the original implementation, because faster reimplementations exist and are equivalent in output:"}]},{"t":"ul","items":[[{"text":"faster-whisper","b":true},{"text":" uses CTranslate2 and is roughly four times quicker with lower memory. This is the default choice on a server."}],[{"text":"whisper.cpp","b":true},{"text":" is the C++ port with the same relationship to Whisper that llama.cpp has to language models: it runs on a laptop CPU, a phone, or a Raspberry Pi."}],[{"text":"distil-whisper","b":true},{"text":" is a distilled English model, about six times faster than large with a small accuracy cost."}]]},{"t":"code","lang":"bash","label":"Bash","text":"pip install faster-whisper\npython - <<'EOF'\nfrom faster_whisper import WhisperModel\nm = WhisperModel(\"large-v3\", device=\"cpu\", compute_type=\"int8\")\nsegments, info = m.transcribe(\"call.wav\", language=\"hi\", vad_filter=True)\nfor s in segments:\n    print(f\"[{s.start:.1f}-{s.end:.1f}] {s.text}\")\nEOF"},{"t":"h2","id":"the-failure-everybody-meets","runs":[{"text":"The failure everybody meets"}]},{"t":"p","runs":[{"text":"Whisper hallucinates on silence. Given a long pause, background noise, or a segment with no speech, it will confidently produce a sentence: very often a piece of text that was common in its training subtitles, such as a thank-you line or a channel sign-off. Studies of medical and legal transcription have found this in a meaningful fraction of segments, and it is worse on recordings with long silences, which is exactly what a phone call or an interview contains."}]},{"t":"p","runs":[{"text":"The fix is voice activity detection: run a VAD to find speech regions and transcribe only those. "},{"text":"vad_filter=True","code":true},{"text":" above does this. It is not optional for production work, and it is the difference between a transcript you can trust and one that invents sentences."}]},{"t":"p","runs":[{"text":"Two smaller cautions. Word-level timestamps are approximate, derived from attention alignment rather than measured; treat them as within a couple of hundred milliseconds, not exact. And accuracy varies enormously by language: word error rates in the well-resourced European languages are in the single digits, while many Indian and African languages sit far higher. For Indian languages specifically, AI4Bharat’s IndicWhisper and related work are usually better than the general model, for the same data-and-tokenizer reason as in the language-model case."}]},{"t":"p","runs":[{"text":"Diarisation","b":true},{"text":" (who spoke when) is a separate model. "},{"text":"pyannote.audio","code":true},{"text":" is the common open choice; it is free but its models are gated and carry their own terms, so read them before shipping."}]},{"t":"h2","id":"synthesis-text-to-speech","runs":[{"text":"Synthesis: text to speech"}]},{"t":"p","runs":[{"text":"The open TTS landscape is more fragmented and the licences are worse. Four worth knowing:"}]},{"t":"ul","items":[[{"text":"Piper.","b":true},{"text":" MIT, tiny, runs in real time on a Raspberry Pi, dozens of languages. It sounds like a good 2015 voice, and for accessibility, announcements, IVR and anything on a low-power device it is the correct answer."}],[{"text":"Kokoro.","b":true},{"text":" Around 82 million parameters, Apache 2.0, and much better sounding than its size suggests. The current default for good quality at low cost."}],[{"text":"XTTS (Coqui).","b":true},{"text":" Voice cloning from a few seconds of audio, multilingual, and released under a licence that is not commercially free. Widely used anyway, which is a compliance problem waiting for someone."}],[{"text":"F5-TTS","b":true},{"text":", "},{"text":"Orpheus","b":true},{"text":" and others in the newer generation, generally permissive, higher quality, heavier."}]]},{"t":"p","runs":[{"text":"The quality ladder here costs compute in a way that recognition does not: a good neural voice is several times more expensive per second of audio than transcribing the same second."}]},{"t":"h2","id":"the-consent-question-stated-plainly","runs":[{"text":"The consent question, stated plainly"}]},{"t":"p","runs":[{"text":"Voice cloning from a short sample is now easy, cheap and open. Cloning a specific person’s voice without their explicit consent is the harmful use of this technology, it is the basis of a large and growing fraud category, and several jurisdictions have started legislating on it. If you build with cloning, get recorded consent from the person whose voice it is, keep it, and label synthetic audio. This is not a legal opinion; it is the minimum that makes the work defensible."}]},{"t":"h2","id":"a-working-pipeline","runs":[{"text":"A working pipeline"}]},{"t":"p","runs":[{"text":"Audio in, transcript out, on a laptop, free: VAD to find speech, faster-whisper at "},{"text":"int8","code":true},{"text":" for transcription, pyannote for speaker labels if you need them, then a small language model for summarising. Every piece of that runs offline, which is often the actual requirement: recordings of conversations are among the most sensitive data most organisations hold, and the reason to self-host is usually not cost."}]}],"check":{"question":[{"text":"A transcript of a one-hour interview contains several plausible sentences at points where the recording is silent. What is happening and what fixes it?"}],"options":[{"text":[{"text":"The audio has clipping artefacts the model is misreading as speech; re-encoding at a higher bitrate removes them."}],"correct":false,"tempting":[{"text":"Encoding quality affects accuracy on real speech but does not cause fabricated text in genuine silence."}]},{"text":[{"text":"The temperature fallback is producing random output; setting temperature to zero eliminates it."}],"correct":false,"tempting":[{"text":"Lowering temperature reduces variety but the model still emits its most likely completion for a segment it should not have transcribed at all."}]},{"text":[{"text":"The model is transcribing beyond the audio length; trimming the file to its true duration fixes it."}],"correct":false,"tempting":[{"text":"Mislocates the cause: the silences are inside the recording, not appended to it."}]},{"text":[{"text":"The model always produces text for whatever segment it is given, so silent segments get its most likely training phrase; gating with voice activity detection stops those segments reaching it."}],"correct":true,"tempting":null}],"order":[0,2,1,3],"why":null},"prev":{"slug":"rerankers-and-the-retrieval-stack","title":"Rerankers, and why retrieval is two stages","minutes":9},"next":{"slug":"open-image-and-video-models","title":"Image and video weights, and their licence traps","minutes":9}}}