Speech, in both directions

Recognition: Whisper and the things built on it

OpenAI’s Whisper is MIT-licensed, which makes it one of the most permissively released useful models in existence. The large models are around 1.5 billion parameters, and the family runs down to tiny at 39 million. It was trained on around 680,000 hours of multilingual audio and it transcribes, translates into English, and detects language.

You will rarely run the original implementation, because faster reimplementations exist and are equivalent in output:

• faster-whisper uses CTranslate2 and is roughly four times quicker with lower memory. This is the default choice on a server.
• whisper.cpp is the C++ port with the same relationship to Whisper that llama.cpp has to language models: it runs on a laptop CPU, a phone, or a Raspberry Pi.
• distil-whisper is a distilled English model, about six times faster than large with a small accuracy cost.

pip install faster-whisper
python - <<'EOF'
from faster_whisper import WhisperModel
m = WhisperModel("large-v3", device="cpu", compute_type="int8")
segments, info = m.transcribe("call.wav", language="hi", vad_filter=True)
for s in segments:
    print(f"[{s.start:.1f}-{s.end:.1f}] {s.text}")
EOF

The failure everybody meets

Whisper hallucinates on silence. Given a long pause, background noise, or a segment with no speech, it will confidently produce a sentence: very often a piece of text that was common in its training subtitles, such as a thank-you line or a channel sign-off. Studies of medical and legal transcription have found this in a meaningful fraction of segments, and it is worse on recordings with long silences, which is exactly what a phone call or an interview contains.

The fix is voice activity detection: run a VAD to find speech regions and transcribe only those. vad_filter=True above does this. It is not optional for production work, and it is the difference between a transcript you can trust and one that invents sentences.

Two smaller cautions. Word-level timestamps are approximate, derived from attention alignment rather than measured; treat them as within a couple of hundred milliseconds, not exact. And accuracy varies enormously by language: word error rates in the well-resourced European languages are in the single digits, while many Indian and African languages sit far higher. For Indian languages specifically, AI4Bharat’s IndicWhisper and related work are usually better than the general model, for the same data-and-tokenizer reason as in the language-model case.

Diarisation (who spoke when) is a separate model. pyannote.audio is the common open choice; it is free but its models are gated and carry their own terms, so read them before shipping.

Synthesis: text to speech

The open TTS landscape is more fragmented and the licences are worse. Four worth knowing:

• Piper. MIT, tiny, runs in real time on a Raspberry Pi, dozens of languages. It sounds like a good 2015 voice, and for accessibility, announcements, IVR and anything on a low-power device it is the correct answer.
• Kokoro. Around 82 million parameters, Apache 2.0, and much better sounding than its size suggests. The current default for good quality at low cost.
• XTTS (Coqui). Voice cloning from a few seconds of audio, multilingual, and released under a licence that is not commercially free. Widely used anyway, which is a compliance problem waiting for someone.
• F5-TTS, Orpheus and others in the newer generation, generally permissive, higher quality, heavier.

The quality ladder here costs compute in a way that recognition does not: a good neural voice is several times more expensive per second of audio than transcribing the same second.

The consent question, stated plainly

Voice cloning from a short sample is now easy, cheap and open. Cloning a specific person’s voice without their explicit consent is the harmful use of this technology, it is the basis of a large and growing fraud category, and several jurisdictions have started legislating on it. If you build with cloning, get recorded consent from the person whose voice it is, keep it, and label synthetic audio. This is not a legal opinion; it is the minimum that makes the work defensible.

A working pipeline

Audio in, transcript out, on a laptop, free: VAD to find speech, faster-whisper at int8 for transcription, pyannote for speaker labels if you need them, then a small language model for summarising. Every piece of that runs offline, which is often the actual requirement: recordings of conversations are among the most sensitive data most organisations hold, and the reason to self-host is usually not cost.