Voice in and out: transcription, speech and the latency budget

Two pipelines and one budget

The classic voice assistant is a cascade: speech comes in, a speech-to-text model transcribes it, the language model answers in text, and a text-to-speech model speaks the answer. Each stage adds delay, and the delays add. People expect a reply to begin within about a second of finishing a sentence; past two seconds the silence reads as a fault. That budget, roughly a thousand milliseconds from the user’s last word to the first spoken syllable of the reply, is the constraint every decision in this lesson serves.

Speech to text

Whisper, released openly by OpenAI, is the free baseline. Through faster-whisper the small models transcribe faster than real time on a CPU; the large model wants a GPU. Hosted transcription from Deepgram, AssemblyAI, Google, Microsoft and OpenAI costs fractions of a cent per minute of audio, which for most products is negligible next to the language model.

For Indian languages, test before choosing. Whisper handles Hindi reasonably and is weaker on Tamil, Bengali and regional accents; AI4Bharat’s Indic models are free and trained specifically on these languages, and are often the better choice. Hinglish (a sentence that switches language halfway through) is the hard case for every model, and it is what a large share of your users will speak. Put twenty real recordings through each candidate and count the errors on names and numbers, which are the words that matter.

Use streaming transcription where it is offered: partial transcripts arrive while the user is still speaking, so the text is nearly complete the moment they stop.

Endpointing: knowing when they have finished

The assistant has to decide that the user has stopped talking, and it does so by detecting silence. A voice-activity detector (Silero VAD is free and small) reports speech or not-speech every few tens of milliseconds, and the endpoint fires after a run of silence. Set that run to about 500 to 800 milliseconds. Shorter, and the assistant cuts people off in the pause before a name or a number; longer, and every reply feels slow, because the endpoint delay is paid in full on every turn before anything else begins.

Barge-in is the companion rule: if the user starts speaking while the assistant is talking, stop the speech immediately. People interrupt, and an assistant that talks over them is unusable.

The model in the middle

Prompt for the spoken register. No markdown, no lists, no headings, short sentences, and numbers written as words where the speech engine would mangle them: “two thousand and twenty-six” rather than a string of digits a synthesiser may read as a phone number. Keep answers short; a paragraph that reads well takes forty seconds to hear.

Then stream, and start speaking on the first sentence. This is the single largest latency win in the cascade. Waiting for the full answer before synthesising means the user waits for the model’s entire generation, several seconds for a long reply. Splitting the stream at sentence boundaries and sending each sentence to the speech engine as it completes brings the first audio forward to the model’s time to first token plus one sentence.

Text to speech

Free: Piper is fast, runs offline on a CPU and sounds acceptable; Coqui’s XTTS clones voices and wants a GPU. Hosted: ElevenLabs, OpenAI, Google and Microsoft price per character, in the region of $15 to $30 per million, so a hundred-word reply costs around a cent. Check language support specifically: Hindi and Tamil voices exist on the large platforms, quality varies, and a voice that is fine in English may be poor in the language your users speak. A cloned voice needs the consent of the person cloned; the create track of this catalogue covers that in full.

Speech-to-speech models

Newer models take audio in and produce audio out directly, skipping the cascade. OpenAI’s Realtime API and Google’s Gemini Live are examples. Latency drops to well under a second, interruptions feel natural, and the model hears tone rather than a transcript. The costs: audio is billed by the minute of conversation, at several cents, which is far above a text cascade; you have less control over each stage; and the transcript is a by-product rather than the source of truth. For a high-value, low-volume conversation they are the better experience. For a support line handling thousands of calls, the cascade’s economics usually win.

The budget, worked

Cascade, naive: 700 ms endpoint, 400 ms transcription, 800 ms to first token, 500 ms to first audio: 2.4 seconds, and it feels broken. Cascade, streamed: 600 ms endpoint, transcription already done, 400 ms to first token with a small model, 200 ms to first audio chunk: about 1.2 seconds. Speech-to-speech: 600 to 900 ms. Write yours down before building; every component vendor quotes its own number in isolation, and only the sum is real.

Telephony, and what voice is bad at

On a phone line the audio is 8 kHz and transcription accuracy drops; test with real call audio, not a laptop microphone. And voice is a poor channel for anything long or exact: a list of ten options, a postcode, a reference number. Read numbers back for confirmation, and offer a text fallback for the details. Where voice shines is for people who cannot comfortably type, which in this readership is many, and that is the reason to build it well, not the reason to build it everywhere.