{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"voice-in-and-out","title":"Voice in and out: transcription, speech and the latency budget","url":"https://addaly.com/learn/building-with-ai/voice-in-and-out","position":48,"of":80,"minutes":9,"answer":"A voice assistant is a cascade of endpointing, transcription, generation and speech whose delays add up, so a reply that feels natural comes from streaming every stage (partial transcripts, a model prompted for spoken register.","keyIdea":"A voice assistant is a cascade of endpointing, transcription, generation and speech whose delays add up, so a reply that feels natural comes from streaming every stage (partial transcripts, a model prompted for spoken register, and speech started on the first sentence) rather than from any one faster component."},"body":{"format":"markdown","text":"## Two pipelines and one budget\n\nThe classic voice assistant is a cascade: speech comes in, a speech-to-text model transcribes it, the language model answers in text, and a text-to-speech model speaks the answer. Each stage adds delay, and the delays add. People expect a reply to begin within about a second of finishing a sentence; past two seconds the silence reads as a fault. That budget, roughly a thousand milliseconds from the user's last word to the first spoken syllable of the reply, is the constraint every decision in this lesson serves.\n\n## Speech to text\n\nWhisper, released openly by OpenAI, is the free baseline. Through `faster-whisper` the small models transcribe faster than real time on a CPU; the large model wants a GPU. Hosted transcription from Deepgram, AssemblyAI, Google, Microsoft and OpenAI costs fractions of a cent per minute of audio, which for most products is negligible next to the language model.\n\nFor Indian languages, test before choosing. Whisper handles Hindi reasonably and is weaker on Tamil, Bengali and regional accents; AI4Bharat's Indic models are free and trained specifically on these languages, and are often the better choice. Hinglish (a sentence that switches language halfway through) is the hard case for every model, and it is what a large share of your users will speak. Put twenty real recordings through each candidate and count the errors on names and numbers, which are the words that matter.\n\nUse streaming transcription where it is offered: partial transcripts arrive while the user is still speaking, so the text is nearly complete the moment they stop.\n\n## Endpointing: knowing when they have finished\n\nThe assistant has to decide that the user has stopped talking, and it does so by detecting silence. A voice-activity detector (Silero VAD is free and small) reports speech or not-speech every few tens of milliseconds, and the endpoint fires after a run of silence. Set that run to about 500 to 800 milliseconds. Shorter, and the assistant cuts people off in the pause before a name or a number; longer, and every reply feels slow, because the endpoint delay is paid in full on every turn before anything else begins.\n\nBarge-in is the companion rule: if the user starts speaking while the assistant is talking, stop the speech immediately. People interrupt, and an assistant that talks over them is unusable.\n\n## The model in the middle\n\nPrompt for the spoken register. No markdown, no lists, no headings, short sentences, and numbers written as words where the speech engine would mangle them: \"two thousand and twenty-six\" rather than a string of digits a synthesiser may read as a phone number. Keep answers short; a paragraph that reads well takes forty seconds to hear.\n\nThen stream, and start speaking on the first sentence. This is the single largest latency win in the cascade. Waiting for the full answer before synthesising means the user waits for the model's entire generation, several seconds for a long reply. Splitting the stream at sentence boundaries and sending each sentence to the speech engine as it completes brings the first audio forward to the model's time to first token plus one sentence.\n\n## Text to speech\n\nFree: Piper is fast, runs offline on a CPU and sounds acceptable; Coqui's XTTS clones voices and wants a GPU. Hosted: ElevenLabs, OpenAI, Google and Microsoft price per character, in the region of $15 to $30 per million, so a hundred-word reply costs around a cent. Check language support specifically: Hindi and Tamil voices exist on the large platforms, quality varies, and a voice that is fine in English may be poor in the language your users speak. A cloned voice needs the consent of the person cloned; the create track of this catalogue covers that in full.\n\n## Speech-to-speech models\n\nNewer models take audio in and produce audio out directly, skipping the cascade. OpenAI's Realtime API and Google's Gemini Live are examples. Latency drops to well under a second, interruptions feel natural, and the model hears tone rather than a transcript. The costs: audio is billed by the minute of conversation, at several cents, which is far above a text cascade; you have less control over each stage; and the transcript is a by-product rather than the source of truth. For a high-value, low-volume conversation they are the better experience. For a support line handling thousands of calls, the cascade's economics usually win.\n\n## The budget, worked\n\nCascade, naive: 700 ms endpoint, 400 ms transcription, 800 ms to first token, 500 ms to first audio: 2.4 seconds, and it feels broken. Cascade, streamed: 600 ms endpoint, transcription already done, 400 ms to first token with a small model, 200 ms to first audio chunk: about 1.2 seconds. Speech-to-speech: 600 to 900 ms. Write yours down before building; every component vendor quotes its own number in isolation, and only the sum is real.\n\n## Telephony, and what voice is bad at\n\nOn a phone line the audio is 8 kHz and transcription accuracy drops; test with real call audio, not a laptop microphone. And voice is a poor channel for anything long or exact: a list of ten options, a postcode, a reference number. Read numbers back for confirmation, and offer a text fallback for the details. Where voice shines is for people who cannot comfortably type, which in this readership is many, and that is the reason to build it well, not the reason to build it everywhere.\n\n> Figure: From the user's last word to the first spoken syllable. Bars (milliseconds to the first spoken syllable): Cascade, nothing streamed 2400; Cascade, every stage streamed 1200; Speech-to-speech model 750. Streaming every stage, rather than making any one component faster, is what moves the number: partial transcripts arrive while the person is still talking, and speech starts on the first completed sentence. Past two seconds the silence reads as a fault."},"check":{"question":"A voice assistant waits for the language model to finish its whole reply before sending the text to the speech engine. Replies start after four seconds. Which change most reduces that wait?","options":[{"text":"Switch to a faster speech engine, since synthesis of the full reply is the bottleneck that holds back the start of playback on every turn.","correct":false,"tempting":"Synthesis time is small next to generation; a faster engine trims a fraction while the wait for the full reply remains."},{"text":"Reduce the silence threshold of the endpoint detector so the pipeline starts sooner.","correct":false,"tempting":"Saves a few hundred milliseconds at the start and cuts users off mid-sentence; it does not touch the generation wait."},{"text":"Stream the model's text and synthesise each sentence as it completes, so audio starts after the first sentence, not the last.","correct":true},{"text":"Use a larger language model that generates its reply in fewer tokens.","correct":false,"tempting":"A larger model is usually slower per token, and the reply length is a prompting matter, not a model-size one."}]},"prev":{"slug":"the-chat-interface-honestly","title":"The chat interface: the controls that are load-bearing","url":"https://addaly.com/learn/building-with-ai/the-chat-interface-honestly"},"next":{"slug":"ai-inside-the-existing-ui","title":"AI inside the existing screen: ghost text, drafts and the acceptance rate","url":"https://addaly.com/learn/building-with-ai/ai-inside-the-existing-ui"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Voice in and out: transcription, speech and the latency budget', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/voice-in-and-out (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Voice in and out: transcription, speech and the latency budget","courseTitle":"Building with AI","module":{"index":5,"slug":"conversation-memory-and-the-interface","title":"Conversation, memory and the interface"},"answer":null,"keyIdea":"A voice assistant is a cascade of endpointing, transcription, generation and speech whose delays add up, so a reply that feels natural comes from streaming every stage (partial transcripts, a model prompted for spoken register, and speech started on the first sentence) rather than from any one faster component.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"milliseconds to the first spoken syllable","title":"From the user’s last word to the first spoken syllable","series":[{"label":"Cascade, nothing streamed","value":2400},{"label":"Cascade, every stage streamed","value":1200,"highlight":true},{"label":"Speech-to-speech model","value":750}],"caption":"Streaming every stage, rather than making any one component faster, is what moves the number: partial transcripts arrive while the person is still talking, and speech starts on the first completed sentence. Past two seconds the silence reads as a fault."},"alt":"Figure: From the user’s last word to the first spoken syllable. Bars (milliseconds to the first spoken syllable): Cascade, nothing streamed 2400; Cascade, every stage streamed 1200; Speech-to-speech model 750. Streaming every stage, rather than making any one component faster, is what moves the number: partial transcripts arrive while the person is still talking, and speech starts on the first completed sentence. Past two seconds the silence reads as a fault."}},"blocks":[{"t":"h2","id":"two-pipelines-and-one-budget","runs":[{"text":"Two pipelines and one budget"}]},{"t":"p","runs":[{"text":"The classic voice assistant is a cascade: speech comes in, a speech-to-text model transcribes it, the language model answers in text, and a text-to-speech model speaks the answer. Each stage adds delay, and the delays add. People expect a reply to begin within about a second of finishing a sentence; past two seconds the silence reads as a fault. That budget, roughly a thousand milliseconds from the user’s last word to the first spoken syllable of the reply, is the constraint every decision in this lesson serves."}]},{"t":"h2","id":"speech-to-text","runs":[{"text":"Speech to text"}]},{"t":"p","runs":[{"text":"Whisper, released openly by OpenAI, is the free baseline. Through "},{"text":"faster-whisper","code":true},{"text":" the small models transcribe faster than real time on a CPU; the large model wants a GPU. Hosted transcription from Deepgram, AssemblyAI, Google, Microsoft and OpenAI costs fractions of a cent per minute of audio, which for most products is negligible next to the language model."}]},{"t":"p","runs":[{"text":"For Indian languages, test before choosing. Whisper handles Hindi reasonably and is weaker on Tamil, Bengali and regional accents; AI4Bharat’s Indic models are free and trained specifically on these languages, and are often the better choice. Hinglish (a sentence that switches language halfway through) is the hard case for every model, and it is what a large share of your users will speak. Put twenty real recordings through each candidate and count the errors on names and numbers, which are the words that matter."}]},{"t":"p","runs":[{"text":"Use streaming transcription where it is offered: partial transcripts arrive while the user is still speaking, so the text is nearly complete the moment they stop."}]},{"t":"h2","id":"endpointing-knowing-when-they-have-finished","runs":[{"text":"Endpointing: knowing when they have finished"}]},{"t":"p","runs":[{"text":"The assistant has to decide that the user has stopped talking, and it does so by detecting silence. A voice-activity detector (Silero VAD is free and small) reports speech or not-speech every few tens of milliseconds, and the endpoint fires after a run of silence. Set that run to about 500 to 800 milliseconds. Shorter, and the assistant cuts people off in the pause before a name or a number; longer, and every reply feels slow, because the endpoint delay is paid in full on every turn before anything else begins."}]},{"t":"p","runs":[{"text":"Barge-in is the companion rule: if the user starts speaking while the assistant is talking, stop the speech immediately. People interrupt, and an assistant that talks over them is unusable."}]},{"t":"h2","id":"the-model-in-the-middle","runs":[{"text":"The model in the middle"}]},{"t":"p","runs":[{"text":"Prompt for the spoken register. No markdown, no lists, no headings, short sentences, and numbers written as words where the speech engine would mangle them: “two thousand and twenty-six” rather than a string of digits a synthesiser may read as a phone number. Keep answers short; a paragraph that reads well takes forty seconds to hear."}]},{"t":"p","runs":[{"text":"Then stream, and start speaking on the first sentence. This is the single largest latency win in the cascade. Waiting for the full answer before synthesising means the user waits for the model’s entire generation, several seconds for a long reply. Splitting the stream at sentence boundaries and sending each sentence to the speech engine as it completes brings the first audio forward to the model’s time to first token plus one sentence."}]},{"t":"h2","id":"text-to-speech","runs":[{"text":"Text to speech"}]},{"t":"p","runs":[{"text":"Free: Piper is fast, runs offline on a CPU and sounds acceptable; Coqui’s XTTS clones voices and wants a GPU. Hosted: ElevenLabs, OpenAI, Google and Microsoft price per character, in the region of $15 to $30 per million, so a hundred-word reply costs around a cent. Check language support specifically: Hindi and Tamil voices exist on the large platforms, quality varies, and a voice that is fine in English may be poor in the language your users speak. A cloned voice needs the consent of the person cloned; the create track of this catalogue covers that in full."}]},{"t":"h2","id":"speech-to-speech-models","runs":[{"text":"Speech-to-speech models"}]},{"t":"p","runs":[{"text":"Newer models take audio in and produce audio out directly, skipping the cascade. OpenAI’s Realtime API and Google’s Gemini Live are examples. Latency drops to well under a second, interruptions feel natural, and the model hears tone rather than a transcript. The costs: audio is billed by the minute of conversation, at several cents, which is far above a text cascade; you have less control over each stage; and the transcript is a by-product rather than the source of truth. For a high-value, low-volume conversation they are the better experience. For a support line handling thousands of calls, the cascade’s economics usually win."}]},{"t":"h2","id":"the-budget-worked","runs":[{"text":"The budget, worked"}]},{"t":"p","runs":[{"text":"Cascade, naive: 700 ms endpoint, 400 ms transcription, 800 ms to first token, 500 ms to first audio: 2.4 seconds, and it feels broken. Cascade, streamed: 600 ms endpoint, transcription already done, 400 ms to first token with a small model, 200 ms to first audio chunk: about 1.2 seconds. Speech-to-speech: 600 to 900 ms. Write yours down before building; every component vendor quotes its own number in isolation, and only the sum is real."}]},{"t":"h2","id":"telephony-and-what-voice-is-bad-at","runs":[{"text":"Telephony, and what voice is bad at"}]},{"t":"p","runs":[{"text":"On a phone line the audio is 8 kHz and transcription accuracy drops; test with real call audio, not a laptop microphone. And voice is a poor channel for anything long or exact: a list of ten options, a postcode, a reference number. Read numbers back for confirmation, and offer a text fallback for the details. Where voice shines is for people who cannot comfortably type, which in this readership is many, and that is the reason to build it well, not the reason to build it everywhere."}]}],"check":{"question":[{"text":"A voice assistant waits for the language model to finish its whole reply before sending the text to the speech engine. Replies start after four seconds. Which change most reduces that wait?"}],"options":[{"text":[{"text":"Switch to a faster speech engine, since synthesis of the full reply is the bottleneck that holds back the start of playback on every turn."}],"correct":false,"tempting":[{"text":"Synthesis time is small next to generation; a faster engine trims a fraction while the wait for the full reply remains."}]},{"text":[{"text":"Reduce the silence threshold of the endpoint detector so the pipeline starts sooner."}],"correct":false,"tempting":[{"text":"Saves a few hundred milliseconds at the start and cuts users off mid-sentence; it does not touch the generation wait."}]},{"text":[{"text":"Stream the model’s text and synthesise each sentence as it completes, so audio starts after the first sentence, not the last."}],"correct":true,"tempting":null},{"text":[{"text":"Use a larger language model that generates its reply in fewer tokens."}],"correct":false,"tempting":[{"text":"A larger model is usually slower per token, and the reply length is a prompting matter, not a model-size one."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"the-chat-interface-honestly","title":"The chat interface: the controls that are load-bearing","minutes":9},"next":{"slug":"ai-inside-the-existing-ui","title":"AI inside the existing screen: ghost text, drafts and the acceptance rate","minutes":8}}}