{"course":{"slug":"ai-actually-explained","title":"AI, Actually Explained","url":"https://addaly.com/learn/ai-actually-explained","level":"beginner","track":"start"},"lesson":{"slug":"speech-to-text","title":"Machines that write down what you said","url":"https://addaly.com/learn/ai-actually-explained/speech-to-text","position":16,"of":92,"minutes":8,"answer":"Speech recognition trained on unevenly distributed voices has roughly double the error rate for some speakers, and models such as Whisper invent plausible sentences during silence because they were never trained to output nothing.","keyIdea":"Speech recognition trained on unevenly distributed voices has roughly double the error rate for some speakers, and models such as Whisper invent plausible sentences during silence because they were never trained to output nothing."},"body":{"format":"markdown","text":"## Where you already meet it\n\nVoice notes turned into text, automatic subtitles, dictation, call-centre transcripts, meeting summaries, the voice assistant in a car. All of it is **automatic speech recognition**, and it improved sharply around 2022 for the same reason everything else did: bigger models trained on far more audio.\n\nThe measure used is **word error rate**: the proportion of words inserted, deleted or substituted compared with a careful human transcript. On clean, read speech in standard American English, the best systems are now under 5%, which is roughly what a professional transcriptionist achieves. On a noisy market street with two people talking over each other, the same system may be over 40%.\n\nThat gap between the published number and the real room is the whole lesson.\n\n## Accent is not a detail\n\nSpeech systems learn from the voices in their training audio, and those voices are not evenly distributed.\n\nThe clearest documented example is a 2020 Stanford study published in *PNAS* that tested five commercial systems from major providers on identical read passages. Average word error rate was about 0.19 for white speakers and about 0.35 for Black speakers: close to double. The systems had not been designed to do this. They had been trained on audio in which one group was far better represented.\n\n> Figure: Word error rate, same systems, different speakers and rooms. Bars (% of words wrong): Clean read speech, standard American English 5; Read passages, white speakers (2020 study) 19; Same passages, Black speakers (2020 study) 35; Two people talking over a noisy street 42. The published figure is the first bar. The 2020 Stanford study in PNAS gave five commercial systems identical read passages and measured roughly double the error rate for Black speakers. Noise does more damage than any of it, and none of these is the number in the brochure.\n\nSimilar gaps appear for regional accents, for second-language speakers, for older voices, and for speech affected by illness or disability. If you have ever felt that a voice assistant works for other people and not for you, this is likely why, and it is not your pronunciation.\n\n## The failure mode that catches people out\n\nThere is a specific and important way modern speech models fail, and it is not what you expect.\n\nOpenAI's Whisper (the most widely used open speech model, free to download and run) has a documented tendency to **invent text during silence or noise**. Given a pause, it sometimes emits a plausible sentence that nobody said. Researchers examining medical and public-meeting transcripts in 2024 found fabricated sentences in a meaningful fraction of segments, including invented clinical detail.\n\nThe mechanism is worth understanding, because it recurs throughout this course. Whisper was trained to produce the text that goes with audio. Given audio containing nothing, it still produces its best guess at accompanying text, because producing nothing was rarely the right answer in training. The model has no way to say \"there was no speech here\".\n\nSo a transcript is not a recording. If the words matter (a consultation, a legal statement, a disciplinary meeting) keep the audio and spot-check the transcript against it, especially around pauses.\n\n## The parts of a transcript nobody recorded\n\nTwo features of a finished transcript were never in the audio at all, and both are predictions.\n\n**Punctuation and capitalisation.** Nobody speaks full stops. The model infers sentence boundaries from rhythm and wording, and where it puts them changes the meaning. A misplaced boundary can join a question to the answer that followed it.\n\n**Speaker labels.** Deciding who spoke each stretch is a separate task called diarisation, run by a separate model that clusters voices. It is markedly less reliable than the words themselves, and it degrades when people interrupt each other, when two speakers sound alike, or when someone is on a phone line. A transcript that attributes a sentence to the wrong person is a serious error in a meeting record, a disciplinary hearing or an interview, and it is the error people check for least.\n\nSo when you read \"Speaker 2: I agreed to that\", two independent systems had to be right. Treat attribution as the weakest claim on the page.\n\n## The free path, and it is a good one\n\nThis is an area where the free tools are competitive with paid services.\n\n- **Whisper**, released with open weights, runs on an ordinary laptop. **whisper.cpp** is an efficient implementation that runs even on a phone, offline, with no account and no upload.\n- **faster-whisper** speeds this up considerably on modest hardware.\n- **Vosk** offers small offline models for many languages, useful where storage and bandwidth are tight.\n- For subtitles, **Subtitle Edit** (free, Windows) and **Aegisub** handle timing and correction.\n\nRunning locally is not only about cost. A transcript produced offline never leaves the device, which for a medical, legal or personal recording is a stronger privacy guarantee than any policy document.\n\n## Practical technique\n\nThe controllable variables are worth more than the model choice. Get the microphone close to the speaker: halving the distance does more for accuracy than any setting. Reduce background noise rather than trying to filter it afterwards. Tell the system the language explicitly instead of letting it detect; misdetection produces spectacular nonsense. And where names, drugs or technical terms recur, most tools accept a hint list, which fixes the errors that matter most."},"check":{"question":"A transcript of a clinic recording contains a sentence about a medication that the doctor never mentioned. The audio at that point is a five-second pause. What does this most likely indicate?","options":[{"text":"The model picked up faint background speech from an adjacent room and transcribed it accurately.","correct":false,"tempting":"Assumes there was real audio to transcribe, which is plausible but does not explain fabrications during genuine silence."},{"text":"The transcription service mixed segments from a different recording during processing.","correct":false,"tempting":"Reaches for a data-handling bug when a well-documented property of the model already explains it."},{"text":"The audio was corrupted, and the model reconstructed the damaged section from surrounding context.","correct":false,"tempting":"Presumes a repair mechanism; the model is not reconstructing damaged audio, it is generating text where there was none."},{"text":"The model generates the text that would accompany the audio, and having no way to output nothing, it produced a plausible sentence for the silence.","correct":true}]},"prev":{"slug":"machine-translation","title":"Translation, and the fluency trap","url":"https://addaly.com/learn/ai-actually-explained/machine-translation"},"next":{"slug":"decisions-made-about-you","title":"When a model decides something that matters","url":"https://addaly.com/learn/ai-actually-explained/decisions-made-about-you"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Machines that write down what you said', in AI, Actually Explained. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-actually-explained/speech-to-text (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Machines that write down what you said","courseTitle":"AI, Actually Explained","module":{"index":2,"slug":"where-it-already-is","title":"Where it already is, without a label"},"answer":null,"keyIdea":"Speech recognition trained on unevenly distributed voices has roughly double the error rate for some speakers, and models such as Whisper invent plausible sentences during silence because they were never trained to output nothing.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"% of words wrong","title":"Word error rate, same systems, different speakers and rooms","series":[{"label":"Clean read speech, standard American English","value":5},{"label":"Read passages, white speakers (2020 study)","value":19},{"label":"Same passages, Black speakers (2020 study)","value":35,"highlight":true},{"label":"Two people talking over a noisy street","value":42}],"caption":"The published figure is the first bar. The 2020 Stanford study in PNAS gave five commercial systems identical read passages and measured roughly double the error rate for Black speakers. Noise does more damage than any of it, and none of these is the number in the brochure."},"alt":"Figure: Word error rate, same systems, different speakers and rooms. Bars (% of words wrong): Clean read speech, standard American English 5; Read passages, white speakers (2020 study) 19; Same passages, Black speakers (2020 study) 35; Two people talking over a noisy street 42. The published figure is the first bar. The 2020 Stanford study in PNAS gave five commercial systems identical read passages and measured roughly double the error rate for Black speakers. Noise does more damage than any of it, and none of these is the number in the brochure."}},"blocks":[{"t":"h2","id":"where-you-already-meet-it","runs":[{"text":"Where you already meet it"}]},{"t":"p","runs":[{"text":"Voice notes turned into text, automatic subtitles, dictation, call-centre transcripts, meeting summaries, the voice assistant in a car. All of it is "},{"text":"automatic speech recognition","b":true},{"text":", and it improved sharply around 2022 for the same reason everything else did: bigger models trained on far more audio."}]},{"t":"p","runs":[{"text":"The measure used is "},{"text":"word error rate","b":true},{"text":": the proportion of words inserted, deleted or substituted compared with a careful human transcript. On clean, read speech in standard American English, the best systems are now under 5%, which is roughly what a professional transcriptionist achieves. On a noisy market street with two people talking over each other, the same system may be over 40%."}]},{"t":"p","runs":[{"text":"That gap between the published number and the real room is the whole lesson."}]},{"t":"h2","id":"accent-is-not-a-detail","runs":[{"text":"Accent is not a detail"}]},{"t":"p","runs":[{"text":"Speech systems learn from the voices in their training audio, and those voices are not evenly distributed."}]},{"t":"p","runs":[{"text":"The clearest documented example is a 2020 Stanford study published in "},{"text":"PNAS","i":true},{"text":" that tested five commercial systems from major providers on identical read passages. Average word error rate was about 0.19 for white speakers and about 0.35 for Black speakers: close to double. The systems had not been designed to do this. They had been trained on audio in which one group was far better represented."}]},{"t":"p","runs":[{"text":"Similar gaps appear for regional accents, for second-language speakers, for older voices, and for speech affected by illness or disability. If you have ever felt that a voice assistant works for other people and not for you, this is likely why, and it is not your pronunciation."}]},{"t":"h2","id":"the-failure-mode-that-catches-people-out","runs":[{"text":"The failure mode that catches people out"}]},{"t":"p","runs":[{"text":"There is a specific and important way modern speech models fail, and it is not what you expect."}]},{"t":"p","runs":[{"text":"OpenAI’s Whisper (the most widely used open speech model, free to download and run) has a documented tendency to "},{"text":"invent text during silence or noise","b":true},{"text":". Given a pause, it sometimes emits a plausible sentence that nobody said. Researchers examining medical and public-meeting transcripts in 2024 found fabricated sentences in a meaningful fraction of segments, including invented clinical detail."}]},{"t":"p","runs":[{"text":"The mechanism is worth understanding, because it recurs throughout this course. Whisper was trained to produce the text that goes with audio. Given audio containing nothing, it still produces its best guess at accompanying text, because producing nothing was rarely the right answer in training. The model has no way to say “there was no speech here”."}]},{"t":"p","runs":[{"text":"So a transcript is not a recording. If the words matter (a consultation, a legal statement, a disciplinary meeting) keep the audio and spot-check the transcript against it, especially around pauses."}]},{"t":"h2","id":"the-parts-of-a-transcript-nobody-recorded","runs":[{"text":"The parts of a transcript nobody recorded"}]},{"t":"p","runs":[{"text":"Two features of a finished transcript were never in the audio at all, and both are predictions."}]},{"t":"p","runs":[{"text":"Punctuation and capitalisation.","b":true},{"text":" Nobody speaks full stops. The model infers sentence boundaries from rhythm and wording, and where it puts them changes the meaning. A misplaced boundary can join a question to the answer that followed it."}]},{"t":"p","runs":[{"text":"Speaker labels.","b":true},{"text":" Deciding who spoke each stretch is a separate task called diarisation, run by a separate model that clusters voices. It is markedly less reliable than the words themselves, and it degrades when people interrupt each other, when two speakers sound alike, or when someone is on a phone line. A transcript that attributes a sentence to the wrong person is a serious error in a meeting record, a disciplinary hearing or an interview, and it is the error people check for least."}]},{"t":"p","runs":[{"text":"So when you read “Speaker 2: I agreed to that”, two independent systems had to be right. Treat attribution as the weakest claim on the page."}]},{"t":"h2","id":"the-free-path-and-it-is-a-good-one","runs":[{"text":"The free path, and it is a good one"}],"role":"free"},{"t":"p","runs":[{"text":"This is an area where the free tools are competitive with paid services."}]},{"t":"ul","items":[[{"text":"Whisper","b":true},{"text":", released with open weights, runs on an ordinary laptop. "},{"text":"whisper.cpp","b":true},{"text":" is an efficient implementation that runs even on a phone, offline, with no account and no upload."}],[{"text":"faster-whisper","b":true},{"text":" speeds this up considerably on modest hardware."}],[{"text":"Vosk","b":true},{"text":" offers small offline models for many languages, useful where storage and bandwidth are tight."}],[{"text":"For subtitles, "},{"text":"Subtitle Edit","b":true},{"text":" (free, Windows) and "},{"text":"Aegisub","b":true},{"text":" handle timing and correction."}]]},{"t":"p","runs":[{"text":"Running locally is not only about cost. A transcript produced offline never leaves the device, which for a medical, legal or personal recording is a stronger privacy guarantee than any policy document."}]},{"t":"h2","id":"practical-technique","runs":[{"text":"Practical technique"}]},{"t":"p","runs":[{"text":"The controllable variables are worth more than the model choice. Get the microphone close to the speaker: halving the distance does more for accuracy than any setting. Reduce background noise rather than trying to filter it afterwards. Tell the system the language explicitly instead of letting it detect; misdetection produces spectacular nonsense. And where names, drugs or technical terms recur, most tools accept a hint list, which fixes the errors that matter most."}]}],"check":{"question":[{"text":"A transcript of a clinic recording contains a sentence about a medication that the doctor never mentioned. The audio at that point is a five-second pause. What does this most likely indicate?"}],"options":[{"text":[{"text":"The model picked up faint background speech from an adjacent room and transcribed it accurately."}],"correct":false,"tempting":[{"text":"Assumes there was real audio to transcribe, which is plausible but does not explain fabrications during genuine silence."}]},{"text":[{"text":"The transcription service mixed segments from a different recording during processing."}],"correct":false,"tempting":[{"text":"Reaches for a data-handling bug when a well-documented property of the model already explains it."}]},{"text":[{"text":"The audio was corrupted, and the model reconstructed the damaged section from surrounding context."}],"correct":false,"tempting":[{"text":"Presumes a repair mechanism; the model is not reconstructing damaged audio, it is generating text where there was none."}]},{"text":[{"text":"The model generates the text that would accompany the audio, and having no way to output nothing, it produced a plausible sentence for the silence."}],"correct":true,"tempting":null}],"order":[2,3,0,1],"why":null},"prev":{"slug":"machine-translation","title":"Translation, and the fluency trap","minutes":9},"next":{"slug":"decisions-made-about-you","title":"When a model decides something that matters","minutes":10}}}