{"course":{"slug":"evaluating-ai","title":"Knowing If It Works","url":"https://addaly.com/learn/evaluating-ai","level":"intermediate","track":"build"},"lesson":{"slug":"evaluating-speech","title":"Speech: word error rate, and the words that matter","url":"https://addaly.com/learn/evaluating-ai/evaluating-speech","position":37,"of":76,"minutes":9,"answer":"Word error rate is meaningless without the text normaliser that produced it, and it weights every word equally while your product depends on names, numbers and dates.","keyIdea":"Word error rate is meaningless without the text normaliser that produced it, and it weights every word equally while your product depends on names, numbers and dates, so measure entity accuracy and end-to-end task success beside it."},"body":{"format":"markdown","text":"## The arithmetic, and why it can exceed 100%\n\nWord error rate is edit distance over words, divided by the reference length:\n\n```\nWER = (substitutions + deletions + insertions) / reference_words\n```\n\nReference: \"please cancel my order for thursday\": six words. The system produces \"please cancel my order thursday morning\": one deletion (\"for\"), one insertion (\"morning\"), so WER is 2/6 = 33%.\n\nBecause insertions are not bounded by the reference length, WER can exceed 100%. A model that hallucinates a paragraph during silence (a well-documented failure of some end-to-end systems) can score 400% on a short clip, which is a useful signal rather than a broken metric.\n\n`jiwer` computes it in two lines and is free.\n\n## The normaliser decides the number\n\nTwo teams can report 8% and 14% on identical audio and identical transcripts. The difference is text normalisation:\n\n- casing and punctuation removed, or not\n- \"25\" against \"twenty five\"\n- \"Dr.\" against \"doctor\"\n- fillers (\"um\", \"uh\") kept or stripped\n- contractions expanded\n- Indian numbering, currency words, and mixed-script transliteration\n\nNone of these choices is wrong; publishing the score without the choices is. Use a published normaliser (the one shipped with Whisper is a de facto standard for English) and state which, or write your own and version it with the results. Then apply the identical normaliser to every system you compare, including the vendor's.\n\nFor languages without clear word boundaries, and for heavily inflected ones, report **character error rate** as well; word-level scoring punishes an inflection difference as heavily as a wrong word.\n\n## WER is not what your product cares about\n\nWER weights every word equally. Your booking assistant does not: \"Thursday\", \"four\", \"Nandini\", and the order number carry the meaning, and \"the\" does not.\n\nMeasure **entity accuracy** separately (dates, times, quantities, names, identifiers, product terms) as a per-slot exact-match rate after normalisation. It is common to see 9% WER with 78% date accuracy, and the second number is the one that predicts complaints.\n\nThen measure the pipeline. If the transcript feeds an assistant, what matters is end-to-end task success, and the relationship is not linear: a system at 12% WER can support 95% intent accuracy, because language models recover from small transcription errors and not from a wrong number. Measure the two stages and the end-to-end result, and improve whichever stage the end-to-end number blames.\n\n## The slices that decide whether it works for your users\n\nSpeech quality varies more across speakers and conditions than almost any other AI task. Report per slice, and build the slices deliberately:\n\n- **Accent and region.** For a country with many first languages, this is the first cut.\n- **Code-switching.** A sentence that starts in Hindi and ends in English defeats systems trained on either alone, and it is how a great many people actually speak.\n- **Background noise**, banded by signal-to-noise ratio: a quiet room, a shop, a street, a moving auto.\n- **Channel.** Telephony audio is 8 kHz and narrowband; a model evaluated on studio recordings will disappoint on a call.\n- **Speaking rate and age.** Children and elderly speakers are consistently worse-served and consistently absent from test sets.\n\nCommon Voice and other openly licensed corpora provide starting material for several of these, and a hundred recordings collected from your own users in their own conditions is worth more than any of it.\n\n## Where the recordings come from\n\nTwo practical constraints shape the set before any metric applies. Recording a person's voice is collecting biometric-adjacent personal data, so consent, retention and access control apply the same way they do to message logs, with the extra difficulty that a voice cannot be redacted the way an email address can. And a public corpus recorded by volunteers reading prompts is read speech, which is easier than spontaneous speech in every way that matters: no false starts, no overlap, no thinking aloud. Where you rely on read speech, say so, because the number will be optimistic against live calls.\n\n## Diarisation, timing and latency\n\nIf the output has speakers or timestamps, add two more measures. **Diarisation error rate** covers missed speech, false speech and speaker confusion. **Alignment error** matters wherever subtitles or highlights are shown; a transcript that is perfect and 400 ms late is a bad subtitle.\n\nFor streaming, latency is part of quality: time to first partial result, and the finalisation delay after the speaker stops. A system that is 2% better on WER and a second slower will feel worse in a live conversation, and neither number alone will tell you that."},"check":{"question":"A vendor reports 6% WER and your own measurement of the same audio gives 13%. What is the most likely explanation to check first?","options":[{"text":"The vendor evaluated on a different, cleaner audio set than the one you used.","correct":false,"tempting":"Worth confirming, but the question states the audio is the same, and the normalisation difference explains a gap of this size on identical audio."},{"text":"Your reference transcripts contain errors, which inflate the measured error rate.","correct":false,"tempting":"Reference quality does matter and typically shifts scores by a point or two, not by seven on well-prepared transcripts."},{"text":"The two measurements used different text normalisation (casing, punctuation, numbers and fillers) which routinely moves WER by several points.","correct":true},{"text":"WER is computed per utterance by some tools and per corpus by others, and the two differ.","correct":false,"tempting":"Aggregation choice does change the number slightly, but it is a much smaller effect than normalisation and is not the standard source of a gap this wide."}]},"prev":{"slug":"evaluating-vision-models","title":"Images: the background the model is actually reading","url":"https://addaly.com/learn/evaluating-ai/evaluating-vision-models"},"next":{"slug":"evaluating-documents-and-ocr","title":"Documents: stages that multiply, and checks that need no labels","url":"https://addaly.com/learn/evaluating-ai/evaluating-documents-and-ocr"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Speech: word error rate, and the words that matter', in Knowing If It Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/evaluating-ai/evaluating-speech (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Speech: word error rate, and the words that matter","courseTitle":"Knowing If It Works","module":{"index":4,"slug":"task-shapes","title":"The systems you will actually be handed"},"answer":null,"keyIdea":"Word error rate is meaningless without the text normaliser that produced it, and it weights every word equally while your product depends on names, numbers and dates, so measure entity accuracy and end-to-end task success beside it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-arithmetic-and-why-it-can-exceed-100","runs":[{"text":"The arithmetic, and why it can exceed 100%"}]},{"t":"p","runs":[{"text":"Word error rate is edit distance over words, divided by the reference length:"}]},{"t":"code","lang":"","label":"Text","text":"WER = (substitutions + deletions + insertions) / reference_words"},{"t":"p","runs":[{"text":"Reference: “please cancel my order for thursday”: six words. The system produces “please cancel my order thursday morning”: one deletion (“for”), one insertion (“morning”), so WER is 2/6 = 33%."}]},{"t":"p","runs":[{"text":"Because insertions are not bounded by the reference length, WER can exceed 100%. A model that hallucinates a paragraph during silence (a well-documented failure of some end-to-end systems) can score 400% on a short clip, which is a useful signal rather than a broken metric."}]},{"t":"p","runs":[{"text":"jiwer","code":true},{"text":" computes it in two lines and is free."}]},{"t":"h2","id":"the-normaliser-decides-the-number","runs":[{"text":"The normaliser decides the number"}]},{"t":"p","runs":[{"text":"Two teams can report 8% and 14% on identical audio and identical transcripts. The difference is text normalisation:"}]},{"t":"ul","items":[[{"text":"casing and punctuation removed, or not"}],[{"text":"“25” against “twenty five”"}],[{"text":"“Dr.” against “doctor”"}],[{"text":"fillers (“um”, “uh”) kept or stripped"}],[{"text":"contractions expanded"}],[{"text":"Indian numbering, currency words, and mixed-script transliteration"}]]},{"t":"p","runs":[{"text":"None of these choices is wrong; publishing the score without the choices is. Use a published normaliser (the one shipped with Whisper is a de facto standard for English) and state which, or write your own and version it with the results. Then apply the identical normaliser to every system you compare, including the vendor’s."}]},{"t":"p","runs":[{"text":"For languages without clear word boundaries, and for heavily inflected ones, report "},{"text":"character error rate","b":true},{"text":" as well; word-level scoring punishes an inflection difference as heavily as a wrong word."}]},{"t":"h2","id":"wer-is-not-what-your-product-cares-about","runs":[{"text":"WER is not what your product cares about"}]},{"t":"p","runs":[{"text":"WER weights every word equally. Your booking assistant does not: “Thursday”, “four”, “Nandini”, and the order number carry the meaning, and “the” does not."}]},{"t":"p","runs":[{"text":"Measure "},{"text":"entity accuracy","b":true},{"text":" separately (dates, times, quantities, names, identifiers, product terms) as a per-slot exact-match rate after normalisation. It is common to see 9% WER with 78% date accuracy, and the second number is the one that predicts complaints."}]},{"t":"p","runs":[{"text":"Then measure the pipeline. If the transcript feeds an assistant, what matters is end-to-end task success, and the relationship is not linear: a system at 12% WER can support 95% intent accuracy, because language models recover from small transcription errors and not from a wrong number. Measure the two stages and the end-to-end result, and improve whichever stage the end-to-end number blames."}]},{"t":"h2","id":"the-slices-that-decide-whether-it-works-for-your-users","runs":[{"text":"The slices that decide whether it works for your users"}]},{"t":"p","runs":[{"text":"Speech quality varies more across speakers and conditions than almost any other AI task. Report per slice, and build the slices deliberately:"}]},{"t":"ul","items":[[{"text":"Accent and region.","b":true},{"text":" For a country with many first languages, this is the first cut."}],[{"text":"Code-switching.","b":true},{"text":" A sentence that starts in Hindi and ends in English defeats systems trained on either alone, and it is how a great many people actually speak."}],[{"text":"Background noise","b":true},{"text":", banded by signal-to-noise ratio: a quiet room, a shop, a street, a moving auto."}],[{"text":"Channel.","b":true},{"text":" Telephony audio is 8 kHz and narrowband; a model evaluated on studio recordings will disappoint on a call."}],[{"text":"Speaking rate and age.","b":true},{"text":" Children and elderly speakers are consistently worse-served and consistently absent from test sets."}]]},{"t":"p","runs":[{"text":"Common Voice and other openly licensed corpora provide starting material for several of these, and a hundred recordings collected from your own users in their own conditions is worth more than any of it."}]},{"t":"h2","id":"where-the-recordings-come-from","runs":[{"text":"Where the recordings come from"}]},{"t":"p","runs":[{"text":"Two practical constraints shape the set before any metric applies. Recording a person’s voice is collecting biometric-adjacent personal data, so consent, retention and access control apply the same way they do to message logs, with the extra difficulty that a voice cannot be redacted the way an email address can. And a public corpus recorded by volunteers reading prompts is read speech, which is easier than spontaneous speech in every way that matters: no false starts, no overlap, no thinking aloud. Where you rely on read speech, say so, because the number will be optimistic against live calls."}]},{"t":"h2","id":"diarisation-timing-and-latency","runs":[{"text":"Diarisation, timing and latency"}]},{"t":"p","runs":[{"text":"If the output has speakers or timestamps, add two more measures. "},{"text":"Diarisation error rate","b":true},{"text":" covers missed speech, false speech and speaker confusion. "},{"text":"Alignment error","b":true},{"text":" matters wherever subtitles or highlights are shown; a transcript that is perfect and 400 ms late is a bad subtitle."}]},{"t":"p","runs":[{"text":"For streaming, latency is part of quality: time to first partial result, and the finalisation delay after the speaker stops. A system that is 2% better on WER and a second slower will feel worse in a live conversation, and neither number alone will tell you that."}]}],"check":{"question":[{"text":"A vendor reports 6% WER and your own measurement of the same audio gives 13%. What is the most likely explanation to check first?"}],"options":[{"text":[{"text":"The vendor evaluated on a different, cleaner audio set than the one you used."}],"correct":false,"tempting":[{"text":"Worth confirming, but the question states the audio is the same, and the normalisation difference explains a gap of this size on identical audio."}]},{"text":[{"text":"Your reference transcripts contain errors, which inflate the measured error rate."}],"correct":false,"tempting":[{"text":"Reference quality does matter and typically shifts scores by a point or two, not by seven on well-prepared transcripts."}]},{"text":[{"text":"The two measurements used different text normalisation (casing, punctuation, numbers and fillers) which routinely moves WER by several points."}],"correct":true,"tempting":null},{"text":[{"text":"WER is computed per utterance by some tools and per corpus by others, and the two differ."}],"correct":false,"tempting":[{"text":"Aggregation choice does change the number slightly, but it is a much smaller effect than normalisation and is not the standard source of a gap this wide."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"evaluating-vision-models","title":"Images: the background the model is actually reading","minutes":9},"next":{"slug":"evaluating-documents-and-ocr","title":"Documents: stages that multiply, and checks that need no labels","minutes":9}}}