Speech: word error rate, and the words that matter

The arithmetic, and why it can exceed 100%

Word error rate is edit distance over words, divided by the reference length:

WER = (substitutions + deletions + insertions) / reference_words

Reference: “please cancel my order for thursday”: six words. The system produces “please cancel my order thursday morning”: one deletion (“for”), one insertion (“morning”), so WER is 2/6 = 33%.

Because insertions are not bounded by the reference length, WER can exceed 100%. A model that hallucinates a paragraph during silence (a well-documented failure of some end-to-end systems) can score 400% on a short clip, which is a useful signal rather than a broken metric.

jiwer computes it in two lines and is free.

The normaliser decides the number

Two teams can report 8% and 14% on identical audio and identical transcripts. The difference is text normalisation:

• casing and punctuation removed, or not
• “25” against “twenty five”
• “Dr.” against “doctor”
• fillers (“um”, “uh”) kept or stripped
• contractions expanded
• Indian numbering, currency words, and mixed-script transliteration

None of these choices is wrong; publishing the score without the choices is. Use a published normaliser (the one shipped with Whisper is a de facto standard for English) and state which, or write your own and version it with the results. Then apply the identical normaliser to every system you compare, including the vendor’s.

For languages without clear word boundaries, and for heavily inflected ones, report character error rate as well; word-level scoring punishes an inflection difference as heavily as a wrong word.

WER is not what your product cares about

WER weights every word equally. Your booking assistant does not: “Thursday”, “four”, “Nandini”, and the order number carry the meaning, and “the” does not.

Measure entity accuracy separately (dates, times, quantities, names, identifiers, product terms) as a per-slot exact-match rate after normalisation. It is common to see 9% WER with 78% date accuracy, and the second number is the one that predicts complaints.

Then measure the pipeline. If the transcript feeds an assistant, what matters is end-to-end task success, and the relationship is not linear: a system at 12% WER can support 95% intent accuracy, because language models recover from small transcription errors and not from a wrong number. Measure the two stages and the end-to-end result, and improve whichever stage the end-to-end number blames.

The slices that decide whether it works for your users

Speech quality varies more across speakers and conditions than almost any other AI task. Report per slice, and build the slices deliberately:

• Accent and region. For a country with many first languages, this is the first cut.
• Code-switching. A sentence that starts in Hindi and ends in English defeats systems trained on either alone, and it is how a great many people actually speak.
• Background noise, banded by signal-to-noise ratio: a quiet room, a shop, a street, a moving auto.
• Channel. Telephony audio is 8 kHz and narrowband; a model evaluated on studio recordings will disappoint on a call.
• Speaking rate and age. Children and elderly speakers are consistently worse-served and consistently absent from test sets.

Common Voice and other openly licensed corpora provide starting material for several of these, and a hundred recordings collected from your own users in their own conditions is worth more than any of it.

Where the recordings come from

Two practical constraints shape the set before any metric applies. Recording a person’s voice is collecting biometric-adjacent personal data, so consent, retention and access control apply the same way they do to message logs, with the extra difficulty that a voice cannot be redacted the way an email address can. And a public corpus recorded by volunteers reading prompts is read speech, which is easier than spontaneous speech in every way that matters: no false starts, no overlap, no thinking aloud. Where you rely on read speech, say so, because the number will be optimistic against live calls.

Diarisation, timing and latency

If the output has speakers or timestamps, add two more measures. Diarisation error rate covers missed speech, false speech and speaker confusion. Alignment error matters wherever subtitles or highlights are shown; a transcript that is perfect and 400 ms late is a bad subtitle.

For streaming, latency is part of quality: time to first partial result, and the finalisation delay after the speaker stops. A system that is 2% better on WER and a second slower will feel worse in a live conversation, and neither number alone will tell you that.