Five per cent, and which five per cent

What a WER figure is actually telling you

Automatic speech recognition is scored with word error rate:

WER = (substitutions + deletions + insertions) / words in the reference

A 5 per cent WER means one word in twenty is wrong. On a 1,000-word video that is fifty errors. Stated as a percentage it sounds like a rounding error; stated as fifty wrong words in ten minutes it sounds like what it is.

The figure also understates the damage, for a structural reason. Errors are not spread evenly. They concentrate on proper nouns, technical terms, numbers and rare words: precisely the content words a viewer is relying on the captions for. A transcript can be 95 per cent accurate and get every name in it wrong, and the 5 per cent that failed is the 5 per cent that mattered.

Where accuracy actually goes

• Overlapping speech. Two people talking at once defeats most systems, and this is most of a real conversation.
• Accents and dialects under-represented in training data. Published research has repeatedly found substantially higher error rates for some speaker groups than others on the same systems; this is a known and documented disparity, not an anomaly.
• Background noise and music.
• Domain vocabulary. Medical, legal, technical and organisational terms.
• Code-switching. Moving between languages mid-sentence, which is ordinary speech for a large part of the world and poorly handled.

Many systems accept a vocabulary or prompt hint. Supplying a list of the names and terms in your video before transcription is the single cheapest accuracy improvement available.

The hallucination problem

Whisper, the most widely used open model, has a documented failure mode that is different in kind from ordinary errors: during silence, music, or non-speech audio, it can generate fluent text that nobody said. Researchers examining transcriptions have found invented sentences, and the phrase “Thank you for watching” appearing at the ends of clips is a well-known artefact of its training data.

This matters because the output is confident and well-formed. A conventional recogniser fails visibly, producing garbage you can see. A generative one fails invisibly, producing a plausible sentence.

Mitigations that help: setting a voice-activity detector to skip non-speech, using condition_on_previous_text=False to stop an error propagating forward, and checking any segment of silence in the output. None of them removes the need to read the transcript.

Captions as a craft, not an export

An auto-generated caption track is a starting point. The conventions that make captions readable are well established:

• 32 to 42 characters per line, at most two lines on screen.
• Reading rate around 160 to 180 words per minute for adult viewers, often expressed as 17 to 20 characters per second. Faster than that and people are still reading when the caption changes.
• Minimum one second on screen, even for one word; maximum about six or seven seconds.
• Break lines at grammatical boundaries (after a clause, before a preposition) not wherever the width runs out. A line broken mid-phrase measurably slows reading.
• Do not cover the speaker’s mouth or burned-in text.
• Identify speakers when it is not obvious, and describe significant non-speech sound for SDH.

Subtitles assume the viewer can hear and translate or transcribe dialogue. SDH (subtitles for the deaf and hard of hearing) additionally carry speaker identification and sound effects. They are different deliverables.

Burned-in or sidecar

Sidecar files (SRT, WebVTT, TTML) are separate and referenced by the player. They can be toggled off, restyled, searched, indexed by search engines, and translated. They are the correct default.

Burned-in captions are pixels. They survive platforms that strip sidecar tracks, they cannot be turned off, they cannot be restyled by a viewer who needs larger text, and they are baked into every version of the file.

Social platforms where video autoplays muted are the real case for burning in, and many accept a sidecar as well. The defensible approach for those is both: burned-in for the feed, sidecar for accessibility.

# transcribe locally
whisper-ctranslate2 --model medium --language en --vad_filter True talk.mp4

# attach as a soft track, no re-encode of the video
ffmpeg -i talk.mp4 -i talk.srt -c copy -c:s mov_text out.mp4

# burn in, with styling
ffmpeg -i talk.mp4 -vf "subtitles=talk.srt:force_style='FontSize=22,Outline=2'" out.mp4

The free path

• Whisper and the faster faster-whisper / whisper.cpp implementations run locally on modest hardware, are free, and handle many languages.
• Subtitle Edit is free and open source, and is a good caption editor with waveform view, timing tools and format conversion.
• Aegisub for styled subtitles.
• ffmpeg for muxing and burning in.
• DaVinci Resolve’s free edition includes transcription and a subtitle track.

The limitation

Captions are a legal requirement in many contexts, and the requirements differ by country, by sector and by whether the content is broadcast, educational, governmental or commercial. Several jurisdictions have specific standards and enforcement histories. This is not something to resolve from a general course: check the rules that apply to your jurisdiction and your kind of publication, and take advice if the answer matters.

The technical limitation is simpler and is the one to hold on to: no automatic system is accurate enough to publish unread. Reading a ten-minute transcript takes about four minutes and catches the names, the numbers and the sentences nobody said.