The other half: turning speech into text

Why this belongs in a course about making things

Every workflow in this module runs both directions. Subtitles, dubbing, transcripts, searchable video, editing by editing the text — all of it depends on recognition, and it fails in ways that are easy to miss precisely because the output reads well.

How it works and what that predicts

Current systems are sequence models. Audio is encoded into features; a decoder generates text tokens conditioned on them, one at a time, exactly like a language model.

That last part is the important one. The decoder is a language model. It produces text that is fluent whether or not it is accurate, because fluency is what the objective rewarded.

The characteristic failures follow:

• Hallucinated text during silence or noise. A recording with a long pause can produce an invented sentence, frequently something that appeared in the training data — a subtitle credit, a stock phrase, a repeated line. Studies of medical and general transcription have documented invented content, including invented clinical statements, at rates that are small per segment and significant across a corpus.
• Plausible substitutions. An unfamiliar name or term is replaced with a common word that fits the sentence. The transcript reads perfectly and says something else.
• Repetition loops, where a phrase repeats for many seconds.
• Silent language switching on multilingual audio, translating rather than transcribing.

Compare this with the older approach, which produced obvious rubbish when it failed. Obvious rubbish is safer. A confident wrong transcript is trusted.

The numbers, honestly

Word error rate on clean, read English is in the low single digits for the best models, which is genuinely at or near human parity for that material. On the material people actually have — overlapping speakers, accents, background noise, telephone bandwidth, domain vocabulary — it is commonly 10 to 25%, and worse for under-served accents.

Do not plan on the headline number. Test on twenty minutes of your own real audio and count the errors yourself. It takes an hour and it is the only figure that means anything for your work.

Working with it properly

Improve the audio first. This is the highest-return action by a wide margin. A close microphone, a quiet room and separate tracks per speaker will do more than any change of model. Free noise reduction in Audacity is worth a pass; aggressive processing is not, because it removes the detail the model needs.

Supply context where the system allows it. Many systems accept an initial prompt or a vocabulary list. Feeding in the names, jargon and product terms that appear in the recording measurably reduces substitutions.

Segment on silence rather than on fixed lengths. Cutting mid-word produces errors at every boundary.

Never ship an unreviewed transcript of anything consequential. Legal, medical, journalistic, or anything a person's account of themselves depends on. The failure mode is fluent invention, and only a human comparing against the audio catches it.

Keep the audio. The transcript is a derived artefact and it is wrong sometimes. The recording is the record.

The free path

Whisper and its many optimised community implementations are free, run offline, and are strong across a wide range of languages. Local processing also means confidential audio never leaves the machine, which matters for legal, medical and journalistic work and is frequently a contractual requirement.

Free editors — Kdenlive, Shotcut, DaVinci Resolve's free edition — have subtitle workflows that import a transcript and let you correct it against the picture, which is by far the fastest way to review one.

There is one setting worth knowing about because it changes the failure profile. Many recognition systems will accept a prompt or an instruction alongside the audio, and some accept a request to transcribe verbatim including hesitations and repetitions. Turning that on makes the output less pleasant to read and considerably more faithful, because the model is no longer tidying speech into prose. For interviews, evidence and anything where what was said matters more than how it reads, verbatim is the right mode, and the tidy default is quietly editorialising.

The limitation to hold onto: recognition quality is not a single number, it varies enormously with the speaker and the conditions, and the errors it makes are the fluent kind. The discipline that follows is simple and rarely observed — measure it on your own material, and read the transcript against the audio whenever it matters.