Machines that write down what you said

Where you already meet it

Voice notes turned into text, automatic subtitles, dictation, call-centre transcripts, meeting summaries, the voice assistant in a car. All of it is automatic speech recognition, and it improved sharply around 2022 for the same reason everything else did: bigger models trained on far more audio.

The measure used is word error rate: the proportion of words inserted, deleted or substituted compared with a careful human transcript. On clean, read speech in standard American English, the best systems are now under 5%, which is roughly what a professional transcriptionist achieves. On a noisy market street with two people talking over each other, the same system may be over 40%.

That gap between the published number and the real room is the whole lesson.

Accent is not a detail

Speech systems learn from the voices in their training audio, and those voices are not evenly distributed.

The clearest documented example is a 2020 Stanford study published in PNAS that tested five commercial systems from major providers on identical read passages. Average word error rate was about 0.19 for white speakers and about 0.35 for Black speakers: close to double. The systems had not been designed to do this. They had been trained on audio in which one group was far better represented.

Similar gaps appear for regional accents, for second-language speakers, for older voices, and for speech affected by illness or disability. If you have ever felt that a voice assistant works for other people and not for you, this is likely why, and it is not your pronunciation.

The failure mode that catches people out

There is a specific and important way modern speech models fail, and it is not what you expect.

OpenAI’s Whisper (the most widely used open speech model, free to download and run) has a documented tendency to invent text during silence or noise. Given a pause, it sometimes emits a plausible sentence that nobody said. Researchers examining medical and public-meeting transcripts in 2024 found fabricated sentences in a meaningful fraction of segments, including invented clinical detail.

The mechanism is worth understanding, because it recurs throughout this course. Whisper was trained to produce the text that goes with audio. Given audio containing nothing, it still produces its best guess at accompanying text, because producing nothing was rarely the right answer in training. The model has no way to say “there was no speech here”.

So a transcript is not a recording. If the words matter (a consultation, a legal statement, a disciplinary meeting) keep the audio and spot-check the transcript against it, especially around pauses.

The parts of a transcript nobody recorded

Two features of a finished transcript were never in the audio at all, and both are predictions.

Punctuation and capitalisation. Nobody speaks full stops. The model infers sentence boundaries from rhythm and wording, and where it puts them changes the meaning. A misplaced boundary can join a question to the answer that followed it.

Speaker labels. Deciding who spoke each stretch is a separate task called diarisation, run by a separate model that clusters voices. It is markedly less reliable than the words themselves, and it degrades when people interrupt each other, when two speakers sound alike, or when someone is on a phone line. A transcript that attributes a sentence to the wrong person is a serious error in a meeting record, a disciplinary hearing or an interview, and it is the error people check for least.

So when you read “Speaker 2: I agreed to that”, two independent systems had to be right. Treat attribution as the weakest claim on the page.

The free path, and it is a good one

This is an area where the free tools are competitive with paid services.

• Whisper, released with open weights, runs on an ordinary laptop. whisper.cpp is an efficient implementation that runs even on a phone, offline, with no account and no upload.
• faster-whisper speeds this up considerably on modest hardware.
• Vosk offers small offline models for many languages, useful where storage and bandwidth are tight.
• For subtitles, Subtitle Edit (free, Windows) and Aegisub handle timing and correction.

Running locally is not only about cost. A transcript produced offline never leaves the device, which for a medical, legal or personal recording is a stronger privacy guarantee than any policy document.

Practical technique

The controllable variables are worth more than the model choice. Get the microphone close to the speaker: halving the distance does more for accuracy than any setting. Reduce background noise rather than trying to filter it afterwards. Tell the system the language explicitly instead of letting it detect; misdetection produces spectacular nonsense. And where names, drugs or technical terms recur, most tools accept a hint list, which fixes the errors that matter most.