Fine-tuning a speech model

Try the free things first

Before training anything, two techniques cost nothing and often solve the problem.

Initial prompt or hotwords. Whisper accepts a text prompt that conditions decoding. Supplying your product names, drug names or technical vocabulary in it substantially improves their transcription with no training at all:

result = model.transcribe(audio, initial_prompt=
    "Addaly, Hyperdrive, Turnstile, Durable Object, bf16, QLoRA.")

A bigger model. The quality difference between a small and a large Whisper checkpoint on accented speech is large, and switching is a one-line change.

If the residual errors after both are still unacceptable, and for a specific accent, a noisy environment or a dense domain vocabulary they often are, then fine-tune.

What the data must look like

Audio at 16 kHz, mono, paired with an exact transcript. That is the whole requirement, and the two parts that go wrong are the sample rate and the exactness.

ds = ds.cast_column("audio", Audio(sampling_rate=16_000))

Whisper’s feature extractor expects 16 kHz. Feed it 44.1 kHz audio without resampling and the model receives a signal stretched in time, which trains badly and is not flagged.

The transcripts must be verbatim, consistently. Decide once whether you write numbers as digits or words, whether you include filler sounds, whether you punctuate, and then apply it everywhere. A model trained on transcripts that are inconsistently punctuated learns to punctuate inconsistently, exactly as with text.

How much audio

Less than people expect for a targeted improvement:

• 1 to 5 hours of labelled domain audio measurably reduces word error rate on that domain’s vocabulary and speakers.
• 10 to 50 hours for a substantial accent or dialect adaptation.
• Hundreds of hours for a low-resource language from a weak starting point.

Common Voice is a large, free, multilingual, CC0 dataset with per-clip accent and demographic metadata, which makes it the standard starting point for accent work, and a good way to test your pipeline before you spend money recording.

The configuration

Whisper is an encoder-decoder, so it trains with a sequence-to-sequence trainer rather than a causal one. Two specifics matter:

• Freeze the encoder for small datasets. The audio encoder is the part that knows what speech sounds like; the decoder is where domain vocabulary lives. Freezing the encoder cuts memory and reduces forgetting.
• LoRA works well here, on the decoder’s attention projections, and keeps the artefact small.

A learning rate around 1e-5, a few thousand steps, and the usual checkpoint discipline.

Word error rate, and the normalisation question

WER is edit distance between the reference and the hypothesis, divided by the number of reference words. A WER of 0.12 means roughly twelve errors per hundred words.

The subtlety that makes published numbers hard to compare: Whisper’s evaluation conventionally applies a text normaliser first: lowercasing, removing punctuation, expanding numbers and contractions. Normalised WER can be several points lower than raw WER on identical output.

So report both, and state which you are quoting:

import evaluate
wer = evaluate.load("wer")
print("raw       ", wer.compute(predictions=preds, references=refs))
print("normalised", wer.compute(predictions=[norm(p) for p in preds],
                                references=[norm(r) for r in refs]))

Which one matters depends on your product. If the transcript is read by a person, punctuation and casing are part of the quality and raw WER is the honest number. If it feeds a search index, normalised is fine.

Measure the thing you care about

WER weights every word equally, and your errors are not equally important. A transcript that gets every common word right and the drug name wrong has a low WER and is dangerous.

So add a targeted measure: recall on a list of terms that matter: product names, medications, place names, numbers. Ten lines of code, and it is frequently the metric that changes the decision.

Serving

After training, faster-whisper (a CTranslate2 reimplementation) runs the same weights several times faster with lower memory, including on CPU, and whisper.cpp runs them on a laptop or a phone. Both are free. As with every other artefact in this course: evaluate the converted model, not the one you trained, because quantisation and reimplementation both change the output.