{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"speech-models-and-accents","title":"Fine-tuning a speech model","url":"https://addaly.com/learn/fine-tuning/speech-models-and-accents","position":61,"of":84,"minutes":9,"answer":"Try an initial prompt with your domain vocabulary and a larger checkpoint before training, resample to 16 kHz mono, freeze the audio encoder on small datasets.","keyIdea":"Try an initial prompt with your domain vocabulary and a larger checkpoint before training, resample to 16 kHz mono, freeze the audio encoder on small datasets, and report raw as well as normalised word error rate alongside recall on the terms that actually matter."},"body":{"format":"markdown","text":"## Try the free things first\n\nBefore training anything, two techniques cost nothing and often solve the problem.\n\n**Initial prompt or hotwords.** Whisper accepts a text prompt that conditions decoding. Supplying your product names, drug names or technical vocabulary in it substantially improves their transcription with no training at all:\n\n```python\nresult = model.transcribe(audio, initial_prompt=\n    \"Addaly, Hyperdrive, Turnstile, Durable Object, bf16, QLoRA.\")\n```\n\n**A bigger model.** The quality difference between a small and a large Whisper checkpoint on accented speech is large, and switching is a one-line change.\n\nIf the residual errors after both are still unacceptable, and for a specific accent, a noisy environment or a dense domain vocabulary they often are, then fine-tune.\n\n## What the data must look like\n\nAudio at 16 kHz, mono, paired with an exact transcript. That is the whole requirement, and the two parts that go wrong are the sample rate and the exactness.\n\n```python\nds = ds.cast_column(\"audio\", Audio(sampling_rate=16_000))\n```\n\nWhisper's feature extractor expects 16 kHz. Feed it 44.1 kHz audio without resampling and the model receives a signal stretched in time, which trains badly and is not flagged.\n\nThe transcripts must be verbatim, consistently. Decide once whether you write numbers as digits or words, whether you include filler sounds, whether you punctuate, and then apply it everywhere. A model trained on transcripts that are inconsistently punctuated learns to punctuate inconsistently, exactly as with text.\n\n## How much audio\n\nLess than people expect for a targeted improvement:\n\n- **1 to 5 hours** of labelled domain audio measurably reduces word error rate on that domain's vocabulary and speakers.\n- **10 to 50 hours** for a substantial accent or dialect adaptation.\n- **Hundreds of hours** for a low-resource language from a weak starting point.\n\nCommon Voice is a large, free, multilingual, CC0 dataset with per-clip accent and demographic metadata, which makes it the standard starting point for accent work, and a good way to test your pipeline before you spend money recording.\n\n## The configuration\n\nWhisper is an encoder-decoder, so it trains with a sequence-to-sequence trainer rather than a causal one. Two specifics matter:\n\n- **Freeze the encoder** for small datasets. The audio encoder is the part that knows what speech sounds like; the decoder is where domain vocabulary lives. Freezing the encoder cuts memory and reduces forgetting.\n- **LoRA works well here**, on the decoder's attention projections, and keeps the artefact small.\n\nA learning rate around 1e-5, a few thousand steps, and the usual checkpoint discipline.\n\n## Word error rate, and the normalisation question\n\nWER is edit distance between the reference and the hypothesis, divided by the number of reference words. A WER of 0.12 means roughly twelve errors per hundred words.\n\nThe subtlety that makes published numbers hard to compare: Whisper's evaluation conventionally applies a text normaliser first: lowercasing, removing punctuation, expanding numbers and contractions. Normalised WER can be several points lower than raw WER on identical output.\n\nSo report both, and state which you are quoting:\n\n```python\nimport evaluate\nwer = evaluate.load(\"wer\")\nprint(\"raw       \", wer.compute(predictions=preds, references=refs))\nprint(\"normalised\", wer.compute(predictions=[norm(p) for p in preds],\n                                references=[norm(r) for r in refs]))\n```\n\nWhich one matters depends on your product. If the transcript is read by a person, punctuation and casing are part of the quality and raw WER is the honest number. If it feeds a search index, normalised is fine.\n\n## Measure the thing you care about\n\nWER weights every word equally, and your errors are not equally important. A transcript that gets every common word right and the drug name wrong has a low WER and is dangerous.\n\nSo add a targeted measure: recall on a list of terms that matter: product names, medications, place names, numbers. Ten lines of code, and it is frequently the metric that changes the decision.\n\n## Serving\n\nAfter training, `faster-whisper` (a CTranslate2 reimplementation) runs the same weights several times faster with lower memory, including on CPU, and `whisper.cpp` runs them on a laptop or a phone. Both are free. As with every other artefact in this course: evaluate the converted model, not the one you trained, because quantisation and reimplementation both change the output."},"check":{"question":"A team fine-tunes Whisper on 8 hours of call audio and reports WER falling from 0.19 to 0.09. A colleague points out the two figures are not comparable. What is the likeliest reason?","options":[{"text":"The baseline used a smaller checkpoint than the fine-tuned model, so model size rather than training explains the gap.","correct":false,"tempting":"Names a genuine confound, but one the team would have controlled, and it does not make the two numbers incomparable in kind."},{"text":"WER cannot be compared across models with different tokenizers, since edit distance depends on the tokenisation.","correct":false,"tempting":"Confuses WER with token-level loss; word error rate is computed on decoded words and is tokenizer-independent."},{"text":"One figure had Whisper's text normaliser applied and the other did not.","correct":true},{"text":"Fine-tuning reduces WER on the training distribution only, so the improvement is not meaningful on other audio.","correct":false,"tempting":"Raises a generalisation concern that would need a held-out set to settle, but does not make the two numbers different measurements."}]},"prev":{"slug":"fine-tuning-a-classifier","title":"Fine-tuning a classifier properly","url":"https://addaly.com/learn/fine-tuning/fine-tuning-a-classifier"},"next":{"slug":"vision-language-models","title":"Fine-tuning a vision-language model","url":"https://addaly.com/learn/fine-tuning/vision-language-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Fine-tuning a speech model', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/speech-models-and-accents (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Fine-tuning a speech model","courseTitle":"Fine-Tuning, and When Not To","module":{"index":7,"slug":"beyond-text-generation","title":"Fine-tuning beyond a chat model"},"answer":null,"keyIdea":"Try an initial prompt with your domain vocabulary and a larger checkpoint before training, resample to 16 kHz mono, freeze the audio encoder on small datasets, and report raw as well as normalised word error rate alongside recall on the terms that actually matter.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"try-the-free-things-first","runs":[{"text":"Try the free things first"}]},{"t":"p","runs":[{"text":"Before training anything, two techniques cost nothing and often solve the problem."}]},{"t":"p","runs":[{"text":"Initial prompt or hotwords.","b":true},{"text":" Whisper accepts a text prompt that conditions decoding. Supplying your product names, drug names or technical vocabulary in it substantially improves their transcription with no training at all:"}]},{"t":"code","lang":"python","label":"Python","text":"result = model.transcribe(audio, initial_prompt=\n    \"Addaly, Hyperdrive, Turnstile, Durable Object, bf16, QLoRA.\")"},{"t":"p","runs":[{"text":"A bigger model.","b":true},{"text":" The quality difference between a small and a large Whisper checkpoint on accented speech is large, and switching is a one-line change."}]},{"t":"p","runs":[{"text":"If the residual errors after both are still unacceptable, and for a specific accent, a noisy environment or a dense domain vocabulary they often are, then fine-tune."}]},{"t":"h2","id":"what-the-data-must-look-like","runs":[{"text":"What the data must look like"}]},{"t":"p","runs":[{"text":"Audio at 16 kHz, mono, paired with an exact transcript. That is the whole requirement, and the two parts that go wrong are the sample rate and the exactness."}]},{"t":"code","lang":"python","label":"Python","text":"ds = ds.cast_column(\"audio\", Audio(sampling_rate=16_000))"},{"t":"p","runs":[{"text":"Whisper’s feature extractor expects 16 kHz. Feed it 44.1 kHz audio without resampling and the model receives a signal stretched in time, which trains badly and is not flagged."}]},{"t":"p","runs":[{"text":"The transcripts must be verbatim, consistently. Decide once whether you write numbers as digits or words, whether you include filler sounds, whether you punctuate, and then apply it everywhere. A model trained on transcripts that are inconsistently punctuated learns to punctuate inconsistently, exactly as with text."}]},{"t":"h2","id":"how-much-audio","runs":[{"text":"How much audio"}]},{"t":"p","runs":[{"text":"Less than people expect for a targeted improvement:"}]},{"t":"ul","items":[[{"text":"1 to 5 hours","b":true},{"text":" of labelled domain audio measurably reduces word error rate on that domain’s vocabulary and speakers."}],[{"text":"10 to 50 hours","b":true},{"text":" for a substantial accent or dialect adaptation."}],[{"text":"Hundreds of hours","b":true},{"text":" for a low-resource language from a weak starting point."}]]},{"t":"p","runs":[{"text":"Common Voice is a large, free, multilingual, CC0 dataset with per-clip accent and demographic metadata, which makes it the standard starting point for accent work, and a good way to test your pipeline before you spend money recording."}]},{"t":"h2","id":"the-configuration","runs":[{"text":"The configuration"}]},{"t":"p","runs":[{"text":"Whisper is an encoder-decoder, so it trains with a sequence-to-sequence trainer rather than a causal one. Two specifics matter:"}]},{"t":"ul","items":[[{"text":"Freeze the encoder","b":true},{"text":" for small datasets. The audio encoder is the part that knows what speech sounds like; the decoder is where domain vocabulary lives. Freezing the encoder cuts memory and reduces forgetting."}],[{"text":"LoRA works well here","b":true},{"text":", on the decoder’s attention projections, and keeps the artefact small."}]]},{"t":"p","runs":[{"text":"A learning rate around 1e-5, a few thousand steps, and the usual checkpoint discipline."}]},{"t":"h2","id":"word-error-rate-and-the-normalisation-question","runs":[{"text":"Word error rate, and the normalisation question"}]},{"t":"p","runs":[{"text":"WER is edit distance between the reference and the hypothesis, divided by the number of reference words. A WER of 0.12 means roughly twelve errors per hundred words."}]},{"t":"p","runs":[{"text":"The subtlety that makes published numbers hard to compare: Whisper’s evaluation conventionally applies a text normaliser first: lowercasing, removing punctuation, expanding numbers and contractions. Normalised WER can be several points lower than raw WER on identical output."}]},{"t":"p","runs":[{"text":"So report both, and state which you are quoting:"}]},{"t":"code","lang":"python","label":"Python","text":"import evaluate\nwer = evaluate.load(\"wer\")\nprint(\"raw       \", wer.compute(predictions=preds, references=refs))\nprint(\"normalised\", wer.compute(predictions=[norm(p) for p in preds],\n                                references=[norm(r) for r in refs]))"},{"t":"p","runs":[{"text":"Which one matters depends on your product. If the transcript is read by a person, punctuation and casing are part of the quality and raw WER is the honest number. If it feeds a search index, normalised is fine."}]},{"t":"h2","id":"measure-the-thing-you-care-about","runs":[{"text":"Measure the thing you care about"}]},{"t":"p","runs":[{"text":"WER weights every word equally, and your errors are not equally important. A transcript that gets every common word right and the drug name wrong has a low WER and is dangerous."}]},{"t":"p","runs":[{"text":"So add a targeted measure: recall on a list of terms that matter: product names, medications, place names, numbers. Ten lines of code, and it is frequently the metric that changes the decision."}]},{"t":"h2","id":"serving","runs":[{"text":"Serving"}]},{"t":"p","runs":[{"text":"After training, "},{"text":"faster-whisper","code":true},{"text":" (a CTranslate2 reimplementation) runs the same weights several times faster with lower memory, including on CPU, and "},{"text":"whisper.cpp","code":true},{"text":" runs them on a laptop or a phone. Both are free. As with every other artefact in this course: evaluate the converted model, not the one you trained, because quantisation and reimplementation both change the output."}]}],"check":{"question":[{"text":"A team fine-tunes Whisper on 8 hours of call audio and reports WER falling from 0.19 to 0.09. A colleague points out the two figures are not comparable. What is the likeliest reason?"}],"options":[{"text":[{"text":"The baseline used a smaller checkpoint than the fine-tuned model, so model size rather than training explains the gap."}],"correct":false,"tempting":[{"text":"Names a genuine confound, but one the team would have controlled, and it does not make the two numbers incomparable in kind."}]},{"text":[{"text":"WER cannot be compared across models with different tokenizers, since edit distance depends on the tokenisation."}],"correct":false,"tempting":[{"text":"Confuses WER with token-level loss; word error rate is computed on decoded words and is tokenizer-independent."}]},{"text":[{"text":"One figure had Whisper’s text normaliser applied and the other did not."}],"correct":true,"tempting":null},{"text":[{"text":"Fine-tuning reduces WER on the training distribution only, so the improvement is not meaningful on other audio."}],"correct":false,"tempting":[{"text":"Raises a generalisation concern that would need a held-out set to settle, but does not make the two numbers different measurements."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"fine-tuning-a-classifier","title":"Fine-tuning a classifier properly","minutes":9},"next":{"slug":"vision-language-models","title":"Fine-tuning a vision-language model","minutes":9}}}