{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"honest-answer-first","title":"The Honest Answer First","url":"https://addaly.com/learn/fine-tuning/honest-answer-first","position":1,"of":84,"minutes":7,"answer":"A prompt changes in ten seconds; a fine-tune changes in a day. Exhaust the fast loop first.","keyIdea":"A prompt changes in ten seconds; a fine-tune changes in a day. Exhaust the fast loop first."},"body":{"format":"markdown","text":"## Start with the honest answer\n\nMost projects that reach for fine-tuning are solved by a better prompt or by retrieval. Not some. Most. This is the first lesson because if the course changes your mind here, it has done its job and you can stop reading.\n\nFine-tuning has a slow loop. A prompt change takes ten seconds and you see the result immediately. A fine-tune takes a dataset, a training run, and an evaluation (a day if you are practised, a week if you are not) and when the result is worse, you often cannot tell which of the twenty things you changed did it. Anything you can learn from a prompt, learn from a prompt.\n\n## The ladder\n\nWork down this list. Stop as soon as the problem is solved.\n\n1. **Write the instruction properly.** State the output shape. State the constraints. Most \"the model won't follow my format\" complaints are a prompt that never specified the format precisely.\n2. **Put three to five real examples in the prompt.** Few-shot moves format and tone further than people expect, and you can change it at 3am without a GPU.\n3. **Give it the facts.** If the answer depends on your documents, policies, or prices, put the relevant text in the context window. This is the answer to \"it doesn't know our stuff\" nearly every time.\n4. **Give it tools.** If the answer depends on a calculation or a live lookup, let it call something.\n5. **Try a stronger base model** with everything above, and compare cost honestly.\n6. **Now consider fine-tuning.**\n\n## The twenty-example test\n\nBefore you build anything, do this. Take twenty real inputs your system got wrong: real ones, from logs, not ones you invented. Hand-write the output you wanted for each. Put five into the prompt as examples and run the other fifteen.\n\n- If it now gets most of the fifteen right, you do not need to fine-tune. You need that prompt.\n- If it fails because it does not know a fact, you need retrieval.\n- If it fails because it cannot do the reasoning at all, fine-tuning is unlikely to rescue it either.\n- If it gets them right only when the examples are in the prompt, and your prompt has no room for them at your volume or your latency budget, that is a real fine-tuning case.\n\nThose twenty hand-written outputs are also the start of your dataset, so the test is not wasted work either way.\n\n## When fine-tuning is right\n\n- **A behaviour you cannot prompt reliably at scale.** A strict output shape, a house register, a domain convention that takes 2,000 tokens of instructions to describe. Fine-tuning moves that into the weights and your prompt gets short.\n- **Cost and latency.** A tuned 7B that matches a large model on your one narrow task can be ten to fifty times cheaper per token, and faster. At a million requests a month that is the whole business case.\n- **Privacy or air-gap.** The model has to run on your hardware, so the large hosted one is not an option at any quality.\n- **A narrow task with plenty of labelled data.** Extraction, classification, routing. Here a small tuned model beats a large prompted one on accuracy and cost at the same time.\n\n## When it is the wrong answer\n\n- Adding facts, especially facts that change. Use retrieval.\n- Fixing hallucination. Training on answers the model does not know makes it more confidently wrong, which the next lesson explains.\n- Teaching reasoning the base cannot do.\n- Anything where you have forty examples, no evaluation set, and a deadline.\n\nThe rest of this course assumes you walked the ladder and still have a real case. That is a respectable place to be. It is just a much smaller group than the people currently opening a fine-tuning notebook.\n\n> Figure: The ladder, and where most projects should stop. Steps: 1. Write the instruction properly (State the output shape and the constraints. Most format complaints are a prompt that never specified the format.); 2. Put three to five real examples in the prompt (Few-shot moves format and tone further than people expect, and you can change it at three in the morning.); 3. Give it the facts (Put the document, policy or price in the context window. This answers \"it does not know our stuff\" nearly every time.); 4. Give it tools (If the answer needs a calculation or a live lookup, let it call something.); 5. Try a stronger base model (With everything above in place, and compare the cost honestly.); 6. Now consider fine-tuning (A slow loop: a dataset, a run, an evaluation, and twenty changes you cannot tell apart afterwards.). Work down it and stop at the first rung that solves the problem. A prompt change takes ten seconds; a fine-tune takes a day if you are practised."},"check":{"question":"A support bot keeps stating your old refund policy, which changed three weeks ago. Your logs show it does this confidently and often. What do you do first?","options":[{"text":"Fine-tune on 2,000 past support transcripts so the model learns how your team handles refunds.","correct":false,"tempting":"Those transcripts do teach house behaviour, but they encode the old policy, so you would train the wrong answer in more firmly."},{"text":"Put the current policy text into the context at answer time, then re-check the same failing cases.","correct":true},{"text":"Fine-tune on 200 examples that state the new policy explicitly.","correct":false,"tempting":"It targets the exact gap, but you repeat the whole exercise at the next policy change, and the model still cannot cite a source."},{"text":"Move to a larger base model, which hallucinates less.","correct":false,"tempting":"Larger models are better calibrated, but no model of any size knows a policy your company wrote three weeks ago."}]},"prev":null,"next":{"slug":"what-it-changes","title":"What Fine-Tuning Changes, and What It Cannot Add","url":"https://addaly.com/learn/fine-tuning/what-it-changes"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The Honest Answer First', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/honest-answer-first (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The Honest Answer First","courseTitle":"Fine-Tuning, and When Not To","module":{"index":1,"slug":"deciding-whether-to-fine-tune","title":"Deciding whether to fine-tune at all"},"answer":null,"keyIdea":"A prompt changes in ten seconds; a fine-tune changes in a day. Exhaust the fast loop first.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 1.1","figure":{"type":"flow","steps":[{"label":"Write the instruction properly","detail":"State the output shape and the constraints. Most format complaints are a prompt that never specified the format."},{"label":"Put three to five real examples in the prompt","detail":"Few-shot moves format and tone further than people expect, and you can change it at three in the morning."},{"label":"Give it the facts","detail":"Put the document, policy or price in the context window. This answers “it does not know our stuff” nearly every time."},{"label":"Give it tools","detail":"If the answer needs a calculation or a live lookup, let it call something."},{"label":"Try a stronger base model","detail":"With everything above in place, and compare the cost honestly."},{"label":"Now consider fine-tuning","detail":"A slow loop: a dataset, a run, an evaluation, and twenty changes you cannot tell apart afterwards."}],"title":"The ladder, and where most projects should stop","caption":"Work down it and stop at the first rung that solves the problem. A prompt change takes ten seconds; a fine-tune takes a day if you are practised."},"alt":"Figure: The ladder, and where most projects should stop. Steps: 1. Write the instruction properly (State the output shape and the constraints. Most format complaints are a prompt that never specified the format.); 2. Put three to five real examples in the prompt (Few-shot moves format and tone further than people expect, and you can change it at three in the morning.); 3. Give it the facts (Put the document, policy or price in the context window. This answers “it does not know our stuff” nearly every time.); 4. Give it tools (If the answer needs a calculation or a live lookup, let it call something.); 5. Try a stronger base model (With everything above in place, and compare the cost honestly.); 6. Now consider fine-tuning (A slow loop: a dataset, a run, an evaluation, and twenty changes you cannot tell apart afterwards.). Work down it and stop at the first rung that solves the problem. A prompt change takes ten seconds; a fine-tune takes a day if you are practised."}},"blocks":[{"t":"h2","id":"start-with-the-honest-answer","runs":[{"text":"Start with the honest answer"}]},{"t":"p","runs":[{"text":"Most projects that reach for fine-tuning are solved by a better prompt or by retrieval. Not some. Most. This is the first lesson because if the course changes your mind here, it has done its job and you can stop reading."}]},{"t":"p","runs":[{"text":"Fine-tuning has a slow loop. A prompt change takes ten seconds and you see the result immediately. A fine-tune takes a dataset, a training run, and an evaluation (a day if you are practised, a week if you are not) and when the result is worse, you often cannot tell which of the twenty things you changed did it. Anything you can learn from a prompt, learn from a prompt."}]},{"t":"h2","id":"the-ladder","runs":[{"text":"The ladder"}]},{"t":"p","runs":[{"text":"Work down this list. Stop as soon as the problem is solved."}]},{"t":"ol","start":1,"items":[[{"text":"Write the instruction properly.","b":true},{"text":" State the output shape. State the constraints. Most “the model won’t follow my format” complaints are a prompt that never specified the format precisely."}],[{"text":"Put three to five real examples in the prompt.","b":true},{"text":" Few-shot moves format and tone further than people expect, and you can change it at 3am without a GPU."}],[{"text":"Give it the facts.","b":true},{"text":" If the answer depends on your documents, policies, or prices, put the relevant text in the context window. This is the answer to “it doesn’t know our stuff” nearly every time."}],[{"text":"Give it tools.","b":true},{"text":" If the answer depends on a calculation or a live lookup, let it call something."}],[{"text":"Try a stronger base model","b":true},{"text":" with everything above, and compare cost honestly."}],[{"text":"Now consider fine-tuning.","b":true}]]},{"t":"h2","id":"the-twenty-example-test","runs":[{"text":"The twenty-example test"}]},{"t":"p","runs":[{"text":"Before you build anything, do this. Take twenty real inputs your system got wrong: real ones, from logs, not ones you invented. Hand-write the output you wanted for each. Put five into the prompt as examples and run the other fifteen."}]},{"t":"ul","items":[[{"text":"If it now gets most of the fifteen right, you do not need to fine-tune. You need that prompt."}],[{"text":"If it fails because it does not know a fact, you need retrieval."}],[{"text":"If it fails because it cannot do the reasoning at all, fine-tuning is unlikely to rescue it either."}],[{"text":"If it gets them right only when the examples are in the prompt, and your prompt has no room for them at your volume or your latency budget, that is a real fine-tuning case."}]]},{"t":"p","runs":[{"text":"Those twenty hand-written outputs are also the start of your dataset, so the test is not wasted work either way."}]},{"t":"h2","id":"when-fine-tuning-is-right","runs":[{"text":"When fine-tuning is right"}]},{"t":"ul","items":[[{"text":"A behaviour you cannot prompt reliably at scale.","b":true},{"text":" A strict output shape, a house register, a domain convention that takes 2,000 tokens of instructions to describe. Fine-tuning moves that into the weights and your prompt gets short."}],[{"text":"Cost and latency.","b":true},{"text":" A tuned 7B that matches a large model on your one narrow task can be ten to fifty times cheaper per token, and faster. At a million requests a month that is the whole business case."}],[{"text":"Privacy or air-gap.","b":true},{"text":" The model has to run on your hardware, so the large hosted one is not an option at any quality."}],[{"text":"A narrow task with plenty of labelled data.","b":true},{"text":" Extraction, classification, routing. Here a small tuned model beats a large prompted one on accuracy and cost at the same time."}]]},{"t":"h2","id":"when-it-is-the-wrong-answer","runs":[{"text":"When it is the wrong answer"}]},{"t":"ul","items":[[{"text":"Adding facts, especially facts that change. Use retrieval."}],[{"text":"Fixing hallucination. Training on answers the model does not know makes it more confidently wrong, which the next lesson explains."}],[{"text":"Teaching reasoning the base cannot do."}],[{"text":"Anything where you have forty examples, no evaluation set, and a deadline."}]]},{"t":"p","runs":[{"text":"The rest of this course assumes you walked the ladder and still have a real case. That is a respectable place to be. It is just a much smaller group than the people currently opening a fine-tuning notebook."}]}],"check":{"question":[{"text":"A support bot keeps stating your old refund policy, which changed three weeks ago. Your logs show it does this confidently and often. What do you do first?"}],"options":[{"text":[{"text":"Fine-tune on 2,000 past support transcripts so the model learns how your team handles refunds."}],"correct":false,"tempting":[{"text":"Those transcripts do teach house behaviour, but they encode the old policy, so you would train the wrong answer in more firmly."}]},{"text":[{"text":"Put the current policy text into the context at answer time, then re-check the same failing cases."}],"correct":true,"tempting":null},{"text":[{"text":"Fine-tune on 200 examples that state the new policy explicitly."}],"correct":false,"tempting":[{"text":"It targets the exact gap, but you repeat the whole exercise at the next policy change, and the model still cannot cite a source."}]},{"text":[{"text":"Move to a larger base model, which hallucinates less."}],"correct":false,"tempting":[{"text":"Larger models are better calibrated, but no model of any size knows a policy your company wrote three weeks ago."}]}],"order":[3,1,2,0],"why":null},"prev":null,"next":{"slug":"what-it-changes","title":"What Fine-Tuning Changes, and What It Cannot Add","minutes":8}}}