{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"when-not-to-fine-tune","title":"When not to fine-tune","url":"https://addaly.com/learn/open-models/when-not-to-fine-tune","position":51,"of":71,"minutes":8,"answer":"Fine-tuning reliably changes form and unreliably changes knowledge, so the honest test is whether it beats your best prompt plus retrieval by a margin that justifies re-tuning every time the base model is superseded.","keyIdea":"Fine-tuning reliably changes form and unreliably changes knowledge, so the honest test is whether it beats your best prompt plus retrieval by a margin that justifies re-tuning every time the base model is superseded."},"body":{"format":"markdown","text":"## What fine-tuning changes, and what it does not\n\nFine-tuning adjusts weights on examples of the behaviour you want. It is very good at some things and unreliable at others, and the difference decides whether you should do it.\n\n**It reliably changes form.** Output format, tone, register, verbosity, structure, the conventions of your domain, which of several plausible responses is preferred, and whether the model does the task at all rather than explaining it. This is what fine-tuning is for.\n\n**It changes knowledge poorly.** You can teach facts by fine-tuning, and it works badly compared with putting the facts in the context. The model learns the surface of the statement, generalises it unpredictably, contradicts it when phrased differently, and forgets other things in the process. Retrieval is better on every axis: cheaper, updatable, citable, and it does not degrade anything else.\n\n**It does not add reasoning capacity.** A 3B model tuned on your data is still a 3B model. Tuning can make it apply what it has more reliably to your task; it cannot make it think further ahead.\n\n## The ladder, in order of cost\n\nWork down this list and stop when the result is good enough.\n\n1. **A better prompt.** An hour, and it fixes more than people expect.\n2. **Three to eight examples in the prompt.** Another hour.\n3. **Constrained output.** Removes format failures entirely.\n4. **Retrieval.** Fixes knowledge problems, which is most of what looks like model weakness.\n5. **A different or larger model.** A day of evaluation.\n6. **Fine-tuning.** Days of data work, then permanent maintenance.\n\nMost teams that fine-tune skipped steps 3 and 4. That is the single most common waste in this part of the field.\n\n## The signals that fine-tuning is right\n\n- **You have, or can get, 300 or more real examples** of exactly the input and the output you want. Below that, few-shot prompting is usually as good.\n- **A long prompt is being paid on every request.** If you are sending 2,000 tokens of instructions and examples a hundred thousand times a month, that is 200 million input tokens. Tuning that behaviour into the weights removes the recurring cost, and this is often the strongest financial argument.\n- **You need a smaller model than the task needs.** A tuned 3B that matches an untuned 30B on your narrow task costs an order of magnitude less to serve and runs where the 30B cannot. This is the main reason fine-tuning is worth it on device.\n- **A format or style must hold over long outputs.** Prompted constraints decay; trained behaviour does not.\n- **The model is systematically wrong in a way you can demonstrate**, on a domain with its own conventions, and you have examples of right.\n\n## The cost people underestimate\n\nThe run is the cheap part. A LoRA on 500 examples is under an hour on a free notebook GPU.\n\nThe expensive parts are the dataset, which is two or three days of careful work by somebody who understands the task, and the **permanence**. A fine-tune is a fork. When a better base model appears in four months, you re-tune, re-evaluate and re-deploy, or you stay on the old base while the field moves. Every fine-tune is a recurring commitment, and teams routinely take on three or four of them before noticing.\n\nBudget it honestly: the first tune costs days, and each subsequent base upgrade costs a day. If the improvement does not justify that annually, do not start.\n\n## A decision you can defend\n\nWrite down, before you begin, three things:\n\n1. The number your fine-tune must beat, measured on a held-out set with the best prompt you could write.\n2. The margin that would count as worth it, given the noise on a set that size.\n3. What you will do when the base model is superseded.\n\nIf you cannot fill in the third line, you are not ready to fine-tune. That is not a rhetorical flourish; abandoned fine-tunes running on two-year-old bases are one of the most common forms of technical debt in this work.\n\n> The most useful sentence in this lesson, if you take only one: measure the best prompted result first, write it down, and treat it as the thing to beat. Half the time nothing beats it and you have saved a week."},"check":{"question":"A team's model gets product specifications wrong. They plan to fine-tune on 2,000 question-and-answer pairs drawn from their specification documents. What is the strongest objection?","options":[{"text":"2,000 examples is too few to teach a model anything, so they need at least ten times more.","correct":false,"tempting":"Volume is not the binding issue; 2,000 examples is ample for behavioural tuning, and more would not fix the mismatch."},{"text":"This is a knowledge problem, which fine-tuning solves poorly and unpredictably, while retrieval solves it cheaply, updatably and with citations.","correct":true},{"text":"Fine-tuning on question-and-answer pairs will damage the model's general instruction following.","correct":false,"tempting":"Forgetting is a real risk to measure, but it is a side effect rather than the reason this plan is the wrong tool."},{"text":"They should use a larger model, since specification detail requires more parameters.","correct":false,"tempting":"A larger model still will not contain their private specifications, so the same gap persists at greater cost."}]},"prev":{"slug":"when-the-model-does-not-know","title":"When the model does not know","url":"https://addaly.com/learn/open-models/when-the-model-does-not-know"},"next":{"slug":"lora-mechanics","title":"What a LoRA actually is","url":"https://addaly.com/learn/open-models/lora-mechanics"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When not to fine-tune', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/when-not-to-fine-tune (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When not to fine-tune","courseTitle":"The Open Model Ecosystem","module":{"index":6,"slug":"adapting-a-model","title":"Adapting a model to your work"},"answer":null,"keyIdea":"Fine-tuning reliably changes form and unreliably changes knowledge, so the honest test is whether it beats your best prompt plus retrieval by a margin that justifies re-tuning every time the base model is superseded.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-fine-tuning-changes-and-what-it-does-not","runs":[{"text":"What fine-tuning changes, and what it does not"}]},{"t":"p","runs":[{"text":"Fine-tuning adjusts weights on examples of the behaviour you want. It is very good at some things and unreliable at others, and the difference decides whether you should do it."}]},{"t":"p","runs":[{"text":"It reliably changes form.","b":true},{"text":" Output format, tone, register, verbosity, structure, the conventions of your domain, which of several plausible responses is preferred, and whether the model does the task at all rather than explaining it. This is what fine-tuning is for."}]},{"t":"p","runs":[{"text":"It changes knowledge poorly.","b":true},{"text":" You can teach facts by fine-tuning, and it works badly compared with putting the facts in the context. The model learns the surface of the statement, generalises it unpredictably, contradicts it when phrased differently, and forgets other things in the process. Retrieval is better on every axis: cheaper, updatable, citable, and it does not degrade anything else."}]},{"t":"p","runs":[{"text":"It does not add reasoning capacity.","b":true},{"text":" A 3B model tuned on your data is still a 3B model. Tuning can make it apply what it has more reliably to your task; it cannot make it think further ahead."}]},{"t":"h2","id":"the-ladder-in-order-of-cost","runs":[{"text":"The ladder, in order of cost"}]},{"t":"p","runs":[{"text":"Work down this list and stop when the result is good enough."}]},{"t":"ol","start":1,"items":[[{"text":"A better prompt.","b":true},{"text":" An hour, and it fixes more than people expect."}],[{"text":"Three to eight examples in the prompt.","b":true},{"text":" Another hour."}],[{"text":"Constrained output.","b":true},{"text":" Removes format failures entirely."}],[{"text":"Retrieval.","b":true},{"text":" Fixes knowledge problems, which is most of what looks like model weakness."}],[{"text":"A different or larger model.","b":true},{"text":" A day of evaluation."}],[{"text":"Fine-tuning.","b":true},{"text":" Days of data work, then permanent maintenance."}]]},{"t":"p","runs":[{"text":"Most teams that fine-tune skipped steps 3 and 4. That is the single most common waste in this part of the field."}]},{"t":"h2","id":"the-signals-that-fine-tuning-is-right","runs":[{"text":"The signals that fine-tuning is right"}]},{"t":"ul","items":[[{"text":"You have, or can get, 300 or more real examples","b":true},{"text":" of exactly the input and the output you want. Below that, few-shot prompting is usually as good."}],[{"text":"A long prompt is being paid on every request.","b":true},{"text":" If you are sending 2,000 tokens of instructions and examples a hundred thousand times a month, that is 200 million input tokens. Tuning that behaviour into the weights removes the recurring cost, and this is often the strongest financial argument."}],[{"text":"You need a smaller model than the task needs.","b":true},{"text":" A tuned 3B that matches an untuned 30B on your narrow task costs an order of magnitude less to serve and runs where the 30B cannot. This is the main reason fine-tuning is worth it on device."}],[{"text":"A format or style must hold over long outputs.","b":true},{"text":" Prompted constraints decay; trained behaviour does not."}],[{"text":"The model is systematically wrong in a way you can demonstrate","b":true},{"text":", on a domain with its own conventions, and you have examples of right."}]]},{"t":"h2","id":"the-cost-people-underestimate","runs":[{"text":"The cost people underestimate"}]},{"t":"p","runs":[{"text":"The run is the cheap part. A LoRA on 500 examples is under an hour on a free notebook GPU."}]},{"t":"p","runs":[{"text":"The expensive parts are the dataset, which is two or three days of careful work by somebody who understands the task, and the "},{"text":"permanence","b":true},{"text":". A fine-tune is a fork. When a better base model appears in four months, you re-tune, re-evaluate and re-deploy, or you stay on the old base while the field moves. Every fine-tune is a recurring commitment, and teams routinely take on three or four of them before noticing."}]},{"t":"p","runs":[{"text":"Budget it honestly: the first tune costs days, and each subsequent base upgrade costs a day. If the improvement does not justify that annually, do not start."}]},{"t":"h2","id":"a-decision-you-can-defend","runs":[{"text":"A decision you can defend"}]},{"t":"p","runs":[{"text":"Write down, before you begin, three things:"}]},{"t":"ol","start":1,"items":[[{"text":"The number your fine-tune must beat, measured on a held-out set with the best prompt you could write."}],[{"text":"The margin that would count as worth it, given the noise on a set that size."}],[{"text":"What you will do when the base model is superseded."}]]},{"t":"p","runs":[{"text":"If you cannot fill in the third line, you are not ready to fine-tune. That is not a rhetorical flourish; abandoned fine-tunes running on two-year-old bases are one of the most common forms of technical debt in this work."}]},{"t":"quote","runs":[{"text":"The most useful sentence in this lesson, if you take only one: measure the best prompted result first, write it down, and treat it as the thing to beat. Half the time nothing beats it and you have saved a week."}]}],"check":{"question":[{"text":"A team’s model gets product specifications wrong. They plan to fine-tune on 2,000 question-and-answer pairs drawn from their specification documents. What is the strongest objection?"}],"options":[{"text":[{"text":"2,000 examples is too few to teach a model anything, so they need at least ten times more."}],"correct":false,"tempting":[{"text":"Volume is not the binding issue; 2,000 examples is ample for behavioural tuning, and more would not fix the mismatch."}]},{"text":[{"text":"This is a knowledge problem, which fine-tuning solves poorly and unpredictably, while retrieval solves it cheaply, updatably and with citations."}],"correct":true,"tempting":null},{"text":[{"text":"Fine-tuning on question-and-answer pairs will damage the model’s general instruction following."}],"correct":false,"tempting":[{"text":"Forgetting is a real risk to measure, but it is a side effect rather than the reason this plan is the wrong tool."}]},{"text":[{"text":"They should use a larger model, since specification detail requires more parameters."}],"correct":false,"tempting":[{"text":"A larger model still will not contain their private specifications, so the same gap persists at greater cost."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"when-the-model-does-not-know","title":"When the model does not know","minutes":9},"next":{"slug":"lora-mechanics","title":"What a LoRA actually is","minutes":9}}}