{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"when-prompting-stops-working","title":"When prompting stops working: the fine-tuning decision","url":"https://addaly.com/learn/building-with-ai/when-prompting-stops-working","position":20,"of":80,"minutes":10,"answer":"Fine-tuning teaches format, style and compression, never facts, so ask first whether the model can do the task when shown the answer.","keyIdea":"Fine-tuning teaches format, style and compression, never facts, so ask first whether the model can do the task when shown the answer, and treat a tuned model as a permanent dependency rather than a one-off project."},"body":{"format":"markdown","text":"## The order of things to try\n\nFine-tuning is the fourth thing to try, not the first. The order, cheapest and fastest first:\n\n1. **Better prompt**: clearer rules, an explicit refusal path, a prefill. Hours.\n2. **Examples**: three to eight covering the boundary. Hours.\n3. **Retrieval or a tool**: if the failure is missing knowledge, no amount of tuning supplies it. Days.\n4. **Fine-tuning**: weeks, and a permanent maintenance obligation.\n\nMost teams that fine-tune early discover afterwards that a corrected prompt would have done it. The diagnostic question is precise: **can the model do this task at all when you show it the answer?** If yes, the problem is instruction or context, and tuning is the expensive route to the same place. If no (if it cannot produce the behaviour even with perfect examples in front of it) tuning is on the table.\n\n## What fine-tuning is actually good at\n\nThe honest list is narrower than the marketing suggests.\n\n**Format and style, learned deeply.** Producing your exact JSON dialect, your house voice, your labelling conventions, every time, without 800 tokens of instructions on each call. This is the most reliable win.\n\n**Compressing a huge prompt.** If a task needs forty examples, tuning moves them into weights. A 3,000-token prompt becomes 200 tokens, which at volume is a large saving in both money and latency.\n\n**Distillation.** Run a frontier model over 5,000 real inputs, keep the outputs a human verified, and train a small model on those pairs. You can often get a small model to within a few points of the big one on your specific narrow task, at a twentieth of the cost. This is the strongest economic case for tuning and it is underused.\n\n**A task with no natural language description.** Scoring leads on a pattern nobody can articulate but that lives in 20,000 historical decisions.\n\n## What it does not do\n\n**It does not add reliable knowledge.** Facts trained in this way are learned as statistical tendencies, not as records. The model will produce them confidently and blend them with adjacent nonsense, and you cannot update or delete one. If the goal is \"know our product catalogue\", the answer is retrieval, and it will remain retrieval.\n\n**It does not fix reasoning.** A small model tuned on examples of good reasoning imitates the appearance of it.\n\n**It does not survive model retirement.** Your tuned model is a fork of a base model, and when that base is deprecated your work does not transfer. Budget for retraining whenever the underlying family moves.\n\n## LoRA, and why tuning got cheap\n\nFull fine-tuning updates every weight and needs enough GPU memory for the model, its gradients and the optimiser state: for a 7B model, well over 60 GB. Out of reach on a laptop.\n\n**LoRA** freezes the original weights and trains a small pair of low-rank matrices injected beside them. You train perhaps 0.1 to 1 per cent of the parameters. **QLoRA** adds 4-bit quantisation of the frozen base, cutting memory further. A 7B model can be LoRA-tuned on a single 16 GB GPU (including a free Colab or Kaggle notebook) in a few hours.\n\nThe free path is real and worth naming: **Unsloth**, **Axolotl** and Hugging Face **PEFT** with **TRL** all do this, all are open source, and a free Colab T4 or Kaggle P100 session is enough for a 7B QLoRA run on a few thousand examples. Hosted fine-tuning from the major providers is the paid alternative; it is easier and you do not own the result.\n\nThe output is an adapter file of a few tens of megabytes that you load on top of the base model, which means you can keep several task-specific adapters and swap them.\n\n## Data is the whole job\n\nThe training run is the easy part. What determines the result:\n\n- **Quantity.** 50 examples does nothing. 500 to 1,000 clean, consistent examples is a real starting point for a narrow task; 5,000 is comfortable.\n- **Consistency above all.** Contradictory labels teach contradiction. If two annotators would disagree on 20 per cent of your data, tuning learns the disagreement and produces it at random. Fix the guidelines first.\n- **A held-out set that never enters training.** Split before you start, and evaluate against the *prompted* baseline you are trying to beat. \"The tuned model gets 91 per cent\" is meaningless without \"and the prompt got 89\".\n- **Coverage of refusals.** If every training example is a successful answer, you have trained the model never to decline. Include the cases where the right output is a refusal or a null.\n\n## The cost nobody quotes\n\nA fine-tune is not a project, it is a dependency. You now own a data pipeline, an evaluation set, a retraining schedule, versioned artefacts, and the obligation to redo all of it when the base model is retired or when your product changes. That ongoing cost is usually larger than the training run, and it is the reason the right answer is so often: improve the prompt, add retrieval, and revisit this in six months when you have 5,000 verified examples that you collected as a by-product of running the prompted version."},"check":{"question":"A team wants their assistant to stop getting product specifications wrong and proposes fine-tuning on the product catalogue. What is the strongest objection?","options":[{"text":"Fine-tuning requires far more examples than a catalogue provides.","correct":false,"tempting":"Raises a data-volume concern that a large catalogue could satisfy, which leaves the plan looking merely difficult rather than misdirected."},{"text":"Facts learned as weights cannot be cited, corrected or deleted when a specification changes, so this is a retrieval problem wearing a training costume.","correct":true},{"text":"It would be cheaper to use a larger model.","correct":false,"tempting":"Trades one cost for another without addressing that no model has been told the current specifications."},{"text":"The tuned model would lose its general conversational ability.","correct":false,"tempting":"Names a real risk of aggressive tuning, but it is a side effect rather than the reason the approach cannot solve the stated problem."}]},"prev":{"slug":"writing-for-many-languages","title":"Building for users who do not write in one language","url":"https://addaly.com/learn/building-with-ai/writing-for-many-languages"},"next":{"slug":"structured-output","title":"Structured output, and what a schema cannot promise","url":"https://addaly.com/learn/building-with-ai/structured-output"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When prompting stops working: the fine-tuning decision', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/when-prompting-stops-working (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When prompting stops working: the fine-tuning decision","courseTitle":"Building with AI","module":{"index":2,"slug":"prompting-as-engineering","title":"Prompting as engineering"},"answer":null,"keyIdea":"Fine-tuning teaches format, style and compression, never facts, so ask first whether the model can do the task when shown the answer, and treat a tuned model as a permanent dependency rather than a one-off project.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-order-of-things-to-try","runs":[{"text":"The order of things to try"}]},{"t":"p","runs":[{"text":"Fine-tuning is the fourth thing to try, not the first. The order, cheapest and fastest first:"}]},{"t":"ol","start":1,"items":[[{"text":"Better prompt","b":true},{"text":": clearer rules, an explicit refusal path, a prefill. Hours."}],[{"text":"Examples","b":true},{"text":": three to eight covering the boundary. Hours."}],[{"text":"Retrieval or a tool","b":true},{"text":": if the failure is missing knowledge, no amount of tuning supplies it. Days."}],[{"text":"Fine-tuning","b":true},{"text":": weeks, and a permanent maintenance obligation."}]]},{"t":"p","runs":[{"text":"Most teams that fine-tune early discover afterwards that a corrected prompt would have done it. The diagnostic question is precise: "},{"text":"can the model do this task at all when you show it the answer?","b":true},{"text":" If yes, the problem is instruction or context, and tuning is the expensive route to the same place. If no (if it cannot produce the behaviour even with perfect examples in front of it) tuning is on the table."}]},{"t":"h2","id":"what-fine-tuning-is-actually-good-at","runs":[{"text":"What fine-tuning is actually good at"}]},{"t":"p","runs":[{"text":"The honest list is narrower than the marketing suggests."}]},{"t":"p","runs":[{"text":"Format and style, learned deeply.","b":true},{"text":" Producing your exact JSON dialect, your house voice, your labelling conventions, every time, without 800 tokens of instructions on each call. This is the most reliable win."}]},{"t":"p","runs":[{"text":"Compressing a huge prompt.","b":true},{"text":" If a task needs forty examples, tuning moves them into weights. A 3,000-token prompt becomes 200 tokens, which at volume is a large saving in both money and latency."}]},{"t":"p","runs":[{"text":"Distillation.","b":true},{"text":" Run a frontier model over 5,000 real inputs, keep the outputs a human verified, and train a small model on those pairs. You can often get a small model to within a few points of the big one on your specific narrow task, at a twentieth of the cost. This is the strongest economic case for tuning and it is underused."}]},{"t":"p","runs":[{"text":"A task with no natural language description.","b":true},{"text":" Scoring leads on a pattern nobody can articulate but that lives in 20,000 historical decisions."}]},{"t":"h2","id":"what-it-does-not-do","runs":[{"text":"What it does not do"}]},{"t":"p","runs":[{"text":"It does not add reliable knowledge.","b":true},{"text":" Facts trained in this way are learned as statistical tendencies, not as records. The model will produce them confidently and blend them with adjacent nonsense, and you cannot update or delete one. If the goal is “know our product catalogue”, the answer is retrieval, and it will remain retrieval."}]},{"t":"p","runs":[{"text":"It does not fix reasoning.","b":true},{"text":" A small model tuned on examples of good reasoning imitates the appearance of it."}]},{"t":"p","runs":[{"text":"It does not survive model retirement.","b":true},{"text":" Your tuned model is a fork of a base model, and when that base is deprecated your work does not transfer. Budget for retraining whenever the underlying family moves."}]},{"t":"h2","id":"lora-and-why-tuning-got-cheap","runs":[{"text":"LoRA, and why tuning got cheap"}]},{"t":"p","runs":[{"text":"Full fine-tuning updates every weight and needs enough GPU memory for the model, its gradients and the optimiser state: for a 7B model, well over 60 GB. Out of reach on a laptop."}]},{"t":"p","runs":[{"text":"LoRA","b":true},{"text":" freezes the original weights and trains a small pair of low-rank matrices injected beside them. You train perhaps 0.1 to 1 per cent of the parameters. "},{"text":"QLoRA","b":true},{"text":" adds 4-bit quantisation of the frozen base, cutting memory further. A 7B model can be LoRA-tuned on a single 16 GB GPU (including a free Colab or Kaggle notebook) in a few hours."}]},{"t":"p","runs":[{"text":"The free path is real and worth naming: "},{"text":"Unsloth","b":true},{"text":", "},{"text":"Axolotl","b":true},{"text":" and Hugging Face "},{"text":"PEFT","b":true},{"text":" with "},{"text":"TRL","b":true},{"text":" all do this, all are open source, and a free Colab T4 or Kaggle P100 session is enough for a 7B QLoRA run on a few thousand examples. Hosted fine-tuning from the major providers is the paid alternative; it is easier and you do not own the result."}]},{"t":"p","runs":[{"text":"The output is an adapter file of a few tens of megabytes that you load on top of the base model, which means you can keep several task-specific adapters and swap them."}]},{"t":"h2","id":"data-is-the-whole-job","runs":[{"text":"Data is the whole job"}]},{"t":"p","runs":[{"text":"The training run is the easy part. What determines the result:"}]},{"t":"ul","items":[[{"text":"Quantity.","b":true},{"text":" 50 examples does nothing. 500 to 1,000 clean, consistent examples is a real starting point for a narrow task; 5,000 is comfortable."}],[{"text":"Consistency above all.","b":true},{"text":" Contradictory labels teach contradiction. If two annotators would disagree on 20 per cent of your data, tuning learns the disagreement and produces it at random. Fix the guidelines first."}],[{"text":"A held-out set that never enters training.","b":true},{"text":" Split before you start, and evaluate against the "},{"text":"prompted","i":true},{"text":" baseline you are trying to beat. “The tuned model gets 91 per cent” is meaningless without “and the prompt got 89”."}],[{"text":"Coverage of refusals.","b":true},{"text":" If every training example is a successful answer, you have trained the model never to decline. Include the cases where the right output is a refusal or a null."}]]},{"t":"h2","id":"the-cost-nobody-quotes","runs":[{"text":"The cost nobody quotes"}]},{"t":"p","runs":[{"text":"A fine-tune is not a project, it is a dependency. You now own a data pipeline, an evaluation set, a retraining schedule, versioned artefacts, and the obligation to redo all of it when the base model is retired or when your product changes. That ongoing cost is usually larger than the training run, and it is the reason the right answer is so often: improve the prompt, add retrieval, and revisit this in six months when you have 5,000 verified examples that you collected as a by-product of running the prompted version."}]}],"check":{"question":[{"text":"A team wants their assistant to stop getting product specifications wrong and proposes fine-tuning on the product catalogue. What is the strongest objection?"}],"options":[{"text":[{"text":"Fine-tuning requires far more examples than a catalogue provides."}],"correct":false,"tempting":[{"text":"Raises a data-volume concern that a large catalogue could satisfy, which leaves the plan looking merely difficult rather than misdirected."}]},{"text":[{"text":"Facts learned as weights cannot be cited, corrected or deleted when a specification changes, so this is a retrieval problem wearing a training costume."}],"correct":true,"tempting":null},{"text":[{"text":"It would be cheaper to use a larger model."}],"correct":false,"tempting":[{"text":"Trades one cost for another without addressing that no model has been told the current specifications."}]},{"text":[{"text":"The tuned model would lose its general conversational ability."}],"correct":false,"tempting":[{"text":"Names a real risk of aggressive tuning, but it is a side effect rather than the reason the approach cannot solve the stated problem."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"writing-for-many-languages","title":"Building for users who do not write in one language","minutes":9},"next":{"slug":"structured-output","title":"Structured output, and what a schema cannot promise","minutes":7}}}