{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"prompt-retrieval-or-training","title":"Prompt, retrieval or training: choosing the right tool","url":"https://addaly.com/learn/running-models-yourself/prompt-retrieval-or-training","position":74,"of":82,"minutes":9,"answer":"Fine-tuning changes behaviour and does not install facts, so a knowledge gap belongs to retrieval and a shape problem belongs to the prompt: train only for a durable behaviour.","keyIdea":"Fine-tuning changes behaviour and does not install facts, so a knowledge gap belongs to retrieval and a shape problem belongs to the prompt: train only for a durable behaviour, and only after writing down the measurement that will tell you whether the run worked."},"body":{"format":"markdown","text":"## Three failures that look identical\n\nA model gives you a bad answer. There are three fundamentally different reasons, they present the same way, and the fixes have wildly different costs. Choosing wrongly here wastes more time than every other mistake in this course combined.\n\n**It does not know the fact.** The information is recent, internal, or absent from its training. Retrieval fixes this. Training does not.\n\n**It knows but produces the wrong shape.** Right content, wrong format, wrong length, wrong register, wrong language. A better prompt fixes this, often completely.\n\n**It cannot hold the behaviour.** You have written the instructions clearly, given examples, and it still drifts after a few turns or gets the judgement wrong on the cases that matter. This is where training earns its cost.\n\n## The expensive mistake, stated plainly\n\nFine-tuning does not install facts. It adjusts behaviour.\n\nWhen you fine-tune on question-and-answer pairs about your product, the model learns the *style* of answering questions about your product. It does not reliably learn the answers. What you get back is a model that answers confidently, in exactly your house voice, about things it does not know, which is worse than the original, because the fluency has removed the signal that used to warn you.\n\nThis is the single most common failed fine-tuning project. If the requirement is \"it should know our documentation\", the answer is retrieval, every time. If the requirement is \"it should answer the way our support team does, in our format, with our escalation rules\", that is a behaviour, and training is a reasonable tool.\n\n> Figure: Four questions, in order, before any training script. Steps: 1. A fact it never saw? (Retrieval. Training adjusts behaviour, not facts); 2. Right content, wrong shape? (A clearer prompt, examples, or a grammar); 3. Drifts on cases that matter? (A LoRA. Write the measurement down first); 4. Cannot reach the domain? (A different base model, almost always). Each rung costs roughly ten times the one before, in time as much as in compute. Teams that skip to fine-tuning frequently discover afterwards that the prompt was carrying a template bug all along, and that the model was fine.\n\n> Retrieval changes what the model is looking at. Training changes what it does with whatever it is looking at.\n\n## Work up the ladder, not down it\n\nEach rung costs roughly ten times the one before, in time as much as compute.\n\n1. **A clearer prompt.** Minutes. Still the highest-return change available, and small models respond to it more than large ones do.\n2. **Few-shot examples in the prompt.** An hour. Three to five well-chosen examples fix most format problems outright, and you can change them without retraining anything.\n3. **Structured output constraints.** An hour. If the problem is that the JSON is sometimes malformed, constrain the grammar rather than training.\n4. **Retrieval.** Days. The right answer for anything factual, changing, or too large to fit in context.\n5. **A LoRA adapter.** A week, most of it on the dataset. Right for a durable behaviour that examples cannot hold.\n6. **Continued pretraining.** Weeks and real money. Right for a language or a domain vocabulary the base never saw.\n\nDo not skip rungs. A team that jumps to fine-tuning frequently discovers afterwards that the prompt was carrying a template bug all along, and that the model was fine.\n\n## Where training wins\n\nFour cases come up repeatedly and are worth recognising.\n\n**Teaching a small model one task a large one already does.** Generate a few thousand examples with a capable model, train a 3B or 7B on them, and you get something that does that one job at a fraction of the cost and latency. This is the most reliable win in the whole area, because you are transferring a behaviour that demonstrably exists.\n\n**A format the model will not hold.** A proprietary markup, a fixed report structure, a domain-specific query language. Examples in the prompt get you most of the way; training gets you the last part and removes the examples from every request.\n\n**Judgement on cases where instructions do not generalise.** Classification with forty labels and subtle boundaries, where the definitions run to several pages. A few hundred labelled examples encode the boundary better than the definitions do.\n\n**Cutting the prompt.** A 2,000-token system prompt sent on every call is a real cost in latency, in tokens and in prefill time. Training that behaviour into the weights removes it from every request forever, which on a high-volume endpoint pays for the training run quickly.\n\n## Decide before you start, and write it down\n\nWrite one sentence before touching a training script: \"After this, the model will do X, measured by Y, on inputs of shape Z.\" If you cannot write Y (the measurement) you are not ready, because you will have no way to know whether the run worked, and a fine-tuned model that feels better is the easiest thing in the world to produce and the hardest to defend.\n\nThe measurement also protects you from the failure that catches good teams: the model gets better at your task and quietly worse at everything else. That is a real phenomenon with a name, it arrives without warning, and you only see it if you were already measuring something outside the task."},"check":{"question":"A team fine-tunes a 7B model on 4,000 question-and-answer pairs drawn from their internal documentation. Afterwards it answers fluently in their house style and gets specific details wrong more often than the base model did. What happened?","options":[{"text":"Training taught the style of answering, not the answers, so guesses arrive confidently.","correct":true},{"text":"The learning rate was too high, so the model overfitted to surface style rather than to content.","correct":false,"tempting":"Overfitting is real but would show as memorised training answers, not as a general shift towards confident wrong answers on unseen questions."},{"text":"The dataset was too small, and several tens of thousands of pairs would have taught the content properly.","correct":false,"tempting":"Scaling the same approach amplifies the mechanism rather than fixing it; factual coverage is retrieval's job regardless of dataset size."},{"text":"The adapter rank was too low to store the documentation, so the facts did not fit into the update.","correct":false,"tempting":"Treats the adapter as storage for content; even a full fine-tune has this failure, which is why the fix is architectural rather than a capacity setting."}]},"prev":{"slug":"when-the-api-simply-wins","title":"When the API simply wins, and when it cannot","url":"https://addaly.com/learn/running-models-yourself/when-the-api-simply-wins"},"next":{"slug":"what-a-lora-actually-is","title":"What a LoRA actually is","url":"https://addaly.com/learn/running-models-yourself/what-a-lora-actually-is"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Prompt, retrieval or training: choosing the right tool', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/prompt-retrieval-or-training (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Prompt, retrieval or training: choosing the right tool","courseTitle":"Running Models Yourself","module":{"index":9,"slug":"making-it-yours","title":"Making it yours: adapters and training"},"answer":null,"keyIdea":"Fine-tuning changes behaviour and does not install facts, so a knowledge gap belongs to retrieval and a shape problem belongs to the prompt: train only for a durable behaviour, and only after writing down the measurement that will tell you whether the run worked.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 9.1","figure":{"type":"flow","steps":[{"label":"A fact it never saw?","detail":"Retrieval. Training adjusts behaviour, not facts"},{"label":"Right content, wrong shape?","detail":"A clearer prompt, examples, or a grammar"},{"label":"Drifts on cases that matter?","detail":"A LoRA. Write the measurement down first"},{"label":"Cannot reach the domain?","detail":"A different base model, almost always"}],"title":"Four questions, in order, before any training script","caption":"Each rung costs roughly ten times the one before, in time as much as in compute. Teams that skip to fine-tuning frequently discover afterwards that the prompt was carrying a template bug all along, and that the model was fine."},"alt":"Figure: Four questions, in order, before any training script. Steps: 1. A fact it never saw? (Retrieval. Training adjusts behaviour, not facts); 2. Right content, wrong shape? (A clearer prompt, examples, or a grammar); 3. Drifts on cases that matter? (A LoRA. Write the measurement down first); 4. Cannot reach the domain? (A different base model, almost always). Each rung costs roughly ten times the one before, in time as much as in compute. Teams that skip to fine-tuning frequently discover afterwards that the prompt was carrying a template bug all along, and that the model was fine."}},"blocks":[{"t":"h2","id":"three-failures-that-look-identical","runs":[{"text":"Three failures that look identical"}]},{"t":"p","runs":[{"text":"A model gives you a bad answer. There are three fundamentally different reasons, they present the same way, and the fixes have wildly different costs. Choosing wrongly here wastes more time than every other mistake in this course combined."}]},{"t":"p","runs":[{"text":"It does not know the fact.","b":true},{"text":" The information is recent, internal, or absent from its training. Retrieval fixes this. Training does not."}]},{"t":"p","runs":[{"text":"It knows but produces the wrong shape.","b":true},{"text":" Right content, wrong format, wrong length, wrong register, wrong language. A better prompt fixes this, often completely."}]},{"t":"p","runs":[{"text":"It cannot hold the behaviour.","b":true},{"text":" You have written the instructions clearly, given examples, and it still drifts after a few turns or gets the judgement wrong on the cases that matter. This is where training earns its cost."}]},{"t":"h2","id":"the-expensive-mistake-stated-plainly","runs":[{"text":"The expensive mistake, stated plainly"}]},{"t":"p","runs":[{"text":"Fine-tuning does not install facts. It adjusts behaviour."}]},{"t":"p","runs":[{"text":"When you fine-tune on question-and-answer pairs about your product, the model learns the "},{"text":"style","i":true},{"text":" of answering questions about your product. It does not reliably learn the answers. What you get back is a model that answers confidently, in exactly your house voice, about things it does not know, which is worse than the original, because the fluency has removed the signal that used to warn you."}]},{"t":"p","runs":[{"text":"This is the single most common failed fine-tuning project. If the requirement is “it should know our documentation”, the answer is retrieval, every time. If the requirement is “it should answer the way our support team does, in our format, with our escalation rules”, that is a behaviour, and training is a reasonable tool."}]},{"t":"quote","runs":[{"text":"Retrieval changes what the model is looking at. Training changes what it does with whatever it is looking at."}]},{"t":"h2","id":"work-up-the-ladder-not-down-it","runs":[{"text":"Work up the ladder, not down it"}]},{"t":"p","runs":[{"text":"Each rung costs roughly ten times the one before, in time as much as compute."}]},{"t":"ol","start":1,"items":[[{"text":"A clearer prompt.","b":true},{"text":" Minutes. Still the highest-return change available, and small models respond to it more than large ones do."}],[{"text":"Few-shot examples in the prompt.","b":true},{"text":" An hour. Three to five well-chosen examples fix most format problems outright, and you can change them without retraining anything."}],[{"text":"Structured output constraints.","b":true},{"text":" An hour. If the problem is that the JSON is sometimes malformed, constrain the grammar rather than training."}],[{"text":"Retrieval.","b":true},{"text":" Days. The right answer for anything factual, changing, or too large to fit in context."}],[{"text":"A LoRA adapter.","b":true},{"text":" A week, most of it on the dataset. Right for a durable behaviour that examples cannot hold."}],[{"text":"Continued pretraining.","b":true},{"text":" Weeks and real money. Right for a language or a domain vocabulary the base never saw."}]]},{"t":"p","runs":[{"text":"Do not skip rungs. A team that jumps to fine-tuning frequently discovers afterwards that the prompt was carrying a template bug all along, and that the model was fine."}]},{"t":"h2","id":"where-training-wins","runs":[{"text":"Where training wins"}]},{"t":"p","runs":[{"text":"Four cases come up repeatedly and are worth recognising."}]},{"t":"p","runs":[{"text":"Teaching a small model one task a large one already does.","b":true},{"text":" Generate a few thousand examples with a capable model, train a 3B or 7B on them, and you get something that does that one job at a fraction of the cost and latency. This is the most reliable win in the whole area, because you are transferring a behaviour that demonstrably exists."}]},{"t":"p","runs":[{"text":"A format the model will not hold.","b":true},{"text":" A proprietary markup, a fixed report structure, a domain-specific query language. Examples in the prompt get you most of the way; training gets you the last part and removes the examples from every request."}]},{"t":"p","runs":[{"text":"Judgement on cases where instructions do not generalise.","b":true},{"text":" Classification with forty labels and subtle boundaries, where the definitions run to several pages. A few hundred labelled examples encode the boundary better than the definitions do."}]},{"t":"p","runs":[{"text":"Cutting the prompt.","b":true},{"text":" A 2,000-token system prompt sent on every call is a real cost in latency, in tokens and in prefill time. Training that behaviour into the weights removes it from every request forever, which on a high-volume endpoint pays for the training run quickly."}]},{"t":"h2","id":"decide-before-you-start-and-write-it-down","runs":[{"text":"Decide before you start, and write it down"}]},{"t":"p","runs":[{"text":"Write one sentence before touching a training script: “After this, the model will do X, measured by Y, on inputs of shape Z.” If you cannot write Y (the measurement) you are not ready, because you will have no way to know whether the run worked, and a fine-tuned model that feels better is the easiest thing in the world to produce and the hardest to defend."}]},{"t":"p","runs":[{"text":"The measurement also protects you from the failure that catches good teams: the model gets better at your task and quietly worse at everything else. That is a real phenomenon with a name, it arrives without warning, and you only see it if you were already measuring something outside the task."}]}],"check":{"question":[{"text":"A team fine-tunes a 7B model on 4,000 question-and-answer pairs drawn from their internal documentation. Afterwards it answers fluently in their house style and gets specific details wrong more often than the base model did. What happened?"}],"options":[{"text":[{"text":"Training taught the style of answering, not the answers, so guesses arrive confidently."}],"correct":true,"tempting":null},{"text":[{"text":"The learning rate was too high, so the model overfitted to surface style rather than to content."}],"correct":false,"tempting":[{"text":"Overfitting is real but would show as memorised training answers, not as a general shift towards confident wrong answers on unseen questions."}]},{"text":[{"text":"The dataset was too small, and several tens of thousands of pairs would have taught the content properly."}],"correct":false,"tempting":[{"text":"Scaling the same approach amplifies the mechanism rather than fixing it; factual coverage is retrieval’s job regardless of dataset size."}]},{"text":[{"text":"The adapter rank was too low to store the documentation, so the facts did not fit into the update."}],"correct":false,"tempting":[{"text":"Treats the adapter as storage for content; even a full fine-tune has this failure, which is why the fix is architectural rather than a capacity setting."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"when-the-api-simply-wins","title":"When the API simply wins, and when it cannot","minutes":8},"next":{"slug":"what-a-lora-actually-is","title":"What a LoRA actually is","minutes":9}}}