Prompt, retrieval or training: choosing the right tool

Three failures that look identical

A model gives you a bad answer. There are three fundamentally different reasons, they present the same way, and the fixes have wildly different costs. Choosing wrongly here wastes more time than every other mistake in this course combined.

It does not know the fact. The information is recent, internal, or absent from its training. Retrieval fixes this. Training does not.

It knows but produces the wrong shape. Right content, wrong format, wrong length, wrong register, wrong language. A better prompt fixes this, often completely.

It cannot hold the behaviour. You have written the instructions clearly, given examples, and it still drifts after a few turns or gets the judgement wrong on the cases that matter. This is where training earns its cost.

The expensive mistake, stated plainly

Fine-tuning does not install facts. It adjusts behaviour.

When you fine-tune on question-and-answer pairs about your product, the model learns the style of answering questions about your product. It does not reliably learn the answers. What you get back is a model that answers confidently, in exactly your house voice, about things it does not know, which is worse than the original, because the fluency has removed the signal that used to warn you.

This is the single most common failed fine-tuning project. If the requirement is “it should know our documentation”, the answer is retrieval, every time. If the requirement is “it should answer the way our support team does, in our format, with our escalation rules”, that is a behaviour, and training is a reasonable tool.

Retrieval changes what the model is looking at. Training changes what it does with whatever it is looking at.

Work up the ladder, not down it

Each rung costs roughly ten times the one before, in time as much as compute.

1. A clearer prompt. Minutes. Still the highest-return change available, and small models respond to it more than large ones do.
2. Few-shot examples in the prompt. An hour. Three to five well-chosen examples fix most format problems outright, and you can change them without retraining anything.
3. Structured output constraints. An hour. If the problem is that the JSON is sometimes malformed, constrain the grammar rather than training.
4. Retrieval. Days. The right answer for anything factual, changing, or too large to fit in context.
5. A LoRA adapter. A week, most of it on the dataset. Right for a durable behaviour that examples cannot hold.
6. Continued pretraining. Weeks and real money. Right for a language or a domain vocabulary the base never saw.

Do not skip rungs. A team that jumps to fine-tuning frequently discovers afterwards that the prompt was carrying a template bug all along, and that the model was fine.

Where training wins

Four cases come up repeatedly and are worth recognising.

Teaching a small model one task a large one already does. Generate a few thousand examples with a capable model, train a 3B or 7B on them, and you get something that does that one job at a fraction of the cost and latency. This is the most reliable win in the whole area, because you are transferring a behaviour that demonstrably exists.

A format the model will not hold. A proprietary markup, a fixed report structure, a domain-specific query language. Examples in the prompt get you most of the way; training gets you the last part and removes the examples from every request.

Judgement on cases where instructions do not generalise. Classification with forty labels and subtle boundaries, where the definitions run to several pages. A few hundred labelled examples encode the boundary better than the definitions do.

Cutting the prompt. A 2,000-token system prompt sent on every call is a real cost in latency, in tokens and in prefill time. Training that behaviour into the weights removes it from every request forever, which on a high-volume endpoint pays for the training run quickly.

Decide before you start, and write it down

Write one sentence before touching a training script: “After this, the model will do X, measured by Y, on inputs of shape Z.” If you cannot write Y (the measurement) you are not ready, because you will have no way to know whether the run worked, and a fine-tuned model that feels better is the easiest thing in the world to produce and the hardest to defend.

The measurement also protects you from the failure that catches good teams: the model gets better at your task and quietly worse at everything else. That is a real phenomenon with a name, it arrives without warning, and you only see it if you were already measuring something outside the task.