When prompting stops working: the fine-tuning decision

The order of things to try

Fine-tuning is the fourth thing to try, not the first. The order, cheapest and fastest first:

1. Better prompt: clearer rules, an explicit refusal path, a prefill. Hours.
2. Examples: three to eight covering the boundary. Hours.
3. Retrieval or a tool: if the failure is missing knowledge, no amount of tuning supplies it. Days.
4. Fine-tuning: weeks, and a permanent maintenance obligation.

Most teams that fine-tune early discover afterwards that a corrected prompt would have done it. The diagnostic question is precise: can the model do this task at all when you show it the answer? If yes, the problem is instruction or context, and tuning is the expensive route to the same place. If no (if it cannot produce the behaviour even with perfect examples in front of it) tuning is on the table.

What fine-tuning is actually good at

The honest list is narrower than the marketing suggests.

Format and style, learned deeply. Producing your exact JSON dialect, your house voice, your labelling conventions, every time, without 800 tokens of instructions on each call. This is the most reliable win.

Compressing a huge prompt. If a task needs forty examples, tuning moves them into weights. A 3,000-token prompt becomes 200 tokens, which at volume is a large saving in both money and latency.

Distillation. Run a frontier model over 5,000 real inputs, keep the outputs a human verified, and train a small model on those pairs. You can often get a small model to within a few points of the big one on your specific narrow task, at a twentieth of the cost. This is the strongest economic case for tuning and it is underused.

A task with no natural language description. Scoring leads on a pattern nobody can articulate but that lives in 20,000 historical decisions.

What it does not do

It does not add reliable knowledge. Facts trained in this way are learned as statistical tendencies, not as records. The model will produce them confidently and blend them with adjacent nonsense, and you cannot update or delete one. If the goal is “know our product catalogue”, the answer is retrieval, and it will remain retrieval.

It does not fix reasoning. A small model tuned on examples of good reasoning imitates the appearance of it.

It does not survive model retirement. Your tuned model is a fork of a base model, and when that base is deprecated your work does not transfer. Budget for retraining whenever the underlying family moves.

LoRA, and why tuning got cheap

Full fine-tuning updates every weight and needs enough GPU memory for the model, its gradients and the optimiser state: for a 7B model, well over 60 GB. Out of reach on a laptop.

LoRA freezes the original weights and trains a small pair of low-rank matrices injected beside them. You train perhaps 0.1 to 1 per cent of the parameters. QLoRA adds 4-bit quantisation of the frozen base, cutting memory further. A 7B model can be LoRA-tuned on a single 16 GB GPU (including a free Colab or Kaggle notebook) in a few hours.

The free path is real and worth naming: Unsloth, Axolotl and Hugging Face PEFT with TRL all do this, all are open source, and a free Colab T4 or Kaggle P100 session is enough for a 7B QLoRA run on a few thousand examples. Hosted fine-tuning from the major providers is the paid alternative; it is easier and you do not own the result.

The output is an adapter file of a few tens of megabytes that you load on top of the base model, which means you can keep several task-specific adapters and swap them.

Data is the whole job

The training run is the easy part. What determines the result:

• Quantity. 50 examples does nothing. 500 to 1,000 clean, consistent examples is a real starting point for a narrow task; 5,000 is comfortable.
• Consistency above all. Contradictory labels teach contradiction. If two annotators would disagree on 20 per cent of your data, tuning learns the disagreement and produces it at random. Fix the guidelines first.
• A held-out set that never enters training. Split before you start, and evaluate against the prompted baseline you are trying to beat. “The tuned model gets 91 per cent” is meaningless without “and the prompt got 89”.
• Coverage of refusals. If every training example is a successful answer, you have trained the model never to decline. Include the cases where the right output is a refusal or a null.

The cost nobody quotes

A fine-tune is not a project, it is a dependency. You now own a data pipeline, an evaluation set, a retraining schedule, versioned artefacts, and the obligation to redo all of it when the base model is retired or when your product changes. That ongoing cost is usually larger than the training run, and it is the reason the right answer is so often: improve the prompt, add retrieval, and revisit this in six months when you have 5,000 verified examples that you collected as a by-product of running the prompted version.