Fine-Tuning, and When Not ToLesson 7 of 84

Contents

Fine-Tuning, and When Not To

Lesson 7 of 84. Sign in to keep your place.

Module 1 · Deciding whether to fine-tune at all

The cost of owning a model

9 min read

One modest fine-tune, before it shipsDataset, 2,000 exampleswith 300 hand-corrected25Evaluation set and canaryset15Reading candidate outputsand choosing10Config, masking bugs, twolearning rates8The GPU runs themselves1hours of a person’s timeThe GPU bill for eight to fifteen runs is about forty dollars. Everything else on this chart is paidin somebody’s attention, and the evaluation set has to come first.
The GPU bill for eight to fifteen runs is about forty dollars. Everything else on this chart is paid in somebody's attention, and the evaluation set has to come first.

Figure

The GPU bill is the small number

Ask somebody what a fine-tune costs and they will quote the training run: four dollars, eleven dollars, a free Colab session. That number is real and it is almost irrelevant. What you are buying is not a run, it is an artefact with a maintenance schedule attached.

Here is the fuller bill for one modest production fine-tune, with the parts nobody quotes.

One-off, before you ship

  • Dataset construction: 2,000 examples, of which maybe 300 are hand-written or hand-corrected. At five minutes each that is 25 hours, and that is the optimistic version.
  • Evaluation set construction: 200 items, labelled, held out, plus a canary set for general capability. Another 15 hours, and it must come first.
  • The training runs themselves: not one, but eight to fifteen, because you will change the template, find a masking bug, try two learning rates and two ranks. Call it $40 of GPU rather than $4.
  • Evaluation of each candidate: cheap per run, expensive in attention.

Recurring, for as long as it lives

  • Serving. Either a card that idles at 3 a.m. or a per-token bill from a provider that hosts adapters. A single always-on L4 is roughly $450 a month; a serverless endpoint is cheaper at low volume and has cold starts you will be asked about.
  • Monitoring. Someone has to look at the outputs, because a fine-tuned model degrades silently as the input distribution moves.
  • Retraining. Which brings us to the real cost.

The base model moves under you

A LoRA adapter is a set of weight deltas for one specific set of base weights. It does not transfer. When the base model you trained against is superseded, and open-weight families have historically shipped a meaningful upgrade every six to nine months, you have three choices, all of them costs:

  1. Stay on the old base. Your product’s quality floor is now frozen at the level of a model from last year, while the free alternative everyone else prompts keeps improving.
  2. Retrain onto the new base. Cheap if your dataset and pipeline are intact, expensive if they are a notebook on a laptop belonging to somebody who has left.
  3. Re-evaluate whether you still need the fine-tune at all. Frequently the honest answer is no: the new base does with a prompt what the old base needed training for. This happens often enough that it should be a scheduled question, not a surprise.

The rule that falls out: keep the dataset, the config, and the evaluation set in version control as first-class artefacts, because the model is the output of those things and will be regenerated several times. A team that can rebuild its fine-tune in an afternoon treats a base model upgrade as an improvement. A team that cannot treats it as a threat.

The second, quieter cost

A fine-tuned model is a dependency your organisation does not know how to reason about. Ask three practical questions about the one you are proposing:

  • If the person who trained it leaves next month, can somebody else reproduce it from what is written down?
  • When a customer complains about one specific output, what is the debugging procedure? For a prompt, you read the prompt. For weights, you look at the dataset and guess.
  • When legal asks what the model was trained on, is there an answer?

If the answers are no, nobody knows, and not really, the fine-tune is not an asset, it is a liability that currently performs well.

Where the sums work

None of this is an argument against fine-tuning. It is an argument for computing the right number. The cases where the full cost is clearly worth it share a shape:

  • High volume. The saving is per-request, so the case strengthens linearly with traffic and the fixed costs amortise. Below roughly a hundred thousand requests a month it is hard to make the arithmetic work on cost alone.
  • A stable task. Something whose definition will not change next quarter: transcript formatting, a fixed extraction schema, one language pair.
  • A constraint no prompt satisfies. Latency budgets under 200 ms, on-device deployment, an air-gapped environment, or data that may not leave the building.

Write the monthly saving and the annual maintenance on the same line before you start. If the saving is $900 a month and the maintenance is a day of engineering a month, you have not saved anything; you have converted a bill into a chore.

Every lesson here is open. Your progress is recorded once you pass the Running Models Yourself exam.

Check your understanding

A team’s LoRA adapter for Llama-3.1-8B performs well. Nine months later a markedly better base model of the same family ships. Why can they not simply load their adapter onto the new base?

Pick the one you would defend. Your answer stays private.

Next lesson · 10 minLicences, and what you may train on

Stuck on something?Ask Rafael, your AI tutor

Previous lesson: Distillation, the case that usually survives

How to cite: Addaly (2026) ‘The cost of owning a model’, in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/the-cost-of-owning-a-model (Accessed: 13 October 2026).