The baseline you must beat

Three baselines, not one

A fine-tune’s result means nothing without something to compare it against, and the comparison people run is almost always too flattering.

1. The frozen prompt baseline. The same base model, with the best prompt you can write, pinned at a specific commit with specific sampling parameters. This tells you whether the training added anything over the fast loop.

2. The cost-matched baseline. This is the one that gets skipped, and it is usually the one that decides the project. If you tuned an 8B model to save money over a hosted frontier model, the honest comparison is not tuned 8B versus frontier. It is tuned 8B versus the smallest untuned model that a good prompt makes adequate, at the same total cost per thousand requests.

Because that is the real alternative. Nobody was going to keep paying frontier prices; they were going to try a cheaper model with a better prompt.

3. The do-nothing baseline. Whatever is in production today, measured on the same set. Sometimes it is better than both, and finding that out at the end of a project is common and avoidable.

Writing the prompt baseline honestly

The temptation is to compare against the prompt you had when you decided to fine-tune: the one whose failures motivated the project. That is a straw man, and everybody builds it without meaning to.

Spend one full day on the prompt first, as though fine-tuning were unavailable:

• Five to eight few-shot examples covering the cases you actually fail on, chosen from your training data.
• An explicit output format, with constrained decoding enforcing it.
• The instruction rewritten twice, with the alternatives measured rather than argued about.
• The retrieval, if there is any, fixed first.

Then freeze it. A meaningful fraction of fine-tuning projects end here, and ending here is a good outcome: a day spent instead of three weeks, and a system anybody on the team can modify.

The comparison table, as numbers

Put every candidate on the same two axes: quality on your held-out set, and cost per thousand requests including serving.

system                       acc    p95      $/1k    notes
production today            0.84   1.4s     4.20    frontier, old prompt
frontier + new prompt       0.91   1.5s     4.20    one day of work
mid-tier + new prompt       0.88   0.9s     0.70    one day of work
tuned 8B (LoRA)             0.92   0.4s     0.31    3 weeks + maintenance
tuned 8B, merged & 4-bit    0.91   0.3s     0.22    quantisation cost 1 pt

Read that table honestly. The fine-tune is one point better than a day’s prompt work on the frontier model, at a fourteenth of the cost and a third of the latency. Whether that is worth three weeks and a maintenance commitment depends entirely on your volume, and now you can compute it rather than argue about it.

If the row for the mid-tier model with a good prompt had read 0.92, the project would have been unnecessary, and the only way to know is to have run it.

Base or instruct: also a baseline

A cheap comparison people skip: the same fine-tune from the base model and from the instruct-tuned variant of the same weights.

Instruct usually wins for instruction-shaped tasks, needing less data because it already follows instructions. Base sometimes wins when you want no inherited behaviour: a very rigid output format where the instruct model’s conversational habits are an obstacle.

It is two runs. Run both once on a new project and you will know which your data prefers.

The intervals

Every number in that table needs an interval, or the comparison is a vibe. For a proportion on n items, a serviceable approximation is ±1.96·√(p(1−p)/n):

• n = 200, p ≈ 0.9 → about ±4 points
• n = 500 → about ±2.6 points
• n = 1,000 → about ±1.9 points

So on a 200-item set, 0.92 against 0.91 is not a difference. You would need roughly ten times the items to resolve one point, and it is usually cheaper to decide that one point does not matter than to build a 2,000-item labelled set to measure it.

And add the seed variance from the reproducibility lesson on top: your training noise floor and your evaluation interval are two separate uncertainties, and the honest comparison carries both.