{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"the-baseline-you-must-beat","title":"The baseline you must beat","url":"https://addaly.com/learn/fine-tuning/the-baseline-you-must-beat","position":68,"of":84,"minutes":9,"answer":"Compare against three baselines (the frozen best prompt, the cost-matched alternative, and production today) on one table of quality and cost per thousand requests.","keyIdea":"Compare against three baselines (the frozen best prompt, the cost-matched alternative, and production today) on one table of quality and cost per thousand requests, because the alternative nobody was ever going to choose is not a baseline and a one-point difference on 200 items is not a difference."},"body":{"format":"markdown","text":"## Three baselines, not one\n\nA fine-tune's result means nothing without something to compare it against, and the comparison people run is almost always too flattering.\n\n**1. The frozen prompt baseline.** The same base model, with the best prompt you can write, pinned at a specific commit with specific sampling parameters. This tells you whether the training added anything over the fast loop.\n\n**2. The cost-matched baseline.** This is the one that gets skipped, and it is usually the one that decides the project. If you tuned an 8B model to save money over a hosted frontier model, the honest comparison is not *tuned 8B versus frontier*. It is *tuned 8B versus the smallest untuned model that a good prompt makes adequate, at the same total cost per thousand requests*.\n\nBecause that is the real alternative. Nobody was going to keep paying frontier prices; they were going to try a cheaper model with a better prompt.\n\n**3. The do-nothing baseline.** Whatever is in production today, measured on the same set. Sometimes it is better than both, and finding that out at the end of a project is common and avoidable.\n\n## Writing the prompt baseline honestly\n\nThe temptation is to compare against the prompt you had when you decided to fine-tune: the one whose failures motivated the project. That is a straw man, and everybody builds it without meaning to.\n\nSpend one full day on the prompt first, as though fine-tuning were unavailable:\n\n- Five to eight few-shot examples covering the cases you actually fail on, chosen from your training data.\n- An explicit output format, with constrained decoding enforcing it.\n- The instruction rewritten twice, with the alternatives measured rather than argued about.\n- The retrieval, if there is any, fixed first.\n\nThen freeze it. A meaningful fraction of fine-tuning projects end here, and ending here is a good outcome: a day spent instead of three weeks, and a system anybody on the team can modify.\n\n## The comparison table, as numbers\n\nPut every candidate on the same two axes: quality on your held-out set, and cost per thousand requests including serving.\n\n```\nsystem                       acc    p95      $/1k    notes\nproduction today            0.84   1.4s     4.20    frontier, old prompt\nfrontier + new prompt       0.91   1.5s     4.20    one day of work\nmid-tier + new prompt       0.88   0.9s     0.70    one day of work\ntuned 8B (LoRA)             0.92   0.4s     0.31    3 weeks + maintenance\ntuned 8B, merged & 4-bit    0.91   0.3s     0.22    quantisation cost 1 pt\n```\n\nRead that table honestly. The fine-tune is one point better than a day's prompt work on the frontier model, at a fourteenth of the cost and a third of the latency. Whether that is worth three weeks and a maintenance commitment depends entirely on your volume, and now you can compute it rather than argue about it.\n\nIf the row for the mid-tier model with a good prompt had read 0.92, the project would have been unnecessary, and the only way to know is to have run it.\n\n## Base or instruct: also a baseline\n\nA cheap comparison people skip: the same fine-tune from the base model and from the instruct-tuned variant of the same weights.\n\nInstruct usually wins for instruction-shaped tasks, needing less data because it already follows instructions. Base sometimes wins when you want no inherited behaviour: a very rigid output format where the instruct model's conversational habits are an obstacle.\n\nIt is two runs. Run both once on a new project and you will know which your data prefers.\n\n> Figure: Cost per thousand requests, same held-out set. Bars ($ per 1,000 requests): Production today: frontier, old prompt 4.2; Frontier plus a new prompt, one day's work 4.2; Mid-tier plus that same new prompt 0.7; Tuned 8B with a LoRA 0.31; Tuned 8B, merged and 4-bit 0.22. Accuracy on those five rows, in order: 0.84, 0.91, 0.88, 0.92, 0.91. The fine-tune is one point better than a day's prompt work on the frontier model, at a fourteenth of the cost. And if the mid-tier row had read 0.92, the project was unnecessary.\n\n## The intervals\n\nEvery number in that table needs an interval, or the comparison is a vibe. For a proportion on n items, a serviceable approximation is ±1.96·√(p(1−p)/n):\n\n- n = 200, p ≈ 0.9 → about ±4 points\n- n = 500 → about ±2.6 points\n- n = 1,000 → about ±1.9 points\n\nSo on a 200-item set, 0.92 against 0.91 is not a difference. You would need roughly ten times the items to resolve one point, and it is usually cheaper to decide that one point does not matter than to build a 2,000-item labelled set to measure it.\n\nAnd add the seed variance from the reproducibility lesson on top: your training noise floor and your evaluation interval are two separate uncertainties, and the honest comparison carries both."},"check":{"question":"A team tunes an 8B model to replace a frontier API and reports it matching frontier quality at a twentieth of the cost. What is the missing comparison that could invalidate the project?","options":[{"text":"A mid-tier untuned model with an equally good prompt, which may already match it with no training.","correct":true},{"text":"The frontier model with the same fine-tuning applied, which would show whether the gains came from the data or from the base model.","correct":false,"tempting":"Proposes an interesting experiment that is usually impossible and does not address whether the project was needed."},{"text":"The 8B model before tuning, to establish that the training changed anything at all.","correct":false,"tempting":"Names a necessary sanity check that the team implicitly has, and which cannot invalidate the project on its own."},{"text":"The frontier model's latency at the same batch size, since cost comparisons must hold throughput constant.","correct":false,"tempting":"Raises a serving detail that belongs in the cost column rather than a comparison that could change the decision."}]},"prev":{"slug":"did-it-help","title":"Did It Help, and What Did It Break","url":"https://addaly.com/learn/fine-tuning/did-it-help"},"next":{"slug":"a-held-out-set-that-holds","title":"A held-out set that actually holds","url":"https://addaly.com/learn/fine-tuning/a-held-out-set-that-holds"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The baseline you must beat', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/the-baseline-you-must-beat (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The baseline you must beat","courseTitle":"Fine-Tuning, and When Not To","module":{"index":8,"slug":"did-it-actually-work","title":"Did it actually work"},"answer":null,"keyIdea":"Compare against three baselines (the frozen best prompt, the cost-matched alternative, and production today) on one table of quality and cost per thousand requests, because the alternative nobody was ever going to choose is not a baseline and a one-point difference on 200 items is not a difference.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"$ per 1,000 requests","title":"Cost per thousand requests, same held-out set","series":[{"label":"Production today: frontier, old prompt","value":4.2},{"label":"Frontier plus a new prompt, one day’s work","value":4.2},{"label":"Mid-tier plus that same new prompt","value":0.7},{"label":"Tuned 8B with a LoRA","value":0.31},{"label":"Tuned 8B, merged and 4-bit","value":0.22,"highlight":true}],"caption":"Accuracy on those five rows, in order: 0.84, 0.91, 0.88, 0.92, 0.91. The fine-tune is one point better than a day’s prompt work on the frontier model, at a fourteenth of the cost. And if the mid-tier row had read 0.92, the project was unnecessary."},"alt":"Figure: Cost per thousand requests, same held-out set. Bars ($ per 1,000 requests): Production today: frontier, old prompt 4.2; Frontier plus a new prompt, one day’s work 4.2; Mid-tier plus that same new prompt 0.7; Tuned 8B with a LoRA 0.31; Tuned 8B, merged and 4-bit 0.22. Accuracy on those five rows, in order: 0.84, 0.91, 0.88, 0.92, 0.91. The fine-tune is one point better than a day’s prompt work on the frontier model, at a fourteenth of the cost. And if the mid-tier row had read 0.92, the project was unnecessary."}},"blocks":[{"t":"h2","id":"three-baselines-not-one","runs":[{"text":"Three baselines, not one"}]},{"t":"p","runs":[{"text":"A fine-tune’s result means nothing without something to compare it against, and the comparison people run is almost always too flattering."}]},{"t":"p","runs":[{"text":"1. The frozen prompt baseline.","b":true},{"text":" The same base model, with the best prompt you can write, pinned at a specific commit with specific sampling parameters. This tells you whether the training added anything over the fast loop."}]},{"t":"p","runs":[{"text":"2. The cost-matched baseline.","b":true},{"text":" This is the one that gets skipped, and it is usually the one that decides the project. If you tuned an 8B model to save money over a hosted frontier model, the honest comparison is not "},{"text":"tuned 8B versus frontier","i":true},{"text":". It is "},{"text":"tuned 8B versus the smallest untuned model that a good prompt makes adequate, at the same total cost per thousand requests","i":true},{"text":"."}]},{"t":"p","runs":[{"text":"Because that is the real alternative. Nobody was going to keep paying frontier prices; they were going to try a cheaper model with a better prompt."}]},{"t":"p","runs":[{"text":"3. The do-nothing baseline.","b":true},{"text":" Whatever is in production today, measured on the same set. Sometimes it is better than both, and finding that out at the end of a project is common and avoidable."}]},{"t":"h2","id":"writing-the-prompt-baseline-honestly","runs":[{"text":"Writing the prompt baseline honestly"}]},{"t":"p","runs":[{"text":"The temptation is to compare against the prompt you had when you decided to fine-tune: the one whose failures motivated the project. That is a straw man, and everybody builds it without meaning to."}]},{"t":"p","runs":[{"text":"Spend one full day on the prompt first, as though fine-tuning were unavailable:"}]},{"t":"ul","items":[[{"text":"Five to eight few-shot examples covering the cases you actually fail on, chosen from your training data."}],[{"text":"An explicit output format, with constrained decoding enforcing it."}],[{"text":"The instruction rewritten twice, with the alternatives measured rather than argued about."}],[{"text":"The retrieval, if there is any, fixed first."}]]},{"t":"p","runs":[{"text":"Then freeze it. A meaningful fraction of fine-tuning projects end here, and ending here is a good outcome: a day spent instead of three weeks, and a system anybody on the team can modify."}]},{"t":"h2","id":"the-comparison-table-as-numbers","runs":[{"text":"The comparison table, as numbers"}]},{"t":"p","runs":[{"text":"Put every candidate on the same two axes: quality on your held-out set, and cost per thousand requests including serving."}]},{"t":"code","lang":"","label":"Text","text":"system                       acc    p95      $/1k    notes\nproduction today            0.84   1.4s     4.20    frontier, old prompt\nfrontier + new prompt       0.91   1.5s     4.20    one day of work\nmid-tier + new prompt       0.88   0.9s     0.70    one day of work\ntuned 8B (LoRA)             0.92   0.4s     0.31    3 weeks + maintenance\ntuned 8B, merged & 4-bit    0.91   0.3s     0.22    quantisation cost 1 pt"},{"t":"p","runs":[{"text":"Read that table honestly. The fine-tune is one point better than a day’s prompt work on the frontier model, at a fourteenth of the cost and a third of the latency. Whether that is worth three weeks and a maintenance commitment depends entirely on your volume, and now you can compute it rather than argue about it."}]},{"t":"p","runs":[{"text":"If the row for the mid-tier model with a good prompt had read 0.92, the project would have been unnecessary, and the only way to know is to have run it."}]},{"t":"h2","id":"base-or-instruct-also-a-baseline","runs":[{"text":"Base or instruct: also a baseline"}]},{"t":"p","runs":[{"text":"A cheap comparison people skip: the same fine-tune from the base model and from the instruct-tuned variant of the same weights."}]},{"t":"p","runs":[{"text":"Instruct usually wins for instruction-shaped tasks, needing less data because it already follows instructions. Base sometimes wins when you want no inherited behaviour: a very rigid output format where the instruct model’s conversational habits are an obstacle."}]},{"t":"p","runs":[{"text":"It is two runs. Run both once on a new project and you will know which your data prefers."}]},{"t":"h2","id":"the-intervals","runs":[{"text":"The intervals"}]},{"t":"p","runs":[{"text":"Every number in that table needs an interval, or the comparison is a vibe. For a proportion on n items, a serviceable approximation is ±1.96·√(p(1−p)/n):"}]},{"t":"ul","items":[[{"text":"n = 200, p ≈ 0.9 → about ±4 points"}],[{"text":"n = 500 → about ±2.6 points"}],[{"text":"n = 1,000 → about ±1.9 points"}]]},{"t":"p","runs":[{"text":"So on a 200-item set, 0.92 against 0.91 is not a difference. You would need roughly ten times the items to resolve one point, and it is usually cheaper to decide that one point does not matter than to build a 2,000-item labelled set to measure it."}]},{"t":"p","runs":[{"text":"And add the seed variance from the reproducibility lesson on top: your training noise floor and your evaluation interval are two separate uncertainties, and the honest comparison carries both."}]}],"check":{"question":[{"text":"A team tunes an 8B model to replace a frontier API and reports it matching frontier quality at a twentieth of the cost. What is the missing comparison that could invalidate the project?"}],"options":[{"text":[{"text":"A mid-tier untuned model with an equally good prompt, which may already match it with no training."}],"correct":true,"tempting":null},{"text":[{"text":"The frontier model with the same fine-tuning applied, which would show whether the gains came from the data or from the base model."}],"correct":false,"tempting":[{"text":"Proposes an interesting experiment that is usually impossible and does not address whether the project was needed."}]},{"text":[{"text":"The 8B model before tuning, to establish that the training changed anything at all."}],"correct":false,"tempting":[{"text":"Names a necessary sanity check that the team implicitly has, and which cannot invalidate the project on its own."}]},{"text":[{"text":"The frontier model’s latency at the same batch size, since cost comparisons must hold throughput constant."}],"correct":false,"tempting":[{"text":"Raises a serving detail that belongs in the cost column rather than a comparison that could change the decision."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"did-it-help","title":"Did It Help, and What Did It Break","minutes":10},"next":{"slug":"a-held-out-set-that-holds","title":"A held-out set that actually holds","minutes":9}}}