{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"the-fine-tuning-dataset","title":"Five hundred examples, done properly","url":"https://addaly.com/learn/open-models/the-fine-tuning-dataset","position":54,"of":71,"minutes":9,"answer":"A fine-tune inherits every property of its dataset, so diversity of real inputs, byte-identical output format, measured label agreement and deduplicated splits matter far more than the number of examples.","keyIdea":"A fine-tune inherits every property of its dataset, so diversity of real inputs, byte-identical output format, measured label agreement and deduplicated splits matter far more than the number of examples."},"body":{"format":"markdown","text":"## The dataset is the model\n\nEverything about a fine-tuned model that you did not inherit from the base came from your examples. Every inconsistency in your labels becomes an inconsistency in the model. Every systematic error becomes a systematic behaviour. This is the part of the work that decides the outcome, and it is the part people rush.\n\n## How many, and of what\n\n**300 to 1,000 examples** is enough for most behavioural tuning. Below 200 the run is unstable and few-shot prompting is usually as good; above a few thousand the returns flatten unless the task is broad.\n\nQuality dominates quantity at this scale. A careful 400 beats a careless 4,000, because the careless set teaches the model to be careless in the same places.\n\n## The format\n\nOne conversation per line, as JSONL:\n\n```json\n{\"messages\":[{\"role\":\"system\",\"content\":\"Classify the ticket.\"},\n             {\"role\":\"user\",\"content\":\"Card declined twice at checkout.\"},\n             {\"role\":\"assistant\",\"content\":\"{\\\"label\\\":\\\"billing\\\",\\\"urgency\\\":\\\"high\\\"}\"}]}\n```\n\nThe trainer applies the model's chat template and, correctly configured, computes loss only on the assistant turn. Check this: training on the loss of the user's tokens teaches the model to generate user messages, and it is a real and quiet defect in hand-rolled training scripts.\n\n## Six properties that matter, in order\n\n**1. Diversity of inputs.** More important than volume. Four hundred examples covering the full range of what arrives beats four thousand variations on the same three shapes. Sample from real traffic across time, source and user type, not from whatever was easy to collect.\n\n**2. Consistency of output format.** Byte-identical. Same key order, same date format, same capitalisation, same phrasing of the standard sentences. The model copies your format exactly, including the inconsistency.\n\n**3. The edge cases you want handled.** Empty input, off-topic input, ambiguous input, the case that should be escalated, the thing that should be refused. If none of your 500 examples shows an abstention, the model will never abstain.\n\n**4. Real inputs, not invented ones.** Invented examples are cleaner, better punctuated and more polite than what people send. A model trained on them handles real messiness worse, and the gap only appears in production.\n\n**5. Label consistency you have measured.** Take 30 items, have a second person label them independently, and count agreement. If two careful people agree 80% of the time, the ceiling on your model is around 80%, and no amount of tuning gets past your own disagreement. This measurement takes an hour and reframes the whole project.\n\n**6. Splits, held from the start.** Training, development and a held-out set you touch once. Deduplicate across them: near-duplicates leaking from training into held-out is the most common reason a fine-tune reports a wonderful number and disappoints in production.\n\n## Where the data comes from\n\n**Production logs, corrected by a person.** The best source by a distance, because the inputs are real by construction. Take the model's current output, have someone fix it, keep the fixed version. Correction is several times faster than writing from scratch.\n\n**A larger model, with the terms checked.** Generate candidate outputs with a stronger model, then have a human accept, edit or reject each. Two cautions: the terms of service of closed commercial APIs generally prohibit using their outputs to train competing models, and an open-weights teacher passes its own licence conditions to the result. And the student inherits the teacher's style and its errors, so review rather than accept wholesale.\n\n**Templating and augmentation.** Useful for coverage of rare categories and dangerous as the bulk of a set, because templated variety is not real variety and the model learns the template.\n\n> Figure: Building five hundred examples in the order that matters. Steps: 1. Sample real traffic (Across time, source and user type, not whatever was easiest to collect); 2. A person corrects the current output (Several times faster than writing from scratch, and the inputs are real by construction); 3. Measure label agreement on 30 items (If two careful people agree 80 per cent of the time, 80 per cent is your ceiling); 4. Make the output format byte-identical (Same key order, same date format, same standard sentences. The model copies the inconsistency too); 5. Split, then deduplicate across the splits (Near-duplicates leaking into the held-out set is why a tune reports well and disappoints); 6. Read 50 of them as if you were the model (Delete every example whose answer was not derivable from the input, or it teaches guessing). The run is under an hour on a free notebook GPU. This is the two or three days, and it is the part that decides the outcome, because everything the tuned model does that it did not inherit from the base came from here.\n\n## The hour that saves the project\n\nBefore training, read 50 of your examples end to end as if you were the model. Ask of each: is the output actually derivable from the input? A surprising number of hand-built datasets contain examples where the correct answer depended on information the model was never given: a system the annotator could see, a convention they knew, context from the previous ticket.\n\nEvery such example teaches the model to guess. Remove them or add the missing context, and the model gets better at the whole task, not just those items."},"check":{"question":"Two careful annotators independently label 30 items from a fine-tuning set and agree on 22 of them. What does this tell you before any training happens?","options":[{"text":"The annotators need better training, and once they agree, the model will match their accuracy.","correct":false,"tempting":"Improving the guidelines is the right response, but the current data already carries the disagreement into the model."},{"text":"Agreement around 73% caps what the model can be measured to achieve, because the labels it learns from and is scored against contain that much disagreement.","correct":true},{"text":"The sample of 30 is too small to conclude anything, so proceed and measure the model instead.","correct":false,"tempting":"Thirty items gives a rough but actionable figure, and measuring the model against inconsistent labels will not reveal the cause."},{"text":"The task is too subjective for fine-tuning and needs a preference-based approach instead.","correct":false,"tempting":"Preference tuning also requires consistent judgements; the ambiguity has to be resolved in the guidelines either way."}]},"prev":{"slug":"qlora-and-memory","title":"Fitting a fine-tune in 16 GB","url":"https://addaly.com/learn/open-models/qlora-and-memory"},"next":{"slug":"running-your-first-fine-tune","title":"A real run, on free hardware","url":"https://addaly.com/learn/open-models/running-your-first-fine-tune"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Five hundred examples, done properly', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/the-fine-tuning-dataset (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Five hundred examples, done properly","courseTitle":"The Open Model Ecosystem","module":{"index":6,"slug":"adapting-a-model","title":"Adapting a model to your work"},"answer":null,"keyIdea":"A fine-tune inherits every property of its dataset, so diversity of real inputs, byte-identical output format, measured label agreement and deduplicated splits matter far more than the number of examples.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Sample real traffic","detail":"Across time, source and user type, not whatever was easiest to collect"},{"label":"A person corrects the current output","detail":"Several times faster than writing from scratch, and the inputs are real by construction"},{"label":"Measure label agreement on 30 items","detail":"If two careful people agree 80 per cent of the time, 80 per cent is your ceiling"},{"label":"Make the output format byte-identical","detail":"Same key order, same date format, same standard sentences. The model copies the inconsistency too"},{"label":"Split, then deduplicate across the splits","detail":"Near-duplicates leaking into the held-out set is why a tune reports well and disappoints"},{"label":"Read 50 of them as if you were the model","detail":"Delete every example whose answer was not derivable from the input, or it teaches guessing"}],"title":"Building five hundred examples in the order that matters","caption":"The run is under an hour on a free notebook GPU. This is the two or three days, and it is the part that decides the outcome, because everything the tuned model does that it did not inherit from the base came from here."},"alt":"Figure: Building five hundred examples in the order that matters. Steps: 1. Sample real traffic (Across time, source and user type, not whatever was easiest to collect); 2. A person corrects the current output (Several times faster than writing from scratch, and the inputs are real by construction); 3. Measure label agreement on 30 items (If two careful people agree 80 per cent of the time, 80 per cent is your ceiling); 4. Make the output format byte-identical (Same key order, same date format, same standard sentences. The model copies the inconsistency too); 5. Split, then deduplicate across the splits (Near-duplicates leaking into the held-out set is why a tune reports well and disappoints); 6. Read 50 of them as if you were the model (Delete every example whose answer was not derivable from the input, or it teaches guessing). The run is under an hour on a free notebook GPU. This is the two or three days, and it is the part that decides the outcome, because everything the tuned model does that it did not inherit from the base came from here."}},"blocks":[{"t":"h2","id":"the-dataset-is-the-model","runs":[{"text":"The dataset is the model"}]},{"t":"p","runs":[{"text":"Everything about a fine-tuned model that you did not inherit from the base came from your examples. Every inconsistency in your labels becomes an inconsistency in the model. Every systematic error becomes a systematic behaviour. This is the part of the work that decides the outcome, and it is the part people rush."}]},{"t":"h2","id":"how-many-and-of-what","runs":[{"text":"How many, and of what"}]},{"t":"p","runs":[{"text":"300 to 1,000 examples","b":true},{"text":" is enough for most behavioural tuning. Below 200 the run is unstable and few-shot prompting is usually as good; above a few thousand the returns flatten unless the task is broad."}]},{"t":"p","runs":[{"text":"Quality dominates quantity at this scale. A careful 400 beats a careless 4,000, because the careless set teaches the model to be careless in the same places."}]},{"t":"h2","id":"the-format","runs":[{"text":"The format"}]},{"t":"p","runs":[{"text":"One conversation per line, as JSONL:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"messages\":[{\"role\":\"system\",\"content\":\"Classify the ticket.\"},\n             {\"role\":\"user\",\"content\":\"Card declined twice at checkout.\"},\n             {\"role\":\"assistant\",\"content\":\"{\\\"label\\\":\\\"billing\\\",\\\"urgency\\\":\\\"high\\\"}\"}]}"},{"t":"p","runs":[{"text":"The trainer applies the model’s chat template and, correctly configured, computes loss only on the assistant turn. Check this: training on the loss of the user’s tokens teaches the model to generate user messages, and it is a real and quiet defect in hand-rolled training scripts."}]},{"t":"h2","id":"six-properties-that-matter-in-order","runs":[{"text":"Six properties that matter, in order"}]},{"t":"p","runs":[{"text":"1. Diversity of inputs.","b":true},{"text":" More important than volume. Four hundred examples covering the full range of what arrives beats four thousand variations on the same three shapes. Sample from real traffic across time, source and user type, not from whatever was easy to collect."}]},{"t":"p","runs":[{"text":"2. Consistency of output format.","b":true},{"text":" Byte-identical. Same key order, same date format, same capitalisation, same phrasing of the standard sentences. The model copies your format exactly, including the inconsistency."}]},{"t":"p","runs":[{"text":"3. The edge cases you want handled.","b":true},{"text":" Empty input, off-topic input, ambiguous input, the case that should be escalated, the thing that should be refused. If none of your 500 examples shows an abstention, the model will never abstain."}]},{"t":"p","runs":[{"text":"4. Real inputs, not invented ones.","b":true},{"text":" Invented examples are cleaner, better punctuated and more polite than what people send. A model trained on them handles real messiness worse, and the gap only appears in production."}]},{"t":"p","runs":[{"text":"5. Label consistency you have measured.","b":true},{"text":" Take 30 items, have a second person label them independently, and count agreement. If two careful people agree 80% of the time, the ceiling on your model is around 80%, and no amount of tuning gets past your own disagreement. This measurement takes an hour and reframes the whole project."}]},{"t":"p","runs":[{"text":"6. Splits, held from the start.","b":true},{"text":" Training, development and a held-out set you touch once. Deduplicate across them: near-duplicates leaking from training into held-out is the most common reason a fine-tune reports a wonderful number and disappoints in production."}]},{"t":"h2","id":"where-the-data-comes-from","runs":[{"text":"Where the data comes from"}]},{"t":"p","runs":[{"text":"Production logs, corrected by a person.","b":true},{"text":" The best source by a distance, because the inputs are real by construction. Take the model’s current output, have someone fix it, keep the fixed version. Correction is several times faster than writing from scratch."}]},{"t":"p","runs":[{"text":"A larger model, with the terms checked.","b":true},{"text":" Generate candidate outputs with a stronger model, then have a human accept, edit or reject each. Two cautions: the terms of service of closed commercial APIs generally prohibit using their outputs to train competing models, and an open-weights teacher passes its own licence conditions to the result. And the student inherits the teacher’s style and its errors, so review rather than accept wholesale."}]},{"t":"p","runs":[{"text":"Templating and augmentation.","b":true},{"text":" Useful for coverage of rare categories and dangerous as the bulk of a set, because templated variety is not real variety and the model learns the template."}]},{"t":"h2","id":"the-hour-that-saves-the-project","runs":[{"text":"The hour that saves the project"}]},{"t":"p","runs":[{"text":"Before training, read 50 of your examples end to end as if you were the model. Ask of each: is the output actually derivable from the input? A surprising number of hand-built datasets contain examples where the correct answer depended on information the model was never given: a system the annotator could see, a convention they knew, context from the previous ticket."}]},{"t":"p","runs":[{"text":"Every such example teaches the model to guess. Remove them or add the missing context, and the model gets better at the whole task, not just those items."}]}],"check":{"question":[{"text":"Two careful annotators independently label 30 items from a fine-tuning set and agree on 22 of them. What does this tell you before any training happens?"}],"options":[{"text":[{"text":"The annotators need better training, and once they agree, the model will match their accuracy."}],"correct":false,"tempting":[{"text":"Improving the guidelines is the right response, but the current data already carries the disagreement into the model."}]},{"text":[{"text":"Agreement around 73% caps what the model can be measured to achieve, because the labels it learns from and is scored against contain that much disagreement."}],"correct":true,"tempting":null},{"text":[{"text":"The sample of 30 is too small to conclude anything, so proceed and measure the model instead."}],"correct":false,"tempting":[{"text":"Thirty items gives a rough but actionable figure, and measuring the model against inconsistent labels will not reveal the cause."}]},{"text":[{"text":"The task is too subjective for fine-tuning and needs a preference-based approach instead."}],"correct":false,"tempting":[{"text":"Preference tuning also requires consistent judgements; the ambiguity has to be resolved in the guidelines either way."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"qlora-and-memory","title":"Fitting a fine-tune in 16 GB","minutes":9},"next":{"slug":"running-your-first-fine-tune","title":"A real run, on free hardware","minutes":10}}}