{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"where-the-data-comes-from","title":"Where the data legitimately comes from","url":"https://addaly.com/learn/fine-tuning/where-the-data-comes-from","position":29,"of":84,"minutes":9,"answer":"Your own traffic has the right distribution, hand-written examples cover what traffic never contains.","keyIdea":"Your own traffic has the right distribution, hand-written examples cover what traffic never contains, and public sets are for replay: with a dataset's own licence and the terms under which its contents were generated being two separate permissions that a provenance column recorded at collection time is the only way to answer later."},"body":{"format":"markdown","text":"## Four sources, in descending order of usefulness\n\n**1. Your own traffic.** Real inputs from real people doing the real job. Nothing else has the right distribution: the typos, the half-sentences, the questions that do not fit your categories. Sample it deliberately: stratify by intent, language and length rather than taking the last thousand rows, which cover the last twenty minutes of one time zone.\n\nThe obligations attach here, not later. Check what your privacy notice promised, redact identifiers before anything lands in a training file, and record the basis on which you are using it. A training set is a copy of your users' words with the access controls taken off.\n\n**2. Adjacent real text.** Support tickets, forum threads, internal documentation, past reports: text produced by people doing the task, even if never typed at a model. It has the vocabulary and the register, and it needs conversion into instruction shape, which is work.\n\n**3. Team-written examples.** Slower, higher quality, and the only source that can cover a case which has never occurred. Twenty hand-written examples at the start of a project are worth more than two thousand scraped ones, because writing them forces you to decide what a good answer actually is, and you will change your mind twice while doing it.\n\n**4. Public instruction datasets.** Useful as a replay mixture rather than as your task data. Which brings the licence question.\n\n## The public sets worth knowing, and their terms\n\nLicences change; check the dataset card for the current terms before you rely on any of this.\n\n- **databricks-dolly-15k**: 15,000 prompt-and-response pairs written by Databricks employees, released CC BY-SA 3.0. Human-written, so no question about another provider's output terms. The share-alike condition is the thing to think about.\n- **OpenAssistant (oasst1, oasst2)**: human-generated, multilingual, crowd-sourced conversation trees, Apache-2.0. The most permissive substantial human-written set available.\n- **No Robots**: 10,000 human-written instruction-and-demonstration pairs, deliberately produced without model assistance. Released under a non-commercial licence, which is exactly the sort of detail that matters and is routinely overlooked.\n- **UltraChat and similar large synthetic sets**: permissively licensed *as datasets*, but generated by calling a hosted model, so the generating provider's terms are a separate question from the dataset's licence.\n\nThat last distinction is the one that catches teams. A dataset's own licence and the terms under which its contents were produced are two different permissions, and a permissive dataset licence cannot grant something the producer did not have.\n\n## The provenance column\n\nOne extra field per example, written at collection time, costs nothing and answers every question anybody will later ask:\n\n```json\n{\"messages\": [...],\n \"source\": \"support_logs\",\n \"licence\": \"internal\",\n \"basis\": \"tos_v4_analytics\",\n \"collected\": \"2026-08-14\",\n \"reviewed_by\": \"rk\"}\n```\n\nSix months later, somebody asks whether the model was trained on customer data. With this column it is a query. Without it, it is an investigation that ends in a guess.\n\n## How much of each\n\nA shape that works for a task fine-tune:\n\n- 60 to 80% task data from your own traffic or adjacent text\n- 10 to 20% hand-written examples covering the cases traffic does not contain: the edge cases, the refusals, the \"I do not have that information\" responses\n- 10 to 20% general instruction data as replay, to stop the model forgetting how to be a model\n\nThe middle slice is the one people skip, and it is the one that determines how the model behaves when something unusual arrives. If every training example is a question the model can answer, you have taught it that every question is answerable. That is exactly the behaviour people call hallucination, and you installed it.\n\n## What not to do\n\nDo not scrape a competitor's product outputs. Do not use a dataset because it is popular without reading its card. Do not take the last hundred thousand log rows because they were easy to get.\n\nAnd do not let dataset size become the goal. The LIMA result (a strong instruction-following model fine-tuned on a thousand carefully curated examples) is the standing evidence that curation beats volume at this scale. A thousand examples you have read is a better dataset than fifty thousand you have not."},"check":{"question":"A team builds a training set entirely from support conversations that were resolved successfully. The tuned model becomes markedly worse at saying it does not know something. What caused this?","options":[{"text":"Every example showed an answerable question being answered, so the model learned that answering is the job.","correct":true},{"text":"Successful conversations are longer, so the model learned to produce long answers, and length correlates with overconfidence.","correct":false,"tempting":"Substitutes a length-mediated story for a direct one; the behaviour appears equally in short answers drawn from the same filtered set."},{"text":"Support logs contain company-specific facts the model cannot store in weights, so it substitutes invented ones.","correct":false,"tempting":"Correctly notes weights are poor at facts, but that would produce wrong answers on known topics rather than a loss of the ability to decline."},{"text":"Filtering to resolved conversations reduced the dataset size below the threshold at which a model retains its pretrained refusal behaviour.","correct":false,"tempting":"Treats a distributional bias as a volume problem; the same bias appears at any size if every example is an answered question."}]},"prev":{"slug":"the-dataset","title":"The Dataset Decides Everything","url":"https://addaly.com/learn/fine-tuning/the-dataset"},"next":{"slug":"instruction-data-formats","title":"The three formats, and the conversion trap","url":"https://addaly.com/learn/fine-tuning/instruction-data-formats"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the data legitimately comes from', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/where-the-data-comes-from (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the data legitimately comes from","courseTitle":"Fine-Tuning, and When Not To","module":{"index":4,"slug":"building-the-dataset","title":"Building the dataset"},"answer":null,"keyIdea":"Your own traffic has the right distribution, hand-written examples cover what traffic never contains, and public sets are for replay: with a dataset’s own licence and the terms under which its contents were generated being two separate permissions that a provenance column recorded at collection time is the only way to answer later.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"four-sources-in-descending-order-of-usefulness","runs":[{"text":"Four sources, in descending order of usefulness"}]},{"t":"p","runs":[{"text":"1. Your own traffic.","b":true},{"text":" Real inputs from real people doing the real job. Nothing else has the right distribution: the typos, the half-sentences, the questions that do not fit your categories. Sample it deliberately: stratify by intent, language and length rather than taking the last thousand rows, which cover the last twenty minutes of one time zone."}]},{"t":"p","runs":[{"text":"The obligations attach here, not later. Check what your privacy notice promised, redact identifiers before anything lands in a training file, and record the basis on which you are using it. A training set is a copy of your users’ words with the access controls taken off."}]},{"t":"p","runs":[{"text":"2. Adjacent real text.","b":true},{"text":" Support tickets, forum threads, internal documentation, past reports: text produced by people doing the task, even if never typed at a model. It has the vocabulary and the register, and it needs conversion into instruction shape, which is work."}]},{"t":"p","runs":[{"text":"3. Team-written examples.","b":true},{"text":" Slower, higher quality, and the only source that can cover a case which has never occurred. Twenty hand-written examples at the start of a project are worth more than two thousand scraped ones, because writing them forces you to decide what a good answer actually is, and you will change your mind twice while doing it."}]},{"t":"p","runs":[{"text":"4. Public instruction datasets.","b":true},{"text":" Useful as a replay mixture rather than as your task data. Which brings the licence question."}]},{"t":"h2","id":"the-public-sets-worth-knowing-and-their-terms","runs":[{"text":"The public sets worth knowing, and their terms"}]},{"t":"p","runs":[{"text":"Licences change; check the dataset card for the current terms before you rely on any of this."}]},{"t":"ul","items":[[{"text":"databricks-dolly-15k","b":true},{"text":": 15,000 prompt-and-response pairs written by Databricks employees, released CC BY-SA 3.0. Human-written, so no question about another provider’s output terms. The share-alike condition is the thing to think about."}],[{"text":"OpenAssistant (oasst1, oasst2)","b":true},{"text":": human-generated, multilingual, crowd-sourced conversation trees, Apache-2.0. The most permissive substantial human-written set available."}],[{"text":"No Robots","b":true},{"text":": 10,000 human-written instruction-and-demonstration pairs, deliberately produced without model assistance. Released under a non-commercial licence, which is exactly the sort of detail that matters and is routinely overlooked."}],[{"text":"UltraChat and similar large synthetic sets","b":true},{"text":": permissively licensed "},{"text":"as datasets","i":true},{"text":", but generated by calling a hosted model, so the generating provider’s terms are a separate question from the dataset’s licence."}]]},{"t":"p","runs":[{"text":"That last distinction is the one that catches teams. A dataset’s own licence and the terms under which its contents were produced are two different permissions, and a permissive dataset licence cannot grant something the producer did not have."}]},{"t":"h2","id":"the-provenance-column","runs":[{"text":"The provenance column"}]},{"t":"p","runs":[{"text":"One extra field per example, written at collection time, costs nothing and answers every question anybody will later ask:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\"messages\": [...],\n \"source\": \"support_logs\",\n \"licence\": \"internal\",\n \"basis\": \"tos_v4_analytics\",\n \"collected\": \"2026-08-14\",\n \"reviewed_by\": \"rk\"}"},{"t":"p","runs":[{"text":"Six months later, somebody asks whether the model was trained on customer data. With this column it is a query. Without it, it is an investigation that ends in a guess."}]},{"t":"h2","id":"how-much-of-each","runs":[{"text":"How much of each"}]},{"t":"p","runs":[{"text":"A shape that works for a task fine-tune:"}]},{"t":"ul","items":[[{"text":"60 to 80% task data from your own traffic or adjacent text"}],[{"text":"10 to 20% hand-written examples covering the cases traffic does not contain: the edge cases, the refusals, the “I do not have that information” responses"}],[{"text":"10 to 20% general instruction data as replay, to stop the model forgetting how to be a model"}]]},{"t":"p","runs":[{"text":"The middle slice is the one people skip, and it is the one that determines how the model behaves when something unusual arrives. If every training example is a question the model can answer, you have taught it that every question is answerable. That is exactly the behaviour people call hallucination, and you installed it."}]},{"t":"h2","id":"what-not-to-do","runs":[{"text":"What not to do"}]},{"t":"p","runs":[{"text":"Do not scrape a competitor’s product outputs. Do not use a dataset because it is popular without reading its card. Do not take the last hundred thousand log rows because they were easy to get."}]},{"t":"p","runs":[{"text":"And do not let dataset size become the goal. The LIMA result (a strong instruction-following model fine-tuned on a thousand carefully curated examples) is the standing evidence that curation beats volume at this scale. A thousand examples you have read is a better dataset than fifty thousand you have not."}]}],"check":{"question":[{"text":"A team builds a training set entirely from support conversations that were resolved successfully. The tuned model becomes markedly worse at saying it does not know something. What caused this?"}],"options":[{"text":[{"text":"Every example showed an answerable question being answered, so the model learned that answering is the job."}],"correct":true,"tempting":null},{"text":[{"text":"Successful conversations are longer, so the model learned to produce long answers, and length correlates with overconfidence."}],"correct":false,"tempting":[{"text":"Substitutes a length-mediated story for a direct one; the behaviour appears equally in short answers drawn from the same filtered set."}]},{"text":[{"text":"Support logs contain company-specific facts the model cannot store in weights, so it substitutes invented ones."}],"correct":false,"tempting":[{"text":"Correctly notes weights are poor at facts, but that would produce wrong answers on known topics rather than a loss of the ability to decline."}]},{"text":[{"text":"Filtering to resolved conversations reduced the dataset size below the threshold at which a model retains its pretrained refusal behaviour."}],"correct":false,"tempting":[{"text":"Treats a distributional bias as a volume problem; the same bias appears at any size if every example is an answered question."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"the-dataset","title":"The Dataset Decides Everything","minutes":11},"next":{"slug":"instruction-data-formats","title":"The three formats, and the conversion trap","minutes":8}}}