{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"preparing-a-dataset-for-training","title":"From a dataset to batches a trainer can eat","url":"https://addaly.com/learn/hugging-face/preparing-a-dataset-for-training","position":47,"of":75,"minutes":9,"answer":"Training needs input_ids, attention_mask and a column literally named labels, with padding done per batch by a collator rather than at map time.","keyIdea":"Training needs `input_ids`, `attention_mask` and a column literally named `labels`, with padding done per batch by a collator rather than at map time, and for instruction tuning the prompt positions are set to `-100` so loss is computed only on the response."},"body":{"format":"markdown","text":"## The three things the model needs\n\nWhatever you are training, the loop wants tensors of the right shape with the right labels. For text that means: `input_ids`, `attention_mask`, and `labels`. Everything in this lesson is getting from a table of strings to those three.\n\n## Classification: labels are just a column\n\n```python\ndef prep(batch):\n    return tok(batch[\"text\"], truncation=True, max_length=512)\n\nds = ds.map(prep, batched=True, remove_columns=[\"text\"])\nds = ds.rename_column(\"label\", \"labels\")\nds.set_format(\"torch\")\n```\n\nThree specific points. The column must be called **`labels`**, plural, because that is what the model's forward signature expects: `label` silently becomes an unused column and the loss is never computed. **Do not pad here**; padding at map time pads everything to 512 and wastes compute. And `set_format(\"torch\")` makes rows come back as tensors without copying the underlying Arrow data.\n\n## Dynamic padding, with a collator\n\n```python\nfrom transformers import DataCollatorWithPadding\ncollator = DataCollatorWithPadding(tokenizer=tok)\n```\n\nThe collator pads each batch to that batch's longest item rather than to a global maximum. Combined with length-grouped batching (`group_by_length=True` in `TrainingArguments`) this is frequently a twofold speed-up for free, for the reasons the batching lesson gave.\n\n> Figure: What one instruction-tuning row has to contain. Layers, top to bottom: input_ids (The whole conversation, rendered by the model's own chat template); attention_mask (1 for real tokens, 0 for padding the collator adds per batch); labels (Plural. Named label instead and the loss is silently never computed.); -100 across the prompt (Loss falls on the response only, so the model does not learn to write the questions). Decode a collated batch and read both the input and the unmasked labels before you start a long run. A doubled beginning-of-sequence token, a chat template that did not render, and masking that covered the wrong span all show up here and nowhere else.\n\n## Causal language modelling: labels are the inputs\n\n```python\nfrom transformers import DataCollatorForLanguageModeling\ncollator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=False)\n```\n\nWith `mlm=False`, the collator copies `input_ids` into `labels` and sets padded positions to `-100`, which the loss ignores. The model shifts internally, so you do not shift yourself: doing it manually as well is a classic off-by-one that produces a model which predicts the current token instead of the next one and appears to learn beautifully while being useless.\n\nWith `mlm=True` it masks 15% of tokens instead, for BERT-style training.\n\n## Instruction tuning: mask the prompt\n\nFor supervised fine-tuning on instruction data, you usually do not want loss on the prompt: only on the response. Otherwise the model spends its capacity learning to generate the questions people ask it.\n\nThe mechanism is setting prompt positions in `labels` to `-100`:\n\n```python\ndef prep(example):\n    prompt = tok.apply_chat_template(example[\"messages\"][:-1],\n                                     tokenize=False, add_generation_prompt=True)\n    full = tok.apply_chat_template(example[\"messages\"], tokenize=False)\n\n    p_ids = tok(prompt, add_special_tokens=False)[\"input_ids\"]\n    f_ids = tok(full, add_special_tokens=False)[\"input_ids\"]\n\n    labels = list(f_ids)\n    labels[:len(p_ids)] = [-100] * len(p_ids)\n    return {\"input_ids\": f_ids, \"labels\": labels,\n            \"attention_mask\": [1] * len(f_ids)}\n```\n\nNote `add_special_tokens=False` on both encodes: the chat template has already inserted the special tokens, and letting the tokenizer add a second beginning-of-sequence marker shifts every offset by one and breaks the masking silently. TRL's `SFTTrainer` does all of this for you, which is why the training module reaches for it; doing it once by hand is how you understand what it is doing.\n\n## Packing, for pretraining\n\nWhen training on plain text, padding wastes a lot. Packing concatenates documents and cuts fixed-length blocks:\n\n```python\ndef group(batch, block=1024):\n    ids = sum(batch[\"input_ids\"], [])\n    n = (len(ids) // block) * block\n    chunks = [ids[i:i+block] for i in range(0, n, block)]\n    return {\"input_ids\": chunks, \"labels\": [c[:] for c in chunks]}\n\nlm_ds = tokenised.map(group, batched=True, remove_columns=tokenised.column_names)\n```\n\nNo padding at all, so every token in every batch is doing work. The cost is that documents bleed across block boundaries and attention crosses between unrelated texts. For pretraining this is standard and the effect is small; for instruction tuning it needs attention masking between packed examples, which the libraries handle and hand-rolled code usually does not.\n\n## Check the batch before you train\n\nThe cheapest hour you will ever save:\n\n```python\nbatch = collator([ds[i] for i in range(4)])\nprint({k: v.shape for k, v in batch.items()})\nprint(tok.decode(batch[\"input_ids\"][0]))\nlab = batch[\"labels\"][0]\nprint(tok.decode([t for t in lab if t != -100]))\n```\n\nRead that decoded text. It is exactly what the model will see. Three things show up here and nowhere else: a doubled beginning-of-sequence token, a chat template that did not render, and prompt masking that covered the wrong span. Each of those costs a full training run if you find it afterwards.\n\n## Do this now\n\nPrint one collated batch for whatever you plan to train, decode both the input and the unmasked labels, and read them out loud. If they are not what you intended, you have just saved the cost of the run."},"check":{"question":"A fine-tune runs without error, the loss falls, and the resulting model generates plausible user questions rather than answering them. What was most likely wrong in data preparation?","options":[{"text":"Padding was applied at map time to a fixed maximum, so the model learned from padded positions.","correct":false,"tempting":"Wasteful and slow, but padded positions are excluded from the loss by the collator."},{"text":"The label column was named `label` rather than `labels`, so no loss was computed on the responses.","correct":false,"tempting":"Would prevent learning entirely rather than teaching the model a different task, and the loss would not fall."},{"text":"The chat template was applied after tokenisation, so role markers were encoded as ordinary text.","correct":false,"tempting":"Produces degraded formatting rather than a systematic role reversal."},{"text":"Prompt tokens were not masked to `-100`, so the model was trained on the questions too.","correct":true}]},"prev":{"slug":"data-quality-before-the-model","title":"Duplicates, leakage, balance, and label noise","url":"https://addaly.com/learn/hugging-face/data-quality-before-the-model"},"next":{"slug":"what-an-embedding-is","title":"A list of numbers where near means similar","url":"https://addaly.com/learn/hugging-face/what-an-embedding-is"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'From a dataset to batches a trainer can eat', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/preparing-a-dataset-for-training (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"From a dataset to batches a trainer can eat","courseTitle":"Hugging Face, End to End","module":{"index":5,"slug":"datasets-and-the-work-before-the-model","title":"Datasets, and the work that happens before any model"},"answer":null,"keyIdea":"Training needs `input_ids`, `attention_mask` and a column literally named `labels`, with padding done per batch by a collator rather than at map time, and for instruction tuning the prompt positions are set to `-100` so loss is computed only on the response.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"What one instruction-tuning row has to contain","layers":[{"label":"input_ids","detail":"The whole conversation, rendered by the model’s own chat template"},{"label":"attention_mask","detail":"1 for real tokens, 0 for padding the collator adds per batch"},{"label":"labels","detail":"Plural. Named label instead and the loss is silently never computed."},{"label":"-100 across the prompt","detail":"Loss falls on the response only, so the model does not learn to write the questions"}],"caption":"Decode a collated batch and read both the input and the unmasked labels before you start a long run. A doubled beginning-of-sequence token, a chat template that did not render, and masking that covered the wrong span all show up here and nowhere else."},"alt":"Figure: What one instruction-tuning row has to contain. Layers, top to bottom: input_ids (The whole conversation, rendered by the model’s own chat template); attention_mask (1 for real tokens, 0 for padding the collator adds per batch); labels (Plural. Named label instead and the loss is silently never computed.); -100 across the prompt (Loss falls on the response only, so the model does not learn to write the questions). Decode a collated batch and read both the input and the unmasked labels before you start a long run. A doubled beginning-of-sequence token, a chat template that did not render, and masking that covered the wrong span all show up here and nowhere else."}},"blocks":[{"t":"h2","id":"the-three-things-the-model-needs","runs":[{"text":"The three things the model needs"}]},{"t":"p","runs":[{"text":"Whatever you are training, the loop wants tensors of the right shape with the right labels. For text that means: "},{"text":"input_ids","code":true},{"text":", "},{"text":"attention_mask","code":true},{"text":", and "},{"text":"labels","code":true},{"text":". Everything in this lesson is getting from a table of strings to those three."}]},{"t":"h2","id":"classification-labels-are-just-a-column","runs":[{"text":"Classification: labels are just a column"}]},{"t":"code","lang":"python","label":"Python","text":"def prep(batch):\n    return tok(batch[\"text\"], truncation=True, max_length=512)\n\nds = ds.map(prep, batched=True, remove_columns=[\"text\"])\nds = ds.rename_column(\"label\", \"labels\")\nds.set_format(\"torch\")"},{"t":"p","runs":[{"text":"Three specific points. The column must be called "},{"text":"labels","b":true,"code":true},{"text":", plural, because that is what the model’s forward signature expects: "},{"text":"label","code":true},{"text":" silently becomes an unused column and the loss is never computed. "},{"text":"Do not pad here","b":true},{"text":"; padding at map time pads everything to 512 and wastes compute. And "},{"text":"set_format(\"torch\")","code":true},{"text":" makes rows come back as tensors without copying the underlying Arrow data."}]},{"t":"h2","id":"dynamic-padding-with-a-collator","runs":[{"text":"Dynamic padding, with a collator"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import DataCollatorWithPadding\ncollator = DataCollatorWithPadding(tokenizer=tok)"},{"t":"p","runs":[{"text":"The collator pads each batch to that batch’s longest item rather than to a global maximum. Combined with length-grouped batching ("},{"text":"group_by_length=True","code":true},{"text":" in "},{"text":"TrainingArguments","code":true},{"text":") this is frequently a twofold speed-up for free, for the reasons the batching lesson gave."}]},{"t":"h2","id":"causal-language-modelling-labels-are-the-inputs","runs":[{"text":"Causal language modelling: labels are the inputs"}]},{"t":"code","lang":"python","label":"Python","text":"from transformers import DataCollatorForLanguageModeling\ncollator = DataCollatorForLanguageModeling(tokenizer=tok, mlm=False)"},{"t":"p","runs":[{"text":"With "},{"text":"mlm=False","code":true},{"text":", the collator copies "},{"text":"input_ids","code":true},{"text":" into "},{"text":"labels","code":true},{"text":" and sets padded positions to "},{"text":"-100","code":true},{"text":", which the loss ignores. The model shifts internally, so you do not shift yourself: doing it manually as well is a classic off-by-one that produces a model which predicts the current token instead of the next one and appears to learn beautifully while being useless."}]},{"t":"p","runs":[{"text":"With "},{"text":"mlm=True","code":true},{"text":" it masks 15% of tokens instead, for BERT-style training."}]},{"t":"h2","id":"instruction-tuning-mask-the-prompt","runs":[{"text":"Instruction tuning: mask the prompt"}]},{"t":"p","runs":[{"text":"For supervised fine-tuning on instruction data, you usually do not want loss on the prompt: only on the response. Otherwise the model spends its capacity learning to generate the questions people ask it."}]},{"t":"p","runs":[{"text":"The mechanism is setting prompt positions in "},{"text":"labels","code":true},{"text":" to "},{"text":"-100","code":true},{"text":":"}]},{"t":"code","lang":"python","label":"Python","text":"def prep(example):\n    prompt = tok.apply_chat_template(example[\"messages\"][:-1],\n                                     tokenize=False, add_generation_prompt=True)\n    full = tok.apply_chat_template(example[\"messages\"], tokenize=False)\n\n    p_ids = tok(prompt, add_special_tokens=False)[\"input_ids\"]\n    f_ids = tok(full, add_special_tokens=False)[\"input_ids\"]\n\n    labels = list(f_ids)\n    labels[:len(p_ids)] = [-100] * len(p_ids)\n    return {\"input_ids\": f_ids, \"labels\": labels,\n            \"attention_mask\": [1] * len(f_ids)}"},{"t":"p","runs":[{"text":"Note "},{"text":"add_special_tokens=False","code":true},{"text":" on both encodes: the chat template has already inserted the special tokens, and letting the tokenizer add a second beginning-of-sequence marker shifts every offset by one and breaks the masking silently. TRL’s "},{"text":"SFTTrainer","code":true},{"text":" does all of this for you, which is why the training module reaches for it; doing it once by hand is how you understand what it is doing."}]},{"t":"h2","id":"packing-for-pretraining","runs":[{"text":"Packing, for pretraining"}]},{"t":"p","runs":[{"text":"When training on plain text, padding wastes a lot. Packing concatenates documents and cuts fixed-length blocks:"}]},{"t":"code","lang":"python","label":"Python","text":"def group(batch, block=1024):\n    ids = sum(batch[\"input_ids\"], [])\n    n = (len(ids) // block) * block\n    chunks = [ids[i:i+block] for i in range(0, n, block)]\n    return {\"input_ids\": chunks, \"labels\": [c[:] for c in chunks]}\n\nlm_ds = tokenised.map(group, batched=True, remove_columns=tokenised.column_names)"},{"t":"p","runs":[{"text":"No padding at all, so every token in every batch is doing work. The cost is that documents bleed across block boundaries and attention crosses between unrelated texts. For pretraining this is standard and the effect is small; for instruction tuning it needs attention masking between packed examples, which the libraries handle and hand-rolled code usually does not."}]},{"t":"h2","id":"check-the-batch-before-you-train","runs":[{"text":"Check the batch before you train"}]},{"t":"p","runs":[{"text":"The cheapest hour you will ever save:"}]},{"t":"code","lang":"python","label":"Python","text":"batch = collator([ds[i] for i in range(4)])\nprint({k: v.shape for k, v in batch.items()})\nprint(tok.decode(batch[\"input_ids\"][0]))\nlab = batch[\"labels\"][0]\nprint(tok.decode([t for t in lab if t != -100]))"},{"t":"p","runs":[{"text":"Read that decoded text. It is exactly what the model will see. Three things show up here and nowhere else: a doubled beginning-of-sequence token, a chat template that did not render, and prompt masking that covered the wrong span. Each of those costs a full training run if you find it afterwards."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Print one collated batch for whatever you plan to train, decode both the input and the unmasked labels, and read them out loud. If they are not what you intended, you have just saved the cost of the run."}]}],"check":{"question":[{"text":"A fine-tune runs without error, the loss falls, and the resulting model generates plausible user questions rather than answering them. What was most likely wrong in data preparation?"}],"options":[{"text":[{"text":"Padding was applied at map time to a fixed maximum, so the model learned from padded positions."}],"correct":false,"tempting":[{"text":"Wasteful and slow, but padded positions are excluded from the loss by the collator."}]},{"text":[{"text":"The label column was named "},{"text":"label","code":true},{"text":" rather than "},{"text":"labels","code":true},{"text":", so no loss was computed on the responses."}],"correct":false,"tempting":[{"text":"Would prevent learning entirely rather than teaching the model a different task, and the loss would not fall."}]},{"text":[{"text":"The chat template was applied after tokenisation, so role markers were encoded as ordinary text."}],"correct":false,"tempting":[{"text":"Produces degraded formatting rather than a systematic role reversal."}]},{"text":[{"text":"Prompt tokens were not masked to "},{"text":"-100","code":true},{"text":", so the model was trained on the questions too."}],"correct":true,"tempting":null}],"order":[0,2,3,1],"why":null},"prev":{"slug":"data-quality-before-the-model","title":"Duplicates, leakage, balance, and label noise","minutes":10},"next":{"slug":"what-an-embedding-is","title":"A list of numbers where near means similar","minutes":9}}}