{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"packing-and-long-context","title":"Packing, document boundaries and long context","url":"https://addaly.com/learn/fine-tuning/packing-and-long-context","position":35,"of":84,"minutes":9,"answer":"Packing removes padding but lets one document attend to the previous one unless the block-diagonal mask and per-document position ids are actually in the batch. And extending context requires both a RoPE scaling change and.","keyIdea":"Packing removes padding but lets one document attend to the previous one unless the block-diagonal mask and per-document position ids are actually in the batch. And extending context requires both a RoPE scaling change and long training data that, for most tasks, does not naturally exist."},"body":{"format":"markdown","text":"## The padding problem\n\nIf your examples average 300 tokens and your training window is 2,048, then padding every sequence to the window means roughly 85% of the tensor is padding. You pay for those matrix multiplications in full.\n\nTwo fixes, with different risk profiles.\n\n**Length-grouped batching** (`group_by_length=True`) sorts similar-length examples together so each batch pads to its own longest member rather than to a global maximum. Safe, simple, and it recovers most of the waste.\n\n**Packing** concatenates examples end to end until the window is full. Zero padding, maximum throughput, and one hazard that is invisible unless you look for it.\n\n## Attention across the seam\n\nIn a packed sequence, example B follows example A in the same tensor. With ordinary causal attention, the tokens of B can attend to every token of A. The model is being asked to predict the start of one customer's ticket having read a different customer's ticket, and it learns a transition that will never occur at inference.\n\nThe consequences are subtle rather than catastrophic (slightly worse quality, a model that sometimes continues past the end of an answer) and they never show up in the loss curve.\n\nThe correct handling has two parts:\n\n1. **A block-diagonal attention mask**, so each document attends only within itself. Implemented efficiently through FlashAttention's variable-length path, which takes cumulative sequence-length offsets rather than a mask tensor.\n2. **Position ids reset per document**, so the second document's first token is at position 0 rather than position 400. Without this, an example's position encoding depends on what happened to be packed before it.\n\nRecent TRL versions handle both when packing is enabled, but implementations have differed over time and across libraries. Verify rather than assume:\n\n```python\nbatch = next(iter(trainer.get_train_dataloader()))\nprint(batch.keys())                 # look for position_ids or cu_seq_lens\nprint(batch[\"position_ids\"][0][:50] if \"position_ids\" in batch else \"none\")\n```\n\nIf position ids climb monotonically across the whole window, documents are not being separated. If you cannot confirm it, use length-grouped batching instead: you lose some throughput and you lose the failure mode entirely.\n\nOne cheap mitigation regardless: make sure an end-of-turn or end-of-sequence token sits between packed documents, so there is at least a learned signal that one thing has finished.\n\n## Extending a model's context\n\nA model pretrained at 8,192 tokens does not simply work at 32,768. Its rotary position embeddings encode positions it has never seen, and quality falls off a cliff past the trained length.\n\nContext extension fine-tuning changes the position encoding and then trains briefly at the longer length. The usual approaches all adjust RoPE's frequency base:\n\n```python\nconfig.rope_scaling = {\"type\": \"yarn\", \"factor\": 4.0,\n                       \"original_max_position_embeddings\": 8192}\n```\n\nLinear interpolation is the simplest and loses the most local resolution; NTK-aware scaling and YaRN adjust different frequency bands differently and preserve short-range behaviour better. All of them want a short fine-tune on long sequences afterwards: typically a few hundred steps, which is cheap in steps and expensive in memory, because attention is quadratic and you are now at four times the length.\n\n## The data problem nobody solves well\n\nHere is the honest difficulty: long-context supervised data barely exists. Real conversations are short. Real tickets are short. If 95% of your examples are under 1,000 tokens and you train with a 32,768 window, you have paid for a long context and taught the model nothing about using one.\n\nThe common synthetic route is to build multi-document tasks: concatenate several related documents and ask a question whose answer requires one of them, or several. It works, and it has a well-documented failure. If all your long-context training is of the \"find the fact in the haystack\" kind, the model becomes good at retrieval within its context and no better at *synthesising* across it: summarising a long document, tracking an entity through a transcript, reconciling two sources that disagree.\n\nSo build both kinds, and evaluate both kinds. And before any of it, check the sequence-length histogram of your data against the window you are about to pay for. If the histogram says you do not need the window, do not buy it."},"check":{"question":"A team enables packing with an implementation that concatenates examples but does not reset position ids or mask across boundaries. What is the most likely symptom in the trained model?","options":[{"text":"Training loss rises sharply, because the model is penalised for predicting tokens of one document from another.","correct":false,"tempting":"Expects a visible loss signal; the model can partially fit the spurious transitions, so the curve looks normal, which is what makes the bug dangerous."},{"text":"The model's maximum usable context shrinks, because position ids beyond the first document were never seen at their true values.","correct":false,"tempting":"Inverts the effect: position ids climb past normal values in this setup, so the model sees more of the position range rather than less."},{"text":"The model sometimes fails to stop and continues into unrelated content, with no sign of it in the loss.","correct":true},{"text":"Gradients from the second document are discarded, so only the first example in each packed window contributes to training.","correct":false,"tempting":"Assumes an exclusion the mechanism does not perform; every unmasked position contributes to the loss regardless of which document it belongs to."}]},"prev":{"slug":"data-mixtures-and-replay","title":"Mixtures and replay","url":"https://addaly.com/learn/fine-tuning/data-mixtures-and-replay"},"next":{"slug":"multi-turn-and-tool-calls","title":"Multi-turn conversations and tool calls","url":"https://addaly.com/learn/fine-tuning/multi-turn-and-tool-calls"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Packing, document boundaries and long context', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/packing-and-long-context (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Packing, document boundaries and long context","courseTitle":"Fine-Tuning, and When Not To","module":{"index":4,"slug":"building-the-dataset","title":"Building the dataset"},"answer":null,"keyIdea":"Packing removes padding but lets one document attend to the previous one unless the block-diagonal mask and per-document position ids are actually in the batch. And extending context requires both a RoPE scaling change and long training data that, for most tasks, does not naturally exist.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-padding-problem","runs":[{"text":"The padding problem"}]},{"t":"p","runs":[{"text":"If your examples average 300 tokens and your training window is 2,048, then padding every sequence to the window means roughly 85% of the tensor is padding. You pay for those matrix multiplications in full."}]},{"t":"p","runs":[{"text":"Two fixes, with different risk profiles."}]},{"t":"p","runs":[{"text":"Length-grouped batching","b":true},{"text":" ("},{"text":"group_by_length=True","code":true},{"text":") sorts similar-length examples together so each batch pads to its own longest member rather than to a global maximum. Safe, simple, and it recovers most of the waste."}]},{"t":"p","runs":[{"text":"Packing","b":true},{"text":" concatenates examples end to end until the window is full. Zero padding, maximum throughput, and one hazard that is invisible unless you look for it."}]},{"t":"h2","id":"attention-across-the-seam","runs":[{"text":"Attention across the seam"}]},{"t":"p","runs":[{"text":"In a packed sequence, example B follows example A in the same tensor. With ordinary causal attention, the tokens of B can attend to every token of A. The model is being asked to predict the start of one customer’s ticket having read a different customer’s ticket, and it learns a transition that will never occur at inference."}]},{"t":"p","runs":[{"text":"The consequences are subtle rather than catastrophic (slightly worse quality, a model that sometimes continues past the end of an answer) and they never show up in the loss curve."}]},{"t":"p","runs":[{"text":"The correct handling has two parts:"}]},{"t":"ol","start":1,"items":[[{"text":"A block-diagonal attention mask","b":true},{"text":", so each document attends only within itself. Implemented efficiently through FlashAttention’s variable-length path, which takes cumulative sequence-length offsets rather than a mask tensor."}],[{"text":"Position ids reset per document","b":true},{"text":", so the second document’s first token is at position 0 rather than position 400. Without this, an example’s position encoding depends on what happened to be packed before it."}]]},{"t":"p","runs":[{"text":"Recent TRL versions handle both when packing is enabled, but implementations have differed over time and across libraries. Verify rather than assume:"}]},{"t":"code","lang":"python","label":"Python","text":"batch = next(iter(trainer.get_train_dataloader()))\nprint(batch.keys())                 # look for position_ids or cu_seq_lens\nprint(batch[\"position_ids\"][0][:50] if \"position_ids\" in batch else \"none\")"},{"t":"p","runs":[{"text":"If position ids climb monotonically across the whole window, documents are not being separated. If you cannot confirm it, use length-grouped batching instead: you lose some throughput and you lose the failure mode entirely."}]},{"t":"p","runs":[{"text":"One cheap mitigation regardless: make sure an end-of-turn or end-of-sequence token sits between packed documents, so there is at least a learned signal that one thing has finished."}]},{"t":"h2","id":"extending-a-model-s-context","runs":[{"text":"Extending a model’s context"}]},{"t":"p","runs":[{"text":"A model pretrained at 8,192 tokens does not simply work at 32,768. Its rotary position embeddings encode positions it has never seen, and quality falls off a cliff past the trained length."}]},{"t":"p","runs":[{"text":"Context extension fine-tuning changes the position encoding and then trains briefly at the longer length. The usual approaches all adjust RoPE’s frequency base:"}]},{"t":"code","lang":"python","label":"Python","text":"config.rope_scaling = {\"type\": \"yarn\", \"factor\": 4.0,\n                       \"original_max_position_embeddings\": 8192}"},{"t":"p","runs":[{"text":"Linear interpolation is the simplest and loses the most local resolution; NTK-aware scaling and YaRN adjust different frequency bands differently and preserve short-range behaviour better. All of them want a short fine-tune on long sequences afterwards: typically a few hundred steps, which is cheap in steps and expensive in memory, because attention is quadratic and you are now at four times the length."}]},{"t":"h2","id":"the-data-problem-nobody-solves-well","runs":[{"text":"The data problem nobody solves well"}]},{"t":"p","runs":[{"text":"Here is the honest difficulty: long-context supervised data barely exists. Real conversations are short. Real tickets are short. If 95% of your examples are under 1,000 tokens and you train with a 32,768 window, you have paid for a long context and taught the model nothing about using one."}]},{"t":"p","runs":[{"text":"The common synthetic route is to build multi-document tasks: concatenate several related documents and ask a question whose answer requires one of them, or several. It works, and it has a well-documented failure. If all your long-context training is of the “find the fact in the haystack” kind, the model becomes good at retrieval within its context and no better at "},{"text":"synthesising","i":true},{"text":" across it: summarising a long document, tracking an entity through a transcript, reconciling two sources that disagree."}]},{"t":"p","runs":[{"text":"So build both kinds, and evaluate both kinds. And before any of it, check the sequence-length histogram of your data against the window you are about to pay for. If the histogram says you do not need the window, do not buy it."}]}],"check":{"question":[{"text":"A team enables packing with an implementation that concatenates examples but does not reset position ids or mask across boundaries. What is the most likely symptom in the trained model?"}],"options":[{"text":[{"text":"Training loss rises sharply, because the model is penalised for predicting tokens of one document from another."}],"correct":false,"tempting":[{"text":"Expects a visible loss signal; the model can partially fit the spurious transitions, so the curve looks normal, which is what makes the bug dangerous."}]},{"text":[{"text":"The model’s maximum usable context shrinks, because position ids beyond the first document were never seen at their true values."}],"correct":false,"tempting":[{"text":"Inverts the effect: position ids climb past normal values in this setup, so the model sees more of the position range rather than less."}]},{"text":[{"text":"The model sometimes fails to stop and continues into unrelated content, with no sign of it in the loss."}],"correct":true,"tempting":null},{"text":[{"text":"Gradients from the second document are discarded, so only the first example in each packed window contributes to training."}],"correct":false,"tempting":[{"text":"Assumes an exclusion the mechanism does not perform; every unmasked position contributes to the loss regardless of which document it belongs to."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"data-mixtures-and-replay","title":"Mixtures and replay","minutes":9},"next":{"slug":"multi-turn-and-tool-calls","title":"Multi-turn conversations and tool calls","minutes":9}}}