{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"batch-size-and-accumulation","title":"Batch size, accumulation and the token count","url":"https://addaly.com/learn/fine-tuning/batch-size-and-accumulation","position":14,"of":84,"minutes":9,"answer":"Effective batch is per-device times accumulation times devices, the unit that actually governs gradient noise is tokens per step rather than examples.","keyIdea":"Effective batch is per-device times accumulation times devices, the unit that actually governs gradient noise is tokens per step rather than examples, and loss must be normalised by real token counts across an accumulation window or long and short examples get equal weight."},"body":{"format":"markdown","text":"## Three numbers that multiply\n\nWhat the optimiser sees is the **effective batch size**:\n\n```\neffective = per_device_batch_size × gradient_accumulation_steps × num_devices\n```\n\nA config with `per_device_train_batch_size=2`, `gradient_accumulation_steps=8` on one GPU has an effective batch of 16. The gradients from eight forward-backward passes are summed before a single optimiser step. Memory is paid for two sequences at a time; the update behaves like sixteen.\n\nThis is the standard trick for training on a small card, and it is close to free: the only cost is that you compute eight forward passes before making any progress, so wall-clock time per optimiser step goes up while total compute stays roughly the same.\n\n## The number that actually matters is tokens\n\nExamples are a bad unit. An effective batch of 16 chat exchanges could be 2,000 tokens or 30,000 depending on how long they are. Gradient noise, and therefore the right learning rate, tracks tokens far better than examples.\n\nLog tokens per optimiser step. Most training frameworks do not print it, and it takes one line to compute. A typical small fine-tune sits somewhere between 8,000 and 64,000 tokens per step; if yours is 900, your gradients are extremely noisy and no learning rate will make the run stable.\n\n## The accumulation bug that shipped for years\n\nThis is the detail no tutorial mentions, and it caught a great many people.\n\nIf each micro-batch computes a *mean* loss over its own tokens, and you then average those means across accumulation steps, you have not reproduced a larger batch: you have given every micro-batch equal weight regardless of how many real tokens it contained. A micro-batch with 40 unmasked tokens and one with 900 count the same. Since masked instruction data has extremely variable completion lengths, the discrepancy is real.\n\nHugging Face's `Trainer` and TRL had exactly this behaviour, and it was fixed in late 2024 (transformers 4.46 and the corresponding TRL release) by normalising over the total number of unmasked tokens in the accumulation window. Two practical consequences:\n\n- Configs and results from before that fix are not directly comparable to results after it, especially where `gradient_accumulation_steps` was large.\n- If you write your own training loop, normalise by the total token count across the accumulation window, not by the number of micro-batches. It is three lines and it is correct:\n\n```python\ntotal_tokens = sum(batch_token_counts)\nfor batch, n in zip(batches, batch_token_counts):\n    loss = model(**batch).loss * n / total_tokens   # weight by real tokens\n    loss.backward()\noptimizer.step()\n```\n\n## Packing: the other way to fix the token count\n\nRather than padding every sequence to the length of the longest in the batch (which on instruction data can mean half your compute is spent on padding tokens) packing concatenates short examples until the buffer is full.\n\nThe gain is substantial: on a dataset whose examples average 300 tokens with a 2,048-token window, packing can cut training time by more than half, because you stop multiplying matrices by padding.\n\nThe hazard is attention leaking across the boundary. Without a correct block-diagonal attention mask, the model reads the end of example A while predicting the start of example B, and learns a transition that does not exist. Modern implementations handle this with position ids reset per document and a FlashAttention variable-length path; `packing=True` in TRL with a recent version does the right thing, but this is worth verifying rather than assuming, because the failure is invisible in the loss curve.\n\nIf you cannot verify it, use **length-grouped batching** instead (`group_by_length=True`). It sorts similar-length examples into the same batch so padding is minimal, with no cross-contamination risk at all. Most of the speed, none of the subtlety.\n\n## Choosing the numbers in practice\n\nA workable procedure on a single card:\n\n1. Set `per_device_train_batch_size` to the largest value that does not run out of memory at your maximum sequence length. Find it by bisection in about four attempts.\n2. Set `gradient_accumulation_steps` so that effective batch lands between 16 and 64 for a LoRA on instruction data.\n3. Log tokens per step. If it is under 4,000, raise accumulation rather than the learning rate.\n4. Switch on gradient checkpointing if step 1 forces you below a per-device batch of 1 at the sequence length you need. It costs roughly 30% more time per step and often lets you double the batch.\n\nAnd one thing not to do: do not chase a large effective batch on a small dataset. With 800 examples and an effective batch of 64, one epoch is twelve optimiser steps. Twelve steps will not move an adapter anywhere useful, no matter how clean each gradient is."},"check":{"question":"A custom training loop averages each micro-batch's mean loss across 8 accumulation steps. The data is instruction-tuned with completions ranging from 20 to 1,200 tokens. What does this do to the gradient, compared with a true batch of the same size?","options":[{"text":"Nothing; averaging means of equal-sized micro-batches is mathematically identical to a single larger batch.","correct":false,"tempting":"Holds only when every micro-batch contains the same number of unmasked tokens, which masked instruction data specifically does not."},{"text":"It makes the run non-deterministic, because floating-point addition order varies across accumulation steps.","correct":false,"tempting":"Names a real but tiny effect, unrelated to the systematic weighting error produced by variable token counts."},{"text":"It doubles the effective learning rate, since summing eight gradients without dividing inflates the step.","correct":false,"tempting":"Describes a failure to average at all; the loop does average, so the magnitude is roughly right and the weighting is what is wrong."},{"text":"It over-weights short completions, since a micro-batch's contribution ignores its token count.","correct":true}]},"prev":{"slug":"learning-rate-and-schedules","title":"Learning rate, warmup and the schedule","url":"https://addaly.com/learn/fine-tuning/learning-rate-and-schedules"},"next":{"slug":"precision-and-mixed-precision","title":"Precision, and why bf16 ended a class of bugs","url":"https://addaly.com/learn/fine-tuning/precision-and-mixed-precision"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Batch size, accumulation and the token count', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/batch-size-and-accumulation (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Batch size, accumulation and the token count","courseTitle":"Fine-Tuning, and When Not To","module":{"index":2,"slug":"how-training-works","title":"How training actually works"},"answer":null,"keyIdea":"Effective batch is per-device times accumulation times devices, the unit that actually governs gradient noise is tokens per step rather than examples, and loss must be normalised by real token counts across an accumulation window or long and short examples get equal weight.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-numbers-that-multiply","runs":[{"text":"Three numbers that multiply"}]},{"t":"p","runs":[{"text":"What the optimiser sees is the "},{"text":"effective batch size","b":true},{"text":":"}]},{"t":"code","lang":"","label":"Text","text":"effective = per_device_batch_size × gradient_accumulation_steps × num_devices"},{"t":"p","runs":[{"text":"A config with "},{"text":"per_device_train_batch_size=2","code":true},{"text":", "},{"text":"gradient_accumulation_steps=8","code":true},{"text":" on one GPU has an effective batch of 16. The gradients from eight forward-backward passes are summed before a single optimiser step. Memory is paid for two sequences at a time; the update behaves like sixteen."}]},{"t":"p","runs":[{"text":"This is the standard trick for training on a small card, and it is close to free: the only cost is that you compute eight forward passes before making any progress, so wall-clock time per optimiser step goes up while total compute stays roughly the same."}]},{"t":"h2","id":"the-number-that-actually-matters-is-tokens","runs":[{"text":"The number that actually matters is tokens"}]},{"t":"p","runs":[{"text":"Examples are a bad unit. An effective batch of 16 chat exchanges could be 2,000 tokens or 30,000 depending on how long they are. Gradient noise, and therefore the right learning rate, tracks tokens far better than examples."}]},{"t":"p","runs":[{"text":"Log tokens per optimiser step. Most training frameworks do not print it, and it takes one line to compute. A typical small fine-tune sits somewhere between 8,000 and 64,000 tokens per step; if yours is 900, your gradients are extremely noisy and no learning rate will make the run stable."}]},{"t":"h2","id":"the-accumulation-bug-that-shipped-for-years","runs":[{"text":"The accumulation bug that shipped for years"}]},{"t":"p","runs":[{"text":"This is the detail no tutorial mentions, and it caught a great many people."}]},{"t":"p","runs":[{"text":"If each micro-batch computes a "},{"text":"mean","i":true},{"text":" loss over its own tokens, and you then average those means across accumulation steps, you have not reproduced a larger batch: you have given every micro-batch equal weight regardless of how many real tokens it contained. A micro-batch with 40 unmasked tokens and one with 900 count the same. Since masked instruction data has extremely variable completion lengths, the discrepancy is real."}]},{"t":"p","runs":[{"text":"Hugging Face’s "},{"text":"Trainer","code":true},{"text":" and TRL had exactly this behaviour, and it was fixed in late 2024 (transformers 4.46 and the corresponding TRL release) by normalising over the total number of unmasked tokens in the accumulation window. Two practical consequences:"}]},{"t":"ul","items":[[{"text":"Configs and results from before that fix are not directly comparable to results after it, especially where "},{"text":"gradient_accumulation_steps","code":true},{"text":" was large."}],[{"text":"If you write your own training loop, normalise by the total token count across the accumulation window, not by the number of micro-batches. It is three lines and it is correct:"}]]},{"t":"code","lang":"python","label":"Python","text":"total_tokens = sum(batch_token_counts)\nfor batch, n in zip(batches, batch_token_counts):\n    loss = model(**batch).loss * n / total_tokens   # weight by real tokens\n    loss.backward()\noptimizer.step()"},{"t":"h2","id":"packing-the-other-way-to-fix-the-token-count","runs":[{"text":"Packing: the other way to fix the token count"}]},{"t":"p","runs":[{"text":"Rather than padding every sequence to the length of the longest in the batch (which on instruction data can mean half your compute is spent on padding tokens) packing concatenates short examples until the buffer is full."}]},{"t":"p","runs":[{"text":"The gain is substantial: on a dataset whose examples average 300 tokens with a 2,048-token window, packing can cut training time by more than half, because you stop multiplying matrices by padding."}]},{"t":"p","runs":[{"text":"The hazard is attention leaking across the boundary. Without a correct block-diagonal attention mask, the model reads the end of example A while predicting the start of example B, and learns a transition that does not exist. Modern implementations handle this with position ids reset per document and a FlashAttention variable-length path; "},{"text":"packing=True","code":true},{"text":" in TRL with a recent version does the right thing, but this is worth verifying rather than assuming, because the failure is invisible in the loss curve."}]},{"t":"p","runs":[{"text":"If you cannot verify it, use "},{"text":"length-grouped batching","b":true},{"text":" instead ("},{"text":"group_by_length=True","code":true},{"text":"). It sorts similar-length examples into the same batch so padding is minimal, with no cross-contamination risk at all. Most of the speed, none of the subtlety."}]},{"t":"h2","id":"choosing-the-numbers-in-practice","runs":[{"text":"Choosing the numbers in practice"}]},{"t":"p","runs":[{"text":"A workable procedure on a single card:"}]},{"t":"ol","start":1,"items":[[{"text":"Set "},{"text":"per_device_train_batch_size","code":true},{"text":" to the largest value that does not run out of memory at your maximum sequence length. Find it by bisection in about four attempts."}],[{"text":"Set "},{"text":"gradient_accumulation_steps","code":true},{"text":" so that effective batch lands between 16 and 64 for a LoRA on instruction data."}],[{"text":"Log tokens per step. If it is under 4,000, raise accumulation rather than the learning rate."}],[{"text":"Switch on gradient checkpointing if step 1 forces you below a per-device batch of 1 at the sequence length you need. It costs roughly 30% more time per step and often lets you double the batch."}]]},{"t":"p","runs":[{"text":"And one thing not to do: do not chase a large effective batch on a small dataset. With 800 examples and an effective batch of 64, one epoch is twelve optimiser steps. Twelve steps will not move an adapter anywhere useful, no matter how clean each gradient is."}]}],"check":{"question":[{"text":"A custom training loop averages each micro-batch’s mean loss across 8 accumulation steps. The data is instruction-tuned with completions ranging from 20 to 1,200 tokens. What does this do to the gradient, compared with a true batch of the same size?"}],"options":[{"text":[{"text":"Nothing; averaging means of equal-sized micro-batches is mathematically identical to a single larger batch."}],"correct":false,"tempting":[{"text":"Holds only when every micro-batch contains the same number of unmasked tokens, which masked instruction data specifically does not."}]},{"text":[{"text":"It makes the run non-deterministic, because floating-point addition order varies across accumulation steps."}],"correct":false,"tempting":[{"text":"Names a real but tiny effect, unrelated to the systematic weighting error produced by variable token counts."}]},{"text":[{"text":"It doubles the effective learning rate, since summing eight gradients without dividing inflates the step."}],"correct":false,"tempting":[{"text":"Describes a failure to average at all; the loop does average, so the magnitude is roughly right and the weighting is what is wrong."}]},{"text":[{"text":"It over-weights short completions, since a micro-batch’s contribution ignores its token count."}],"correct":true,"tempting":null}],"order":[2,3,1,0],"why":null},"prev":{"slug":"learning-rate-and-schedules","title":"Learning rate, warmup and the schedule","minutes":9},"next":{"slug":"precision-and-mixed-precision","title":"Precision, and why bf16 ended a class of bugs","minutes":9}}}