{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"padding-and-attention-masks","title":"Padding on the wrong side ruins generation quietly","url":"https://addaly.com/learn/hugging-face/padding-and-attention-masks","position":13,"of":75,"minutes":9,"answer":"The attention mask makes padding invisible to the model, but a decoder-only model generates from the last position.","keyIdea":"The attention mask makes padding invisible to the model, but a decoder-only model generates from the last position, so right padding makes it continue from a pad token and quietly degrades batched output while single requests still look fine."},"body":{"format":"markdown","text":"## Why padding exists at all\n\nA batch is a rectangle. Three sentences of 4, 11 and 6 tokens cannot be stacked into a tensor until they are the same length, so the short ones get filled with a pad token:\n\n```python\nenc = tok([\"hi\", \"a much longer sentence here\"], padding=True, return_tensors=\"pt\")\nprint(enc[\"input_ids\"])\nprint(enc[\"attention_mask\"])\n```\n\nThe mask is the important half. It is 1 for real tokens and 0 for padding, and the attention mechanism uses it to make padded positions contribute nothing. Pass `input_ids` without the mask and the model attends to the padding as if it were content, a mistake that is easy to make when people build tensors by hand and then wonder why batched results differ from one-at-a-time results.\n\n> Rule: whatever the tokenizer returns, pass all of it. `model(**enc)`, never `model(enc[\"input_ids\"])`.\n\n## The side matters, and only for generation\n\nFor an encoder model doing classification, padding on the right is fine. Every position is visible to every other, the mask zeroes the pad, the pooled output is correct.\n\nFor a decoder-only model doing generation, right padding is a silent quality bug. Generation continues from the **last position** of the sequence. If the sequence ends in padding, the model is being asked to continue from a pad token, having seen your prompt some distance back. Depending on the model and the mask handling, you get output that is degraded, generic, or occasionally empty. No error, no warning, and the same prompt run alone produces a good answer, so people conclude that batching breaks the model.\n\nThe fix is one line:\n\n```python\ntok.padding_side = \"left\"\n```\n\nNow every sequence ends with its real final token and generation starts from the right place. Do this for every `AutoModelForCausalLM` you batch. For training a causal model, right padding is correct, because labels are shifted and the loss ignores padded positions. The side depends on the operation, not on the model.\n\n## The missing pad token\n\nMany decoder-only checkpoints ship with no pad token, because generation one at a time never needs one. Batch them and you get:\n\n> ValueError: Asking to pad but the tokenizer does not have a padding token.\n\nThe standard fix:\n\n```python\nif tok.pad_token is None:\n    tok.pad_token = tok.eos_token\n```\n\nThis is fine for inference. For training it is worth thinking about once: with `pad_token == eos_token`, a naive label setup can teach the model that end-of-sequence is followed by more end-of-sequence, which makes it fail to stop. The usual answer is to set padded label positions to `-100`, which the loss ignores, and the data collators in the library do this for you.\n\n## Truncation, and the side of that too\n\n```python\nenc = tok(texts, padding=True, truncation=True, max_length=1024, return_tensors=\"pt\")\n```\n\n`padding=True` pads to the longest item in the batch. `padding=\"max_length\"` pads everything to `max_length`, which wastes compute but produces uniform shapes: occasionally required by compiled or exported models.\n\n`truncation_side` defaults to right. For a chat history, cutting from the right removes the most recent turn, which is the one that mattered. Set `tok.truncation_side = \"left\"` when the end of the input is the important part.\n\n## The cost of padding\n\nPadding is wasted computation. A batch of 32 where one item is 2,000 tokens and the rest are 50 runs as if all 32 were 2,000: roughly a fortyfold waste. Two cheap mitigations:\n\n1. **Sort by length before batching**, so each batch is internally uniform. This alone often doubles throughput on mixed-length data and costs about four lines.\n2. **Cap the batch by token count rather than by item count** (for example 8,000 tokens per batch) so a batch of long items is small and a batch of short items is large.\n\nServing stacks like vLLM and text-generation-inference remove the problem entirely with continuous batching, but the sorting trick works today in a plain script and needs no new dependency.\n\n## The five-line sanity check\n\nAfter any change to batching, run one input alone and the same input inside a batch of eight, and compare:\n\n```python\nsolo = pipe(\"What is the capital of Karnataka?\")\nbatched = pipe([\"filler\"] * 7 + [\"What is the capital of Karnataka?\"])[-1]\n```\n\nThey should agree. When they do not, the padding side is the first suspect, the missing mask the second.\n\n## Do this now\n\nTake any small instruct model, batch four prompts of very different lengths with the default right padding, and look at the shortest prompt's answer. Then set `padding_side = \"left\"` and run the identical batch. The difference is the bug this lesson exists for, and seeing it once makes it permanent."},"check":{"question":"A chat model gives good answers one request at a time and vague or empty ones when the same prompts are batched. What should be checked first?","options":[{"text":"Whether the attention mask is being passed, since without it the padded positions are attended to as if they were real content.","correct":false,"tempting":"A real bug, but it would also degrade batched encoder classification, and here the model generates from the wrong final position."},{"text":"Whether the batch exceeds the model's context window, causing silent truncation of the earlier prompts in the batch.","correct":false,"tempting":"Treats a batch as one long sequence; items in a batch each get their own context window."},{"text":"Whether sampling parameters differ between the two call paths, since batched calls often inherit different generation defaults.","correct":false,"tempting":"Sampling variance would produce different good answers, not systematically vague ones for short prompts."},{"text":"Whether `padding_side` is still the default right, so short prompts end in padding and generation continues from a pad token.","correct":true}]},"prev":{"slug":"tokenizers-in-practice","title":"Your text is integers, and the count is not the word count","url":"https://addaly.com/learn/hugging-face/tokenizers-in-practice"},"next":{"slug":"from-logits-to-text","title":"Logits, and the four knobs between them and words","url":"https://addaly.com/learn/hugging-face/from-logits-to-text"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Padding on the wrong side ruins generation quietly', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/padding-and-attention-masks (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Padding on the wrong side ruins generation quietly","courseTitle":"Hugging Face, End to End","module":{"index":2,"slug":"transformers-from-text-to-text","title":"Transformers, from characters to output"},"answer":null,"keyIdea":"The attention mask makes padding invisible to the model, but a decoder-only model generates from the last position, so right padding makes it continue from a pad token and quietly degrades batched output while single requests still look fine.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"why-padding-exists-at-all","runs":[{"text":"Why padding exists at all"}]},{"t":"p","runs":[{"text":"A batch is a rectangle. Three sentences of 4, 11 and 6 tokens cannot be stacked into a tensor until they are the same length, so the short ones get filled with a pad token:"}]},{"t":"code","lang":"python","label":"Python","text":"enc = tok([\"hi\", \"a much longer sentence here\"], padding=True, return_tensors=\"pt\")\nprint(enc[\"input_ids\"])\nprint(enc[\"attention_mask\"])"},{"t":"p","runs":[{"text":"The mask is the important half. It is 1 for real tokens and 0 for padding, and the attention mechanism uses it to make padded positions contribute nothing. Pass "},{"text":"input_ids","code":true},{"text":" without the mask and the model attends to the padding as if it were content, a mistake that is easy to make when people build tensors by hand and then wonder why batched results differ from one-at-a-time results."}]},{"t":"quote","runs":[{"text":"Rule: whatever the tokenizer returns, pass all of it. "},{"text":"model(**enc)","code":true},{"text":", never "},{"text":"model(enc[\"input_ids\"])","code":true},{"text":"."}]},{"t":"h2","id":"the-side-matters-and-only-for-generation","runs":[{"text":"The side matters, and only for generation"}]},{"t":"p","runs":[{"text":"For an encoder model doing classification, padding on the right is fine. Every position is visible to every other, the mask zeroes the pad, the pooled output is correct."}]},{"t":"p","runs":[{"text":"For a decoder-only model doing generation, right padding is a silent quality bug. Generation continues from the "},{"text":"last position","b":true},{"text":" of the sequence. If the sequence ends in padding, the model is being asked to continue from a pad token, having seen your prompt some distance back. Depending on the model and the mask handling, you get output that is degraded, generic, or occasionally empty. No error, no warning, and the same prompt run alone produces a good answer, so people conclude that batching breaks the model."}]},{"t":"p","runs":[{"text":"The fix is one line:"}]},{"t":"code","lang":"python","label":"Python","text":"tok.padding_side = \"left\""},{"t":"p","runs":[{"text":"Now every sequence ends with its real final token and generation starts from the right place. Do this for every "},{"text":"AutoModelForCausalLM","code":true},{"text":" you batch. For training a causal model, right padding is correct, because labels are shifted and the loss ignores padded positions. The side depends on the operation, not on the model."}]},{"t":"h2","id":"the-missing-pad-token","runs":[{"text":"The missing pad token"}]},{"t":"p","runs":[{"text":"Many decoder-only checkpoints ship with no pad token, because generation one at a time never needs one. Batch them and you get:"}]},{"t":"quote","runs":[{"text":"ValueError: Asking to pad but the tokenizer does not have a padding token."}]},{"t":"p","runs":[{"text":"The standard fix:"}]},{"t":"code","lang":"python","label":"Python","text":"if tok.pad_token is None:\n    tok.pad_token = tok.eos_token"},{"t":"p","runs":[{"text":"This is fine for inference. For training it is worth thinking about once: with "},{"text":"pad_token == eos_token","code":true},{"text":", a naive label setup can teach the model that end-of-sequence is followed by more end-of-sequence, which makes it fail to stop. The usual answer is to set padded label positions to "},{"text":"-100","code":true},{"text":", which the loss ignores, and the data collators in the library do this for you."}]},{"t":"h2","id":"truncation-and-the-side-of-that-too","runs":[{"text":"Truncation, and the side of that too"}]},{"t":"code","lang":"python","label":"Python","text":"enc = tok(texts, padding=True, truncation=True, max_length=1024, return_tensors=\"pt\")"},{"t":"p","runs":[{"text":"padding=True","code":true},{"text":" pads to the longest item in the batch. "},{"text":"padding=\"max_length\"","code":true},{"text":" pads everything to "},{"text":"max_length","code":true},{"text":", which wastes compute but produces uniform shapes: occasionally required by compiled or exported models."}]},{"t":"p","runs":[{"text":"truncation_side","code":true},{"text":" defaults to right. For a chat history, cutting from the right removes the most recent turn, which is the one that mattered. Set "},{"text":"tok.truncation_side = \"left\"","code":true},{"text":" when the end of the input is the important part."}]},{"t":"h2","id":"the-cost-of-padding","runs":[{"text":"The cost of padding"}]},{"t":"p","runs":[{"text":"Padding is wasted computation. A batch of 32 where one item is 2,000 tokens and the rest are 50 runs as if all 32 were 2,000: roughly a fortyfold waste. Two cheap mitigations:"}]},{"t":"ol","start":1,"items":[[{"text":"Sort by length before batching","b":true},{"text":", so each batch is internally uniform. This alone often doubles throughput on mixed-length data and costs about four lines."}],[{"text":"Cap the batch by token count rather than by item count","b":true},{"text":" (for example 8,000 tokens per batch) so a batch of long items is small and a batch of short items is large."}]]},{"t":"p","runs":[{"text":"Serving stacks like vLLM and text-generation-inference remove the problem entirely with continuous batching, but the sorting trick works today in a plain script and needs no new dependency."}]},{"t":"h2","id":"the-five-line-sanity-check","runs":[{"text":"The five-line sanity check"}]},{"t":"p","runs":[{"text":"After any change to batching, run one input alone and the same input inside a batch of eight, and compare:"}]},{"t":"code","lang":"python","label":"Python","text":"solo = pipe(\"What is the capital of Karnataka?\")\nbatched = pipe([\"filler\"] * 7 + [\"What is the capital of Karnataka?\"])[-1]"},{"t":"p","runs":[{"text":"They should agree. When they do not, the padding side is the first suspect, the missing mask the second."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Take any small instruct model, batch four prompts of very different lengths with the default right padding, and look at the shortest prompt’s answer. Then set "},{"text":"padding_side = \"left\"","code":true},{"text":" and run the identical batch. The difference is the bug this lesson exists for, and seeing it once makes it permanent."}]}],"check":{"question":[{"text":"A chat model gives good answers one request at a time and vague or empty ones when the same prompts are batched. What should be checked first?"}],"options":[{"text":[{"text":"Whether the attention mask is being passed, since without it the padded positions are attended to as if they were real content."}],"correct":false,"tempting":[{"text":"A real bug, but it would also degrade batched encoder classification, and here the model generates from the wrong final position."}]},{"text":[{"text":"Whether the batch exceeds the model’s context window, causing silent truncation of the earlier prompts in the batch."}],"correct":false,"tempting":[{"text":"Treats a batch as one long sequence; items in a batch each get their own context window."}]},{"text":[{"text":"Whether sampling parameters differ between the two call paths, since batched calls often inherit different generation defaults."}],"correct":false,"tempting":[{"text":"Sampling variance would produce different good answers, not systematically vague ones for short prompts."}]},{"text":[{"text":"Whether "},{"text":"padding_side","code":true},{"text":" is still the default right, so short prompts end in padding and generation continues from a pad token."}],"correct":true,"tempting":null}],"order":[1,3,0,2],"why":null},"prev":{"slug":"tokenizers-in-practice","title":"Your text is integers, and the count is not the word count","minutes":10},"next":{"slug":"from-logits-to-text","title":"Logits, and the four knobs between them and words","minutes":10}}}