{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"loads-but-outputs-rubbish","title":"It loaded, it ran, the output is nonsense","url":"https://addaly.com/learn/hugging-face/loads-but-outputs-rubbish","position":18,"of":75,"minutes":9,"answer":"Fluent-but-wrong output has a short list of causes to check in order (base instead of instruct, missing chat template, mismatched tokenizer, padding side, dtype overflow, over-aggressive quantization.","keyIdea":"Fluent-but-wrong output has a short list of causes to check in order (base instead of instruct, missing chat template, mismatched tokenizer, padding side, dtype overflow, over-aggressive quantization, inherited generation defaults and silent truncation) and the model itself is the last suspect, not the first."},"body":{"format":"markdown","text":"## The worst category of bug\n\nNo traceback. No warning you noticed. The model produced fluent, confident, wrong text, or a classifier returned labels that look plausible and are random. Everything downstream works perfectly on garbage.\n\nThis lesson is a checklist in order of how often each cause is the cause. Work down it rather than guessing; each step is under a minute.\n\n## 1. Is it an instruct model?\n\n```python\nprint(\"chat_template:\", tok.chat_template is not None)\n```\n\n`False` on a model you are prompting conversationally means you have a base checkpoint. A base model continues text; asked \"What is the capital of Bihar?\" it may reply with three more quiz questions. Nothing is broken. Get the `-Instruct` variant.\n\n> Figure: The order to check, not the order to guess. Steps: 1. Instruct? (Is chat_template set); 2. Templated? (Not typed by hand); 3. Same repo? (Tokenizer and weights); 4. Left padding? (Only if batched); 5. Model last (Nine checks first). Each step is under a minute and they are ordered by how often each is the cause. The model itself is the tenth suspect, not the first, because fluent-but-wrong output is far more often a configuration mismatch than a bad checkpoint.\n\n## 2. Did you apply the template?\n\nCovered at length earlier, and it stays at number two because it stays that common. If you built the prompt with f-strings and role labels, stop and use `apply_chat_template`.\n\n## 3. Is the tokenizer from the same repository as the weights?\n\n```python\nprint(len(tok), model.config.vocab_size)\n```\n\nThese should match, or differ only by a handful of added tokens. A serious mismatch means the integers mean different things to the two halves, which produces exactly the fluent-nonsense signature. This is the classic failure when loading an adapter against the wrong base.\n\n## 4. Are the padding side and mask right?\n\nIf single calls are good and batched calls are bad, this is it. `tok.padding_side = \"left\"` for decoder-only generation, and pass the mask.\n\n## 5. What dtype is it actually in?\n\n```python\nprint(next(model.parameters()).dtype)\n```\n\nTwo real failures here:\n\n- **float16 overflow.** Some models, notably the T5 family and a few older checkpoints, were trained in bfloat16 and overflow in float16: you get `nan` in the logits and empty or garbage output. `bfloat16` has the same exponent range as float32 and does not; use it when the GPU supports it.\n- **float16 on CPU.** Slow, and on some builds numerically wrong. Use float32 on CPU unless you have measured otherwise.\n\nA quick test: if `torch.isnan(out.logits).any()` is true, you have found it.\n\n## 6. Is the quantization too aggressive?\n\n4-bit is fine for chat and noticeably worse for arithmetic, code and long structured output. If a model is good at fp16 and bad at 4-bit on your task, that is the quantization, not a bug. Step up to 8-bit or a `Q5_K_M` GGUF and re-check before blaming anything else.\n\n## 7. Are the generation parameters sane?\n\n```python\nprint(model.generation_config)\n```\n\nLook for `temperature` above 1.2 with no top-p, a `repetition_penalty` above about 1.3 (which starts suppressing ordinary words), and `do_sample: false` ignoring everything you passed.\n\n## 8. Did the input get truncated?\n\n```python\nprint(enc[\"input_ids\"].shape[-1], model.config.max_position_embeddings)\n```\n\nSilent truncation removes the end of a long document, usually the part that mattered. For a classifier reading complaints, this alone can halve accuracy.\n\n## 9. For classifiers: what does `id2label` say?\n\n`{0: 'LABEL_0', 1: 'LABEL_1'}` means nobody told the model which is which, and a coin flip on polarity is a 50% chance you have it backwards across your whole dataset.\n\n## 10. Only now, suspect the model\n\nAfter nine checks, it is reasonable to conclude the model is unsuited. Confirm it cheaply: run the same ten inputs through a well-known model of similar size. If that one is also poor, the task is harder than you thought or your inputs are unusual: both useful findings.\n\n## The minimal reproduction\n\nWhen you report this to a discussion thread or a colleague, include: model id **with revision**, transformers and torch versions, the exact rendered prompt (`tokenize=False`), the generation kwargs, the dtype, and the device. Half the questions on the Hub cannot be answered because the rendered prompt is missing, and the rendered prompt is the answer about a third of the time.\n\n## Do this now\n\nDeliberately break one working script four ways: drop the template, right-pad a batch, force float16 on a T5, and load the base instead of the instruct checkpoint. Look at each failure. You are building a library of signatures, and recognising a signature turns an afternoon into a minute."},"check":{"question":"A T5-family summariser returns empty strings on a GPU under float16 but works on CPU under float32. What is happening?","options":[{"text":"The GPU kernels for encoder-decoder attention require float32, so the half-precision path falls back to an untested code branch.","correct":false,"tempting":"Invents a kernel restriction; half precision is well supported for these architectures."},{"text":"Activations exceed float16's range, producing NaN logits, so decoding immediately hits the end token and emits nothing.","correct":true},{"text":"The model's weights were saved in bfloat16, and converting them to float16 rounds small values to zero, erasing the decoder's embeddings.","correct":false,"tempting":"Names the right precision pair but the wrong mechanism: the problem is exponent range and overflow to NaN, not underflow of small weights."},{"text":"CPU and GPU use different tokenizer implementations, so the device change altered the input ids rather than the arithmetic.","correct":false,"tempting":"Tokenization is device-independent and happens before anything reaches the accelerator."}]},"prev":{"slug":"streaming-and-stopping","title":"Showing tokens as they arrive, and making them stop","url":"https://addaly.com/learn/hugging-face/streaming-and-stopping"},"next":{"slug":"running-it-locally","title":"Your disk fills up before your patience does","url":"https://addaly.com/learn/hugging-face/running-it-locally"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'It loaded, it ran, the output is nonsense', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/loads-but-outputs-rubbish (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"It loaded, it ran, the output is nonsense","courseTitle":"Hugging Face, End to End","module":{"index":2,"slug":"transformers-from-text-to-text","title":"Transformers, from characters to output"},"answer":null,"keyIdea":"Fluent-but-wrong output has a short list of causes to check in order (base instead of instruct, missing chat template, mismatched tokenizer, padding side, dtype overflow, over-aggressive quantization, inherited generation defaults and silent truncation) and the model itself is the last suspect, not the first.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"flow","steps":[{"label":"Instruct?","detail":"Is chat_template set"},{"label":"Templated?","detail":"Not typed by hand"},{"label":"Same repo?","detail":"Tokenizer and weights"},{"label":"Left padding?","detail":"Only if batched"},{"label":"Model last","detail":"Nine checks first"}],"title":"The order to check, not the order to guess","caption":"Each step is under a minute and they are ordered by how often each is the cause. The model itself is the tenth suspect, not the first, because fluent-but-wrong output is far more often a configuration mismatch than a bad checkpoint."},"alt":"Figure: The order to check, not the order to guess. Steps: 1. Instruct? (Is chat_template set); 2. Templated? (Not typed by hand); 3. Same repo? (Tokenizer and weights); 4. Left padding? (Only if batched); 5. Model last (Nine checks first). Each step is under a minute and they are ordered by how often each is the cause. The model itself is the tenth suspect, not the first, because fluent-but-wrong output is far more often a configuration mismatch than a bad checkpoint."}},"blocks":[{"t":"h2","id":"the-worst-category-of-bug","runs":[{"text":"The worst category of bug"}]},{"t":"p","runs":[{"text":"No traceback. No warning you noticed. The model produced fluent, confident, wrong text, or a classifier returned labels that look plausible and are random. Everything downstream works perfectly on garbage."}]},{"t":"p","runs":[{"text":"This lesson is a checklist in order of how often each cause is the cause. Work down it rather than guessing; each step is under a minute."}]},{"t":"h2","id":"1-is-it-an-instruct-model","runs":[{"text":"1. Is it an instruct model?"}]},{"t":"code","lang":"python","label":"Python","text":"print(\"chat_template:\", tok.chat_template is not None)"},{"t":"p","runs":[{"text":"False","code":true},{"text":" on a model you are prompting conversationally means you have a base checkpoint. A base model continues text; asked “What is the capital of Bihar?” it may reply with three more quiz questions. Nothing is broken. Get the "},{"text":"-Instruct","code":true},{"text":" variant."}]},{"t":"h2","id":"2-did-you-apply-the-template","runs":[{"text":"2. Did you apply the template?"}]},{"t":"p","runs":[{"text":"Covered at length earlier, and it stays at number two because it stays that common. If you built the prompt with f-strings and role labels, stop and use "},{"text":"apply_chat_template","code":true},{"text":"."}]},{"t":"h2","id":"3-is-the-tokenizer-from-the-same-repository-as-the-weights","runs":[{"text":"3. Is the tokenizer from the same repository as the weights?"}]},{"t":"code","lang":"python","label":"Python","text":"print(len(tok), model.config.vocab_size)"},{"t":"p","runs":[{"text":"These should match, or differ only by a handful of added tokens. A serious mismatch means the integers mean different things to the two halves, which produces exactly the fluent-nonsense signature. This is the classic failure when loading an adapter against the wrong base."}]},{"t":"h2","id":"4-are-the-padding-side-and-mask-right","runs":[{"text":"4. Are the padding side and mask right?"}]},{"t":"p","runs":[{"text":"If single calls are good and batched calls are bad, this is it. "},{"text":"tok.padding_side = \"left\"","code":true},{"text":" for decoder-only generation, and pass the mask."}]},{"t":"h2","id":"5-what-dtype-is-it-actually-in","runs":[{"text":"5. What dtype is it actually in?"}]},{"t":"code","lang":"python","label":"Python","text":"print(next(model.parameters()).dtype)"},{"t":"p","runs":[{"text":"Two real failures here:"}]},{"t":"ul","items":[[{"text":"float16 overflow.","b":true},{"text":" Some models, notably the T5 family and a few older checkpoints, were trained in bfloat16 and overflow in float16: you get "},{"text":"nan","code":true},{"text":" in the logits and empty or garbage output. "},{"text":"bfloat16","code":true},{"text":" has the same exponent range as float32 and does not; use it when the GPU supports it."}],[{"text":"float16 on CPU.","b":true},{"text":" Slow, and on some builds numerically wrong. Use float32 on CPU unless you have measured otherwise."}]]},{"t":"p","runs":[{"text":"A quick test: if "},{"text":"torch.isnan(out.logits).any()","code":true},{"text":" is true, you have found it."}]},{"t":"h2","id":"6-is-the-quantization-too-aggressive","runs":[{"text":"6. Is the quantization too aggressive?"}]},{"t":"p","runs":[{"text":"4-bit is fine for chat and noticeably worse for arithmetic, code and long structured output. If a model is good at fp16 and bad at 4-bit on your task, that is the quantization, not a bug. Step up to 8-bit or a "},{"text":"Q5_K_M","code":true},{"text":" GGUF and re-check before blaming anything else."}]},{"t":"h2","id":"7-are-the-generation-parameters-sane","runs":[{"text":"7. Are the generation parameters sane?"}]},{"t":"code","lang":"python","label":"Python","text":"print(model.generation_config)"},{"t":"p","runs":[{"text":"Look for "},{"text":"temperature","code":true},{"text":" above 1.2 with no top-p, a "},{"text":"repetition_penalty","code":true},{"text":" above about 1.3 (which starts suppressing ordinary words), and "},{"text":"do_sample: false","code":true},{"text":" ignoring everything you passed."}]},{"t":"h2","id":"8-did-the-input-get-truncated","runs":[{"text":"8. Did the input get truncated?"}]},{"t":"code","lang":"python","label":"Python","text":"print(enc[\"input_ids\"].shape[-1], model.config.max_position_embeddings)"},{"t":"p","runs":[{"text":"Silent truncation removes the end of a long document, usually the part that mattered. For a classifier reading complaints, this alone can halve accuracy."}]},{"t":"h2","id":"9-for-classifiers-what-does-id2label-say","runs":[{"text":"9. For classifiers: what does "},{"text":"id2label","code":true},{"text":" say?"}]},{"t":"p","runs":[{"text":"{0: 'LABEL_0', 1: 'LABEL_1'}","code":true},{"text":" means nobody told the model which is which, and a coin flip on polarity is a 50% chance you have it backwards across your whole dataset."}]},{"t":"h2","id":"10-only-now-suspect-the-model","runs":[{"text":"10. Only now, suspect the model"}]},{"t":"p","runs":[{"text":"After nine checks, it is reasonable to conclude the model is unsuited. Confirm it cheaply: run the same ten inputs through a well-known model of similar size. If that one is also poor, the task is harder than you thought or your inputs are unusual: both useful findings."}]},{"t":"h2","id":"the-minimal-reproduction","runs":[{"text":"The minimal reproduction"}]},{"t":"p","runs":[{"text":"When you report this to a discussion thread or a colleague, include: model id "},{"text":"with revision","b":true},{"text":", transformers and torch versions, the exact rendered prompt ("},{"text":"tokenize=False","code":true},{"text":"), the generation kwargs, the dtype, and the device. Half the questions on the Hub cannot be answered because the rendered prompt is missing, and the rendered prompt is the answer about a third of the time."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Deliberately break one working script four ways: drop the template, right-pad a batch, force float16 on a T5, and load the base instead of the instruct checkpoint. Look at each failure. You are building a library of signatures, and recognising a signature turns an afternoon into a minute."}]}],"check":{"question":[{"text":"A T5-family summariser returns empty strings on a GPU under float16 but works on CPU under float32. What is happening?"}],"options":[{"text":[{"text":"The GPU kernels for encoder-decoder attention require float32, so the half-precision path falls back to an untested code branch."}],"correct":false,"tempting":[{"text":"Invents a kernel restriction; half precision is well supported for these architectures."}]},{"text":[{"text":"Activations exceed float16’s range, producing NaN logits, so decoding immediately hits the end token and emits nothing."}],"correct":true,"tempting":null},{"text":[{"text":"The model’s weights were saved in bfloat16, and converting them to float16 rounds small values to zero, erasing the decoder’s embeddings."}],"correct":false,"tempting":[{"text":"Names the right precision pair but the wrong mechanism: the problem is exponent range and overflow to NaN, not underflow of small weights."}]},{"text":[{"text":"CPU and GPU use different tokenizer implementations, so the device change altered the input ids rather than the arithmetic."}],"correct":false,"tempting":[{"text":"Tokenization is device-independent and happens before anything reaches the accelerator."}]}],"order":[0,2,3,1],"why":null},"prev":{"slug":"streaming-and-stopping","title":"Showing tokens as they arrive, and making them stop","minutes":8},"next":{"slug":"running-it-locally","title":"Your disk fills up before your patience does","minutes":11}}}