{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"debugging-a-bad-output","title":"The diagnostic ladder for a bad output","url":"https://addaly.com/learn/running-models-yourself/debugging-a-bad-output","position":54,"of":82,"minutes":9,"answer":"Work down the ladder in order (hardware, prompt string, stop tokens, sampler at temperature 0, quantisation, prompt design, then the weights) because the cheap causes are also by far the most common.","keyIdea":"Work down the ladder in order (hardware, prompt string, stop tokens, sampler at temperature 0, quantisation, prompt design, then the weights) because the cheap causes are also by far the most common."},"body":{"format":"markdown","text":"## The order matters because the cheap causes are the common ones\n\nWhen a local model produces something poor, there are six possible causes and people almost always investigate them in the wrong order: starting with the model and ending with the template, when the reverse is correct.\n\nWork down this list. Each step is minutes, and most problems are resolved in the first three.\n\n> Figure: The ladder, in the order that finds it fastest. Layers, top to bottom: 1. Is it on the hardware you think? (nvidia-smi during generation. Zero means the CPU); 2. Is the prompt what you think it is? (Print the templated string and the token count); 3. Is it stopping in the right place? (A missing end marker invents the next user turn); 4. Sampler off, temperature zero (If it is good now, the sampler was the problem); 5. Is it the quantisation? (The same prompt against a higher-bit copy); 6. Is it the prompt design? (Too many instructions, format described not shown); 7. Only now, is it the model? (Run it against the full-precision weights). Each step is minutes and most problems are resolved in the first three. Working the other way, starting with the weights and ending with the template, is how an afternoon disappears and a perfectly good model gets blamed.\n\n## 1. Is it actually running on the hardware you think?\n\n`nvidia-smi` during generation. If utilisation is zero, the model is on the CPU and everything else you observe is a symptom of that. Thirty seconds, and it explains a surprising share of \"the model is bad\" reports where the real complaint was speed.\n\n## 2. Is the prompt what you think it is?\n\nPrint the exact string the model receives, template and all.\n\n```python\nprint(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))\n```\n\nLook for: a missing or wrong template, a doubled beginning-of-sequence token, a system prompt that has silently vanished because the model has no system role, and a missing generation prompt. Then check the token count against your configured window. If the count exceeds the window, the model never saw the beginning of your input and nothing further is worth investigating.\n\n## 3. Is generation stopping in the right place?\n\nIf the output runs on, invents the next user turn, or trails into unrelated text, the end-of-turn token is not in the stop list. Add it explicitly and re-test.\n\n## 4. Are the sampler settings sane?\n\nSet temperature to 0 and turn everything else off. If the output becomes good, the problem was in the sampler and you can reintroduce settings one at a time. If it stays bad, sampling was never the issue and you have eliminated a whole area.\n\nThis single test (everything off, temperature 0) is the highest-value diagnostic in the list, because it cleanly separates two halves of the problem space.\n\n## 5. Is it the quantisation?\n\nRun the same prompt against a higher-bit quant of the same model, or against a hosted copy of the full-precision weights. If the higher precision is clearly better, you have found it. If not, the quant is exonerated and you have a real answer rather than a suspicion.\n\nThis is worth doing before any hardware decision, because \"buy more memory to run a better quant\" is only justified once you have evidence the quant is the constraint.\n\n## 6. Is it the prompt design?\n\nToo many simultaneous instructions, key constraints buried in the middle, format described rather than demonstrated, no examples. Rewrite along the lines of the prompting lessons and re-test on the same ten items.\n\n## 7. Is it the model?\n\nOnly now. Run the same prompts against the same weights hosted at full precision. If they fail there too, the weights cannot do the task, and the options are a different model, a different size class, decomposition into smaller steps, or retrieval so the answer comes from a document instead of the model's memory.\n\n## Symptom to cause, at a glance\n\n- **Fluent but ignores the system prompt**: template or system-role support.\n- **Repeats a phrase until the token limit**: sampler; try DRY.\n- **Answers well then invents a conversation**: missing stop token.\n- **Forgets the start of a long document**: context window, silently truncated.\n- **Produces prose where JSON was requested**: no grammar, no anchoring.\n- **Good on short inputs, confidently wrong on long ones**: past your working context limit.\n- **Broken indentation and missing brackets**: repetition penalty too high.\n- **Correct but unbearably slow**: not a quality problem at all; go back to step 1.\n- **Different answers to the same prompt at temperature 0**: floating-point and batch composition, not a bug.\n\n## The habit that makes this work\n\nChange one thing at a time and keep the same ten test items in front of you throughout. Two changes at once produce a result you cannot attribute, and a new set of test items produces a comparison you cannot make.\n\nAnd write down what fixed it. Every one of these problems recurs (on the next model, on the next machine, when a colleague hits the same wall) and a five-line note is the difference between remembering the answer and rediscovering it."},"check":{"question":"A local model gives excellent answers about the end of a pasted 15,000-token report and nonsense about its opening pages. Which diagnostic step will most likely find the cause, and why?","options":[{"text":"Compare against a higher-bit quant, since low-bit quantisation damages long-context recall first.","correct":false,"tempting":"Quantisation does affect long-context behaviour, but it would degrade recall generally rather than removing the beginning of the input specifically."},{"text":"Set temperature to 0 with all samplers off, since sampler settings degrade coherence over long inputs.","correct":false,"tempting":"The highest-value general diagnostic, but sampler behaviour does not vary with position in the input, so it cannot produce a start-versus-end asymmetry."},{"text":"Check the prompt token count against the configured context window, because a window smaller than the input drops the oldest tokens without any error.","correct":true},{"text":"Add the end-of-turn token to the stop list, since a run-on generation can overwrite earlier context.","correct":false,"tempting":"Confuses generation termination with input handling; stop tokens have no effect on what the model was given to read."}]},"prev":{"slug":"determinism-and-reproducibility","title":"Why the same seed gives different answers","url":"https://addaly.com/learn/running-models-yourself/determinism-and-reproducibility"},"next":{"slug":"a-local-eval-harness","title":"A small harness that tells you whether a change helped","url":"https://addaly.com/learn/running-models-yourself/a-local-eval-harness"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The diagnostic ladder for a bad output', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/debugging-a-bad-output (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The diagnostic ladder for a bad output","courseTitle":"Running Models Yourself","module":{"index":6,"slug":"getting-good-output","title":"Getting good output"},"answer":null,"keyIdea":"Work down the ladder in order (hardware, prompt string, stop tokens, sampler at temperature 0, quantisation, prompt design, then the weights) because the cheap causes are also by far the most common.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"The ladder, in the order that finds it fastest","layers":[{"label":"1. Is it on the hardware you think?","detail":"nvidia-smi during generation. Zero means the CPU"},{"label":"2. Is the prompt what you think it is?","detail":"Print the templated string and the token count"},{"label":"3. Is it stopping in the right place?","detail":"A missing end marker invents the next user turn"},{"label":"4. Sampler off, temperature zero","detail":"If it is good now, the sampler was the problem"},{"label":"5. Is it the quantisation?","detail":"The same prompt against a higher-bit copy"},{"label":"6. Is it the prompt design?","detail":"Too many instructions, format described not shown"},{"label":"7. Only now, is it the model?","detail":"Run it against the full-precision weights"}],"caption":"Each step is minutes and most problems are resolved in the first three. Working the other way, starting with the weights and ending with the template, is how an afternoon disappears and a perfectly good model gets blamed."},"alt":"Figure: The ladder, in the order that finds it fastest. Layers, top to bottom: 1. Is it on the hardware you think? (nvidia-smi during generation. Zero means the CPU); 2. Is the prompt what you think it is? (Print the templated string and the token count); 3. Is it stopping in the right place? (A missing end marker invents the next user turn); 4. Sampler off, temperature zero (If it is good now, the sampler was the problem); 5. Is it the quantisation? (The same prompt against a higher-bit copy); 6. Is it the prompt design? (Too many instructions, format described not shown); 7. Only now, is it the model? (Run it against the full-precision weights). Each step is minutes and most problems are resolved in the first three. Working the other way, starting with the weights and ending with the template, is how an afternoon disappears and a perfectly good model gets blamed."}},"blocks":[{"t":"h2","id":"the-order-matters-because-the-cheap-causes-are-the-common-on","runs":[{"text":"The order matters because the cheap causes are the common ones"}]},{"t":"p","runs":[{"text":"When a local model produces something poor, there are six possible causes and people almost always investigate them in the wrong order: starting with the model and ending with the template, when the reverse is correct."}]},{"t":"p","runs":[{"text":"Work down this list. Each step is minutes, and most problems are resolved in the first three."}]},{"t":"h2","id":"1-is-it-actually-running-on-the-hardware-you-think","runs":[{"text":"1. Is it actually running on the hardware you think?"}]},{"t":"p","runs":[{"text":"nvidia-smi","code":true},{"text":" during generation. If utilisation is zero, the model is on the CPU and everything else you observe is a symptom of that. Thirty seconds, and it explains a surprising share of “the model is bad” reports where the real complaint was speed."}]},{"t":"h2","id":"2-is-the-prompt-what-you-think-it-is","runs":[{"text":"2. Is the prompt what you think it is?"}]},{"t":"p","runs":[{"text":"Print the exact string the model receives, template and all."}]},{"t":"code","lang":"python","label":"Python","text":"print(tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True))"},{"t":"p","runs":[{"text":"Look for: a missing or wrong template, a doubled beginning-of-sequence token, a system prompt that has silently vanished because the model has no system role, and a missing generation prompt. Then check the token count against your configured window. If the count exceeds the window, the model never saw the beginning of your input and nothing further is worth investigating."}]},{"t":"h2","id":"3-is-generation-stopping-in-the-right-place","runs":[{"text":"3. Is generation stopping in the right place?"}]},{"t":"p","runs":[{"text":"If the output runs on, invents the next user turn, or trails into unrelated text, the end-of-turn token is not in the stop list. Add it explicitly and re-test."}]},{"t":"h2","id":"4-are-the-sampler-settings-sane","runs":[{"text":"4. Are the sampler settings sane?"}]},{"t":"p","runs":[{"text":"Set temperature to 0 and turn everything else off. If the output becomes good, the problem was in the sampler and you can reintroduce settings one at a time. If it stays bad, sampling was never the issue and you have eliminated a whole area."}]},{"t":"p","runs":[{"text":"This single test (everything off, temperature 0) is the highest-value diagnostic in the list, because it cleanly separates two halves of the problem space."}]},{"t":"h2","id":"5-is-it-the-quantisation","runs":[{"text":"5. Is it the quantisation?"}]},{"t":"p","runs":[{"text":"Run the same prompt against a higher-bit quant of the same model, or against a hosted copy of the full-precision weights. If the higher precision is clearly better, you have found it. If not, the quant is exonerated and you have a real answer rather than a suspicion."}]},{"t":"p","runs":[{"text":"This is worth doing before any hardware decision, because “buy more memory to run a better quant” is only justified once you have evidence the quant is the constraint."}]},{"t":"h2","id":"6-is-it-the-prompt-design","runs":[{"text":"6. Is it the prompt design?"}]},{"t":"p","runs":[{"text":"Too many simultaneous instructions, key constraints buried in the middle, format described rather than demonstrated, no examples. Rewrite along the lines of the prompting lessons and re-test on the same ten items."}]},{"t":"h2","id":"7-is-it-the-model","runs":[{"text":"7. Is it the model?"}]},{"t":"p","runs":[{"text":"Only now. Run the same prompts against the same weights hosted at full precision. If they fail there too, the weights cannot do the task, and the options are a different model, a different size class, decomposition into smaller steps, or retrieval so the answer comes from a document instead of the model’s memory."}]},{"t":"h2","id":"symptom-to-cause-at-a-glance","runs":[{"text":"Symptom to cause, at a glance"}]},{"t":"ul","items":[[{"text":"Fluent but ignores the system prompt","b":true},{"text":": template or system-role support."}],[{"text":"Repeats a phrase until the token limit","b":true},{"text":": sampler; try DRY."}],[{"text":"Answers well then invents a conversation","b":true},{"text":": missing stop token."}],[{"text":"Forgets the start of a long document","b":true},{"text":": context window, silently truncated."}],[{"text":"Produces prose where JSON was requested","b":true},{"text":": no grammar, no anchoring."}],[{"text":"Good on short inputs, confidently wrong on long ones","b":true},{"text":": past your working context limit."}],[{"text":"Broken indentation and missing brackets","b":true},{"text":": repetition penalty too high."}],[{"text":"Correct but unbearably slow","b":true},{"text":": not a quality problem at all; go back to step 1."}],[{"text":"Different answers to the same prompt at temperature 0","b":true},{"text":": floating-point and batch composition, not a bug."}]]},{"t":"h2","id":"the-habit-that-makes-this-work","runs":[{"text":"The habit that makes this work"}]},{"t":"p","runs":[{"text":"Change one thing at a time and keep the same ten test items in front of you throughout. Two changes at once produce a result you cannot attribute, and a new set of test items produces a comparison you cannot make."}]},{"t":"p","runs":[{"text":"And write down what fixed it. Every one of these problems recurs (on the next model, on the next machine, when a colleague hits the same wall) and a five-line note is the difference between remembering the answer and rediscovering it."}]}],"check":{"question":[{"text":"A local model gives excellent answers about the end of a pasted 15,000-token report and nonsense about its opening pages. Which diagnostic step will most likely find the cause, and why?"}],"options":[{"text":[{"text":"Compare against a higher-bit quant, since low-bit quantisation damages long-context recall first."}],"correct":false,"tempting":[{"text":"Quantisation does affect long-context behaviour, but it would degrade recall generally rather than removing the beginning of the input specifically."}]},{"text":[{"text":"Set temperature to 0 with all samplers off, since sampler settings degrade coherence over long inputs."}],"correct":false,"tempting":[{"text":"The highest-value general diagnostic, but sampler behaviour does not vary with position in the input, so it cannot produce a start-versus-end asymmetry."}]},{"text":[{"text":"Check the prompt token count against the configured context window, because a window smaller than the input drops the oldest tokens without any error."}],"correct":true,"tempting":null},{"text":[{"text":"Add the end-of-turn token to the stop list, since a run-on generation can overwrite earlier context."}],"correct":false,"tempting":[{"text":"Confuses generation termination with input handling; stop tokens have no effect on what the model was given to read."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"determinism-and-reproducibility","title":"Why the same seed gives different answers","minutes":8},"next":{"slug":"a-local-eval-harness","title":"A small harness that tells you whether a change helped","minutes":9}}}