{"course":{"slug":"ai-images","title":"Image Generation, in Practice","url":"https://addaly.com/learn/ai-images","level":"beginner","track":"create"},"lesson":{"slug":"quantisation-and-offload","title":"Making a big model fit a small card","url":"https://addaly.com/learn/ai-images/quantisation-and-offload","position":8,"of":69,"minutes":8,"answer":"Quantisation and offloading trade quality or speed for memory in known amounts: fp8 costs almost nothing, Q4 costs a little detail, and CPU offload costs seconds rather than quality.","keyIdea":"Quantisation and offloading trade quality or speed for memory in known amounts: fp8 costs almost nothing, Q4 costs a little detail, and CPU offload costs seconds rather than quality."},"body":{"format":"markdown","text":"## The model does not fit. Now what?\n\nFLUX dev in full precision is about 23 GB of weights. Your card has 8. There are three honest options: use a smaller model, shrink the weights, or keep most of them somewhere else. The second and third are worth understanding because the trade-offs are specific rather than vague.\n\n## Shrinking the weights\n\nWeights are numbers, and precision is how many bits each number gets.\n\n- **bf16/fp16**: 16 bits. The normal full-quality format. FLUX dev: ~23 GB.\n- **fp8**: 8 bits. Halves it to ~11 GB. Quality difference is hard to see in side-by-side tests; fine detail in skin and foliage shifts slightly, composition does not.\n- **GGUF Q8**: ~12 GB, quality near fp8.\n- **GGUF Q5_K_M**: ~8 GB.\n- **GGUF Q4_K_S**: ~6.5 GB. Now you can see it: fine text degrades faster, very small faces lose structure, and the model is slightly more likely to ignore a difficult part of the prompt.\n\nThe mechanism behind that last point is worth stating rather than just the outcome. Quantisation rounds each weight to a coarser grid. The error per weight is tiny, but it accumulates through the layers that resolve fine spatial structure, so the things that need the most precise weights (small text, distant faces, a thin correctly-shaped hand) degrade first, while broad composition is untouched. This is why a Q4 model looks perfectly good on a landscape and disappoints on a portrait with a sign in the background.\n\n> Figure: FLUX dev, the same model at five precisions. Bars (GB of weights held in memory): bf16, full quality 23; GGUF Q8 12; fp8 11; GGUF Q5_K_M 8; GGUF Q4_K_S 6.5. fp8 costs almost nothing you can see. Q4 costs small text, distant faces and thin repeated structure first, because those are the parts that need the most precise weights.\n\nGGUF support in ComfyUI comes from a community node pack rather than the core, which is worth knowing because it is one more thing that can break on an update.\n\n## Keeping weights elsewhere\n\nOffloading leaves parts of the model in system RAM and moves them to the GPU only when needed.\n\n- **Model CPU offload** moves whole components in and out as they are used. Costs a few seconds per image.\n- **Sequential CPU offload** goes layer by layer. It makes almost anything fit almost anything, and it is slow: commonly three to ten times slower.\n- **Tiled VAE** splits the decode into overlapping tiles so the final step stops being the memory peak.\n\nOffloading costs time and nothing else. Quantisation costs quality and nothing else. If you generate rarely and want the best picture, offload. If you generate constantly and can accept a hair less fine detail, quantise. Doing both is normal on an 8 GB card.\n\n## A worked configuration\n\nFLUX dev on an 8 GB card, which is a common situation:\n\n1. FLUX dev Q5_K_M GGUF, about 8 GB: too big alone, so:\n2. Keep the T5 text encoder on CPU. It is large and used once per prompt, so the transfer cost is paid once rather than per step.\n3. Enable tiled VAE decode.\n4. Generate at 1024×1024, 20 steps.\n\nExpect roughly 60 to 90 seconds an image. That is slow enough to change your method (you stop browsing and start planning), but it is a real working setup on a card that officially cannot run the model.\n\n## The failure this causes\n\nThe combination that catches people is quantisation plus LoRA. A LoRA computes a small correction to specific weights; when those weights have been rounded to a coarse grid, the correction lands slightly off, and effects are weaker or subtly wrong. A character LoRA that looks exactly right on fp16 can be a near-miss on Q4, and the instinct is to raise the LoRA strength, which then over-applies the style while still missing the identity.\n\nIf a LoRA is not behaving and you are on a quantised model, test it once at a higher precision before you spend an hour tuning strengths. You will often find the LoRA was fine.\n\n## Choosing where to spend the memory\n\nOn a constrained card you are dividing a fixed budget between four consumers, and it helps to know their relative appetites:\n\n1. **The denoising network**: the bulk of it, and the part quantisation shrinks.\n2. **The text encoder**: for FLUX-class models the T5 encoder is several gigabytes on its own. It runs once per prompt, so it is the best candidate for CPU offload: you pay the transfer once rather than on every step.\n3. **Activations during sampling**: scales with resolution and batch size. Halving the batch halves this.\n4. **The VAE decode**: one short spike at the end, solved by tiling.\n\nThe order to attack them in is 2, 4, 3, 1: offload the encoder, tile the decode, reduce the batch, and only then start rounding the weights that make the picture.\n\n## Measure it rather than trusting the forum\n\nQuality claims about quantisation are unusually unreliable online, because people compare a Q4 render against an fp16 render at a *different seed* and describe the difference as loss. It is not; it is a different picture.\n\nThe test that actually answers the question takes four minutes. Fix the seed, the prompt and every parameter. Render once at the higher precision and once at the lower. Open both at 100% and look at three places: small background text, the smallest face in frame, and any thin repeated structure such as railings or foliage. If you cannot tell them apart there, the quantised model is free quality for your work, and you should stop paying seconds for precision you cannot see."},"check":{"question":"Someone running a Q4 quantised model finds their character LoRA gives only a loose resemblance. They raise LoRA strength from 0.8 to 1.2 and the style becomes heavy-handed while the face still misses. What is happening?","options":[{"text":"The LoRA was trained at too low a rank to carry a specific identity, and no amount of added strength can make up for capacity that was never there.","correct":false,"tempting":"A plausible separate defect, but it would misbehave identically at full precision, which is the test this situation calls for."},{"text":"Strength above 1.0 is outside the valid range and saturates the adapter, so any value above 1.0 will degrade the output no matter what else is true.","correct":false,"tempting":"Values above 1.0 are unusual but legal and often useful; the damage here is over-application of what the LoRA does carry, not an invalid range."},{"text":"Quantisation rounded the base weights the correction is computed against, so it lands off-target and strength amplifies the error too.","correct":true},{"text":"The quantised build uses a different architecture, so only some of the LoRA's keys match and the remainder are skipped without any warning being shown.","correct":false,"tempting":"Key mismatch does occur across base families and produces a warning or a no-op, not a partial resemblance that scales with strength."}]},"prev":{"slug":"speed-and-steps","title":"Steps, samplers and the seconds you should actually expect","url":"https://addaly.com/learn/ai-images/speed-and-steps"},"next":{"slug":"the-safety-filter-differs","title":"Every platform's filter is different, and that is a production risk","url":"https://addaly.com/learn/ai-images/the-safety-filter-differs"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Making a big model fit a small card', in Image Generation, in Practice. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-images/quantisation-and-offload (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Making a big model fit a small card","courseTitle":"Image Generation, in Practice","module":{"index":1,"slug":"the-kit","title":"The kit, and what each piece of it costs you"},"answer":null,"keyIdea":"Quantisation and offloading trade quality or speed for memory in known amounts: fp8 costs almost nothing, Q4 costs a little detail, and CPU offload costs seconds rather than quality.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"GB of weights held in memory","title":"FLUX dev, the same model at five precisions","series":[{"label":"bf16, full quality","value":23},{"label":"GGUF Q8","value":12},{"label":"fp8","value":11},{"label":"GGUF Q5_K_M","value":8,"highlight":true},{"label":"GGUF Q4_K_S","value":6.5,"highlight":true}],"caption":"fp8 costs almost nothing you can see. Q4 costs small text, distant faces and thin repeated structure first, because those are the parts that need the most precise weights."},"alt":"Figure: FLUX dev, the same model at five precisions. Bars (GB of weights held in memory): bf16, full quality 23; GGUF Q8 12; fp8 11; GGUF Q5_K_M 8; GGUF Q4_K_S 6.5. fp8 costs almost nothing you can see. Q4 costs small text, distant faces and thin repeated structure first, because those are the parts that need the most precise weights."}},"blocks":[{"t":"h2","id":"the-model-does-not-fit-now-what","runs":[{"text":"The model does not fit. Now what?"}]},{"t":"p","runs":[{"text":"FLUX dev in full precision is about 23 GB of weights. Your card has 8. There are three honest options: use a smaller model, shrink the weights, or keep most of them somewhere else. The second and third are worth understanding because the trade-offs are specific rather than vague."}]},{"t":"h2","id":"shrinking-the-weights","runs":[{"text":"Shrinking the weights"}]},{"t":"p","runs":[{"text":"Weights are numbers, and precision is how many bits each number gets."}]},{"t":"ul","items":[[{"text":"bf16/fp16","b":true},{"text":": 16 bits. The normal full-quality format. FLUX dev: ~23 GB."}],[{"text":"fp8","b":true},{"text":": 8 bits. Halves it to ~11 GB. Quality difference is hard to see in side-by-side tests; fine detail in skin and foliage shifts slightly, composition does not."}],[{"text":"GGUF Q8","b":true},{"text":": ~12 GB, quality near fp8."}],[{"text":"GGUF Q5_K_M","b":true},{"text":": ~8 GB."}],[{"text":"GGUF Q4_K_S","b":true},{"text":": ~6.5 GB. Now you can see it: fine text degrades faster, very small faces lose structure, and the model is slightly more likely to ignore a difficult part of the prompt."}]]},{"t":"p","runs":[{"text":"The mechanism behind that last point is worth stating rather than just the outcome. Quantisation rounds each weight to a coarser grid. The error per weight is tiny, but it accumulates through the layers that resolve fine spatial structure, so the things that need the most precise weights (small text, distant faces, a thin correctly-shaped hand) degrade first, while broad composition is untouched. This is why a Q4 model looks perfectly good on a landscape and disappoints on a portrait with a sign in the background."}]},{"t":"p","runs":[{"text":"GGUF support in ComfyUI comes from a community node pack rather than the core, which is worth knowing because it is one more thing that can break on an update."}]},{"t":"h2","id":"keeping-weights-elsewhere","runs":[{"text":"Keeping weights elsewhere"}]},{"t":"p","runs":[{"text":"Offloading leaves parts of the model in system RAM and moves them to the GPU only when needed."}]},{"t":"ul","items":[[{"text":"Model CPU offload","b":true},{"text":" moves whole components in and out as they are used. Costs a few seconds per image."}],[{"text":"Sequential CPU offload","b":true},{"text":" goes layer by layer. It makes almost anything fit almost anything, and it is slow: commonly three to ten times slower."}],[{"text":"Tiled VAE","b":true},{"text":" splits the decode into overlapping tiles so the final step stops being the memory peak."}]]},{"t":"p","runs":[{"text":"Offloading costs time and nothing else. Quantisation costs quality and nothing else. If you generate rarely and want the best picture, offload. If you generate constantly and can accept a hair less fine detail, quantise. Doing both is normal on an 8 GB card."}]},{"t":"h2","id":"a-worked-configuration","runs":[{"text":"A worked configuration"}]},{"t":"p","runs":[{"text":"FLUX dev on an 8 GB card, which is a common situation:"}]},{"t":"ol","start":1,"items":[[{"text":"FLUX dev Q5_K_M GGUF, about 8 GB: too big alone, so:"}],[{"text":"Keep the T5 text encoder on CPU. It is large and used once per prompt, so the transfer cost is paid once rather than per step."}],[{"text":"Enable tiled VAE decode."}],[{"text":"Generate at 1024×1024, 20 steps."}]]},{"t":"p","runs":[{"text":"Expect roughly 60 to 90 seconds an image. That is slow enough to change your method (you stop browsing and start planning), but it is a real working setup on a card that officially cannot run the model."}]},{"t":"h2","id":"the-failure-this-causes","runs":[{"text":"The failure this causes"}]},{"t":"p","runs":[{"text":"The combination that catches people is quantisation plus LoRA. A LoRA computes a small correction to specific weights; when those weights have been rounded to a coarse grid, the correction lands slightly off, and effects are weaker or subtly wrong. A character LoRA that looks exactly right on fp16 can be a near-miss on Q4, and the instinct is to raise the LoRA strength, which then over-applies the style while still missing the identity."}]},{"t":"p","runs":[{"text":"If a LoRA is not behaving and you are on a quantised model, test it once at a higher precision before you spend an hour tuning strengths. You will often find the LoRA was fine."}]},{"t":"h2","id":"choosing-where-to-spend-the-memory","runs":[{"text":"Choosing where to spend the memory"}]},{"t":"p","runs":[{"text":"On a constrained card you are dividing a fixed budget between four consumers, and it helps to know their relative appetites:"}]},{"t":"ol","start":1,"items":[[{"text":"The denoising network","b":true},{"text":": the bulk of it, and the part quantisation shrinks."}],[{"text":"The text encoder","b":true},{"text":": for FLUX-class models the T5 encoder is several gigabytes on its own. It runs once per prompt, so it is the best candidate for CPU offload: you pay the transfer once rather than on every step."}],[{"text":"Activations during sampling","b":true},{"text":": scales with resolution and batch size. Halving the batch halves this."}],[{"text":"The VAE decode","b":true},{"text":": one short spike at the end, solved by tiling."}]]},{"t":"p","runs":[{"text":"The order to attack them in is 2, 4, 3, 1: offload the encoder, tile the decode, reduce the batch, and only then start rounding the weights that make the picture."}]},{"t":"h2","id":"measure-it-rather-than-trusting-the-forum","runs":[{"text":"Measure it rather than trusting the forum"}]},{"t":"p","runs":[{"text":"Quality claims about quantisation are unusually unreliable online, because people compare a Q4 render against an fp16 render at a "},{"text":"different seed","i":true},{"text":" and describe the difference as loss. It is not; it is a different picture."}]},{"t":"p","runs":[{"text":"The test that actually answers the question takes four minutes. Fix the seed, the prompt and every parameter. Render once at the higher precision and once at the lower. Open both at 100% and look at three places: small background text, the smallest face in frame, and any thin repeated structure such as railings or foliage. If you cannot tell them apart there, the quantised model is free quality for your work, and you should stop paying seconds for precision you cannot see."}]}],"check":{"question":[{"text":"Someone running a Q4 quantised model finds their character LoRA gives only a loose resemblance. They raise LoRA strength from 0.8 to 1.2 and the style becomes heavy-handed while the face still misses. What is happening?"}],"options":[{"text":[{"text":"The LoRA was trained at too low a rank to carry a specific identity, and no amount of added strength can make up for capacity that was never there."}],"correct":false,"tempting":[{"text":"A plausible separate defect, but it would misbehave identically at full precision, which is the test this situation calls for."}]},{"text":[{"text":"Strength above 1.0 is outside the valid range and saturates the adapter, so any value above 1.0 will degrade the output no matter what else is true."}],"correct":false,"tempting":[{"text":"Values above 1.0 are unusual but legal and often useful; the damage here is over-application of what the LoRA does carry, not an invalid range."}]},{"text":[{"text":"Quantisation rounded the base weights the correction is computed against, so it lands off-target and strength amplifies the error too."}],"correct":true,"tempting":null},{"text":[{"text":"The quantised build uses a different architecture, so only some of the LoRA’s keys match and the remainder are skipped without any warning being shown."}],"correct":false,"tempting":[{"text":"Key mismatch does occur across base families and produces a warning or a no-op, not a partial resemblance that scales with strength."}]}],"order":[1,0,2,3],"why":null},"prev":{"slug":"speed-and-steps","title":"Steps, samplers and the seconds you should actually expect","minutes":8},"next":{"slug":"the-safety-filter-differs","title":"Every platform’s filter is different, and that is a production risk","minutes":7}}}