Making a big model fit a small card

The model does not fit. Now what?

FLUX dev in full precision is about 23 GB of weights. Your card has 8. There are three honest options: use a smaller model, shrink the weights, or keep most of them somewhere else. The second and third are worth understanding because the trade-offs are specific rather than vague.

Shrinking the weights

Weights are numbers, and precision is how many bits each number gets.

• bf16/fp16: 16 bits. The normal full-quality format. FLUX dev: ~23 GB.
• fp8: 8 bits. Halves it to ~11 GB. Quality difference is hard to see in side-by-side tests; fine detail in skin and foliage shifts slightly, composition does not.
• GGUF Q8: ~12 GB, quality near fp8.
• GGUF Q5_K_M: ~8 GB.
• GGUF Q4_K_S: ~6.5 GB. Now you can see it: fine text degrades faster, very small faces lose structure, and the model is slightly more likely to ignore a difficult part of the prompt.

The mechanism behind that last point is worth stating rather than just the outcome. Quantisation rounds each weight to a coarser grid. The error per weight is tiny, but it accumulates through the layers that resolve fine spatial structure, so the things that need the most precise weights (small text, distant faces, a thin correctly-shaped hand) degrade first, while broad composition is untouched. This is why a Q4 model looks perfectly good on a landscape and disappoints on a portrait with a sign in the background.

GGUF support in ComfyUI comes from a community node pack rather than the core, which is worth knowing because it is one more thing that can break on an update.

Keeping weights elsewhere

Offloading leaves parts of the model in system RAM and moves them to the GPU only when needed.

• Model CPU offload moves whole components in and out as they are used. Costs a few seconds per image.
• Sequential CPU offload goes layer by layer. It makes almost anything fit almost anything, and it is slow: commonly three to ten times slower.
• Tiled VAE splits the decode into overlapping tiles so the final step stops being the memory peak.

Offloading costs time and nothing else. Quantisation costs quality and nothing else. If you generate rarely and want the best picture, offload. If you generate constantly and can accept a hair less fine detail, quantise. Doing both is normal on an 8 GB card.

A worked configuration

FLUX dev on an 8 GB card, which is a common situation:

1. FLUX dev Q5_K_M GGUF, about 8 GB: too big alone, so:
2. Keep the T5 text encoder on CPU. It is large and used once per prompt, so the transfer cost is paid once rather than per step.
3. Enable tiled VAE decode.
4. Generate at 1024×1024, 20 steps.

Expect roughly 60 to 90 seconds an image. That is slow enough to change your method (you stop browsing and start planning), but it is a real working setup on a card that officially cannot run the model.

The failure this causes

The combination that catches people is quantisation plus LoRA. A LoRA computes a small correction to specific weights; when those weights have been rounded to a coarse grid, the correction lands slightly off, and effects are weaker or subtly wrong. A character LoRA that looks exactly right on fp16 can be a near-miss on Q4, and the instinct is to raise the LoRA strength, which then over-applies the style while still missing the identity.

If a LoRA is not behaving and you are on a quantised model, test it once at a higher precision before you spend an hour tuning strengths. You will often find the LoRA was fine.

Choosing where to spend the memory

On a constrained card you are dividing a fixed budget between four consumers, and it helps to know their relative appetites:

1. The denoising network: the bulk of it, and the part quantisation shrinks.
2. The text encoder: for FLUX-class models the T5 encoder is several gigabytes on its own. It runs once per prompt, so it is the best candidate for CPU offload: you pay the transfer once rather than on every step.
3. Activations during sampling: scales with resolution and batch size. Halving the batch halves this.
4. The VAE decode: one short spike at the end, solved by tiling.

The order to attack them in is 2, 4, 3, 1: offload the encoder, tile the decode, reduce the batch, and only then start rounding the weights that make the picture.

Measure it rather than trusting the forum

Quality claims about quantisation are unusually unreliable online, because people compare a Q4 render against an fp16 render at a different seed and describe the difference as loss. It is not; it is a different picture.

The test that actually answers the question takes four minutes. Fix the seed, the prompt and every parameter. Render once at the higher precision and once at the lower. Open both at 100% and look at three places: small background text, the smallest face in frame, and any thin repeated structure such as railings or foliage. If you cannot tell them apart there, the quantised model is free quality for your work, and you should stop paying seconds for precision you cannot see.