{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"gpu-quantisation-formats","title":"GPTQ, AWQ and the GPU-side formats","url":"https://addaly.com/learn/running-models-yourself/gpu-quantisation-formats","position":22,"of":82,"minutes":9,"answer":"GPTQ compensates for rounding error as it goes and AWQ rescales the few channels that see large activations, but the kernel that reads the file matters as much as the algorithm that wrote it.","keyIdea":"GPTQ compensates for rounding error as it goes and AWQ rescales the few channels that see large activations, but the kernel that reads the file matters as much as the algorithm that wrote it."},"body":{"format":"markdown","text":"## A different design problem\n\nGGUF was designed to run anywhere, including on a CPU, and to support partial offload. The GPU-side formats were designed for one situation: the whole model in VRAM, served to many concurrent users, with kernels tuned for one vendor's hardware. Those different goals explain every difference between them.\n\n## GPTQ\n\nQuantises one layer at a time. For each layer it uses information about the curvature of the layer's error surface (an approximation to second-order information) to decide the rounding of each weight, and then adjusts the remaining un-quantised weights in that layer to compensate for the error already introduced.\n\nThat compensation step is the idea. Rounding weight A up creates an error; the remaining weights are nudged so the layer's overall output moves back towards where it was. It is a greedy correction and it works well.\n\nParameters you will see: 4-bit, group size 128, and `desc_act` (also called act-order), which processes columns in order of activation importance and improves quality at some cost in kernel speed. Requires a calibration dataset, typically a few hundred sequences.\n\n## AWQ\n\nStarts from an observation: a small fraction of weight channels (around 1%) see consistently large activations, and preserving those channels preserves most of the quality. Rather than keeping them in higher precision, which would make the kernels irregular and slow, AWQ scales those channels up before quantising and scales the corresponding activations down afterwards. The maths cancels; the important channels land in a better part of the quantisation range.\n\nThe result is uniform 4-bit weights, so the kernels stay fast and simple, with quality usually a little above GPTQ at the same bit rate. It also needs calibration data, and it is less sensitive to which data, because it is measuring activation magnitude rather than fitting an error surface.\n\n## EXL2 and EXL3\n\nExLlama's formats target a different user: one person on one NVIDIA card wanting maximum tokens per second. Rather than a fixed bit rate, they measure per-tensor sensitivity and assign different bit rates to different tensors to hit a target average, which is why you download a \"4.65 bpw\" model rather than a named tier. Combined with hand-written kernels, quantised cache and speculative decoding, this is the fastest single-user route on consumer NVIDIA hardware.\n\nThe costs are real: NVIDIA only, and throughput degrades much more sharply than vLLM as concurrent users are added.\n\n## compressed-tensors and the kernel question\n\nThe format that matters most in current serving is less a quantisation algorithm than a container. `compressed-tensors` describes what was done to each tensor (bit width, symmetry, group size, whether activations are quantised too) so a server can pick the right kernel. Models published this way are what vLLM and its relatives consume, and the accompanying tool, `llm-compressor`, produces them.\n\nAnd this is where a practical fact hides: **the kernel matters as much as the format**. A GPTQ model loaded with a naive kernel that dequantises to fp16 before every matrix multiply is slower than fp16. The same file loaded with a Marlin kernel, which fuses dequantisation into the multiply and is tuned for the memory layout, runs several times faster. When someone reports that 4-bit was no faster than 16-bit, this is almost always what happened, and the fix is a flag or a newer engine, not a different quant.\n\n## Choosing between them\n\n- **Model does not fit in VRAM, or you are not on NVIDIA, or you are on a Mac, or you are on a CPU**: GGUF. It is the only option that works, and this covers most readers.\n- **Model fits in VRAM, NVIDIA, several concurrent users**: AWQ or a compressed-tensors 4-bit build under vLLM. AWQ is the safe default; check whether your engine has a fast kernel for the specific scheme.\n- **Model fits, NVIDIA, single user, speed is everything**: EXL2 or EXL3 under TabbyAPI.\n- **Ada, Hopper or newer with room to spare**: FP8, covered next lesson, which is nearly lossless and needs no calibration.\n\n## The comparison to avoid\n\nDo not benchmark GGUF against AWQ and conclude one is better. You would be comparing two different bit allocations, two kernel implementations, two engines and probably two context configurations. The honest question is narrower and more useful: for *this* model on *my* hardware with *my* pattern of use, which combination gives acceptable quality at the best speed? That is three downloads and an afternoon, and the answer does not generalise to anybody else's machine."},"check":{"question":"An engineer converts a model to 4-bit GPTQ and finds inference is slightly slower than the original fp16 on the same card, with memory usage correctly reduced. What is the most likely explanation?","options":[{"text":"GPTQ's act-order option reorders columns, and the resulting irregular access pattern always costs more than the memory saving.","correct":false,"tempting":"Names a real option with a real cost, but act-order is a modest kernel penalty rather than something that erases a fourfold reduction in bytes read."},{"text":"The engine is dequantising each tensor to fp16 before the matrix multiply rather than using a fused kernel, so it pays the memory traffic of fp16 plus the unpacking work.","correct":true},{"text":"4-bit quantisation reduces memory but not computation, so speed is unchanged in principle and the small loss is measurement noise.","correct":false,"tempting":"Accepts a wrong model of the bottleneck; decode is memory-bound, so fewer bytes per weight is precisely what should make it faster."},{"text":"The calibration dataset was too small, leaving high-magnitude outliers that force the kernel into a slow path.","correct":false,"tempting":"Confuses a quality outcome with a performance one; calibration affects accuracy, not which kernel executes."}]},"prev":{"slug":"k-quants-and-importance-matrices","title":"K-quants and importance matrices","url":"https://addaly.com/learn/running-models-yourself/k-quants-and-importance-matrices"},"next":{"slug":"fp8-and-native-low-precision","title":"FP8 and models born quantised","url":"https://addaly.com/learn/running-models-yourself/fp8-and-native-low-precision"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'GPTQ, AWQ and the GPU-side formats', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/gpu-quantisation-formats (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"GPTQ, AWQ and the GPU-side formats","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"GPTQ compensates for rounding error as it goes and AWQ rescales the few channels that see large activations, but the kernel that reads the file matters as much as the algorithm that wrote it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"a-different-design-problem","runs":[{"text":"A different design problem"}]},{"t":"p","runs":[{"text":"GGUF was designed to run anywhere, including on a CPU, and to support partial offload. The GPU-side formats were designed for one situation: the whole model in VRAM, served to many concurrent users, with kernels tuned for one vendor’s hardware. Those different goals explain every difference between them."}]},{"t":"h2","id":"gptq","runs":[{"text":"GPTQ"}]},{"t":"p","runs":[{"text":"Quantises one layer at a time. For each layer it uses information about the curvature of the layer’s error surface (an approximation to second-order information) to decide the rounding of each weight, and then adjusts the remaining un-quantised weights in that layer to compensate for the error already introduced."}]},{"t":"p","runs":[{"text":"That compensation step is the idea. Rounding weight A up creates an error; the remaining weights are nudged so the layer’s overall output moves back towards where it was. It is a greedy correction and it works well."}]},{"t":"p","runs":[{"text":"Parameters you will see: 4-bit, group size 128, and "},{"text":"desc_act","code":true},{"text":" (also called act-order), which processes columns in order of activation importance and improves quality at some cost in kernel speed. Requires a calibration dataset, typically a few hundred sequences."}]},{"t":"h2","id":"awq","runs":[{"text":"AWQ"}]},{"t":"p","runs":[{"text":"Starts from an observation: a small fraction of weight channels (around 1%) see consistently large activations, and preserving those channels preserves most of the quality. Rather than keeping them in higher precision, which would make the kernels irregular and slow, AWQ scales those channels up before quantising and scales the corresponding activations down afterwards. The maths cancels; the important channels land in a better part of the quantisation range."}]},{"t":"p","runs":[{"text":"The result is uniform 4-bit weights, so the kernels stay fast and simple, with quality usually a little above GPTQ at the same bit rate. It also needs calibration data, and it is less sensitive to which data, because it is measuring activation magnitude rather than fitting an error surface."}]},{"t":"h2","id":"exl2-and-exl3","runs":[{"text":"EXL2 and EXL3"}]},{"t":"p","runs":[{"text":"ExLlama’s formats target a different user: one person on one NVIDIA card wanting maximum tokens per second. Rather than a fixed bit rate, they measure per-tensor sensitivity and assign different bit rates to different tensors to hit a target average, which is why you download a “4.65 bpw” model rather than a named tier. Combined with hand-written kernels, quantised cache and speculative decoding, this is the fastest single-user route on consumer NVIDIA hardware."}]},{"t":"p","runs":[{"text":"The costs are real: NVIDIA only, and throughput degrades much more sharply than vLLM as concurrent users are added."}]},{"t":"h2","id":"compressed-tensors-and-the-kernel-question","runs":[{"text":"compressed-tensors and the kernel question"}]},{"t":"p","runs":[{"text":"The format that matters most in current serving is less a quantisation algorithm than a container. "},{"text":"compressed-tensors","code":true},{"text":" describes what was done to each tensor (bit width, symmetry, group size, whether activations are quantised too) so a server can pick the right kernel. Models published this way are what vLLM and its relatives consume, and the accompanying tool, "},{"text":"llm-compressor","code":true},{"text":", produces them."}]},{"t":"p","runs":[{"text":"And this is where a practical fact hides: "},{"text":"the kernel matters as much as the format","b":true},{"text":". A GPTQ model loaded with a naive kernel that dequantises to fp16 before every matrix multiply is slower than fp16. The same file loaded with a Marlin kernel, which fuses dequantisation into the multiply and is tuned for the memory layout, runs several times faster. When someone reports that 4-bit was no faster than 16-bit, this is almost always what happened, and the fix is a flag or a newer engine, not a different quant."}]},{"t":"h2","id":"choosing-between-them","runs":[{"text":"Choosing between them"}]},{"t":"ul","items":[[{"text":"Model does not fit in VRAM, or you are not on NVIDIA, or you are on a Mac, or you are on a CPU","b":true},{"text":": GGUF. It is the only option that works, and this covers most readers."}],[{"text":"Model fits in VRAM, NVIDIA, several concurrent users","b":true},{"text":": AWQ or a compressed-tensors 4-bit build under vLLM. AWQ is the safe default; check whether your engine has a fast kernel for the specific scheme."}],[{"text":"Model fits, NVIDIA, single user, speed is everything","b":true},{"text":": EXL2 or EXL3 under TabbyAPI."}],[{"text":"Ada, Hopper or newer with room to spare","b":true},{"text":": FP8, covered next lesson, which is nearly lossless and needs no calibration."}]]},{"t":"h2","id":"the-comparison-to-avoid","runs":[{"text":"The comparison to avoid"}]},{"t":"p","runs":[{"text":"Do not benchmark GGUF against AWQ and conclude one is better. You would be comparing two different bit allocations, two kernel implementations, two engines and probably two context configurations. The honest question is narrower and more useful: for "},{"text":"this","i":true},{"text":" model on "},{"text":"my","i":true},{"text":" hardware with "},{"text":"my","i":true},{"text":" pattern of use, which combination gives acceptable quality at the best speed? That is three downloads and an afternoon, and the answer does not generalise to anybody else’s machine."}]}],"check":{"question":[{"text":"An engineer converts a model to 4-bit GPTQ and finds inference is slightly slower than the original fp16 on the same card, with memory usage correctly reduced. What is the most likely explanation?"}],"options":[{"text":[{"text":"GPTQ’s act-order option reorders columns, and the resulting irregular access pattern always costs more than the memory saving."}],"correct":false,"tempting":[{"text":"Names a real option with a real cost, but act-order is a modest kernel penalty rather than something that erases a fourfold reduction in bytes read."}]},{"text":[{"text":"The engine is dequantising each tensor to fp16 before the matrix multiply rather than using a fused kernel, so it pays the memory traffic of fp16 plus the unpacking work."}],"correct":true,"tempting":null},{"text":[{"text":"4-bit quantisation reduces memory but not computation, so speed is unchanged in principle and the small loss is measurement noise."}],"correct":false,"tempting":[{"text":"Accepts a wrong model of the bottleneck; decode is memory-bound, so fewer bytes per weight is precisely what should make it faster."}]},{"text":[{"text":"The calibration dataset was too small, leaving high-magnitude outliers that force the kernel into a slow path."}],"correct":false,"tempting":[{"text":"Confuses a quality outcome with a performance one; calibration affects accuracy, not which kernel executes."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"k-quants-and-importance-matrices","title":"K-quants and importance matrices","minutes":9},"next":{"slug":"fp8-and-native-low-precision","title":"FP8 and models born quantised","minutes":8}}}