GPTQ, AWQ and the GPU-side formats

A different design problem

GGUF was designed to run anywhere, including on a CPU, and to support partial offload. The GPU-side formats were designed for one situation: the whole model in VRAM, served to many concurrent users, with kernels tuned for one vendor’s hardware. Those different goals explain every difference between them.

GPTQ

Quantises one layer at a time. For each layer it uses information about the curvature of the layer’s error surface (an approximation to second-order information) to decide the rounding of each weight, and then adjusts the remaining un-quantised weights in that layer to compensate for the error already introduced.

That compensation step is the idea. Rounding weight A up creates an error; the remaining weights are nudged so the layer’s overall output moves back towards where it was. It is a greedy correction and it works well.

Parameters you will see: 4-bit, group size 128, and desc_act (also called act-order), which processes columns in order of activation importance and improves quality at some cost in kernel speed. Requires a calibration dataset, typically a few hundred sequences.

AWQ

Starts from an observation: a small fraction of weight channels (around 1%) see consistently large activations, and preserving those channels preserves most of the quality. Rather than keeping them in higher precision, which would make the kernels irregular and slow, AWQ scales those channels up before quantising and scales the corresponding activations down afterwards. The maths cancels; the important channels land in a better part of the quantisation range.

The result is uniform 4-bit weights, so the kernels stay fast and simple, with quality usually a little above GPTQ at the same bit rate. It also needs calibration data, and it is less sensitive to which data, because it is measuring activation magnitude rather than fitting an error surface.

EXL2 and EXL3

ExLlama’s formats target a different user: one person on one NVIDIA card wanting maximum tokens per second. Rather than a fixed bit rate, they measure per-tensor sensitivity and assign different bit rates to different tensors to hit a target average, which is why you download a “4.65 bpw” model rather than a named tier. Combined with hand-written kernels, quantised cache and speculative decoding, this is the fastest single-user route on consumer NVIDIA hardware.

The costs are real: NVIDIA only, and throughput degrades much more sharply than vLLM as concurrent users are added.

compressed-tensors and the kernel question

The format that matters most in current serving is less a quantisation algorithm than a container. compressed-tensors describes what was done to each tensor (bit width, symmetry, group size, whether activations are quantised too) so a server can pick the right kernel. Models published this way are what vLLM and its relatives consume, and the accompanying tool, llm-compressor, produces them.

And this is where a practical fact hides: the kernel matters as much as the format. A GPTQ model loaded with a naive kernel that dequantises to fp16 before every matrix multiply is slower than fp16. The same file loaded with a Marlin kernel, which fuses dequantisation into the multiply and is tuned for the memory layout, runs several times faster. When someone reports that 4-bit was no faster than 16-bit, this is almost always what happened, and the fix is a flag or a newer engine, not a different quant.

Choosing between them

• Model does not fit in VRAM, or you are not on NVIDIA, or you are on a Mac, or you are on a CPU: GGUF. It is the only option that works, and this covers most readers.
• Model fits in VRAM, NVIDIA, several concurrent users: AWQ or a compressed-tensors 4-bit build under vLLM. AWQ is the safe default; check whether your engine has a fast kernel for the specific scheme.
• Model fits, NVIDIA, single user, speed is everything: EXL2 or EXL3 under TabbyAPI.
• Ada, Hopper or newer with room to spare: FP8, covered next lesson, which is nearly lossless and needs no calibration.

The comparison to avoid

Do not benchmark GGUF against AWQ and conclude one is better. You would be comparing two different bit allocations, two kernel implementations, two engines and probably two context configurations. The honest question is narrower and more useful: for this model on my hardware with my pattern of use, which combination gives acceptable quality at the best speed? That is three downloads and an afternoon, and the answer does not generalise to anybody else’s machine.