Inside a GGUF file

One file, everything in it

GGUF is the container format used by llama.cpp and everything built on it. Its design goal was simple: a single file that a program can open and run with no accompanying Python, no config directory, no tokeniser package, and no code execution.

The layout is: a magic number and version, then a key-value metadata section, then a tensor index, then the tensor data itself, aligned so it can be memory-mapped directly.

The metadata is the interesting part. It carries the architecture name, layer counts, head counts, rope parameters, the full tokeniser (vocabulary, merges, special token ids) and the chat template as a Jinja string. That is why llama-server -m model.gguf needs nothing else, and why a GGUF from 2024 still runs today.

Inspect it:

python3 llama.cpp/gguf-py/scripts/gguf_dump.py model.gguf --no-tensors

You will see entries such as llama.context_length, llama.attention.head_count_kv, tokenizer.chat_template and general.quantization_version. Every number you need for the memory sum is there, on your own disk, with no network call.

Why one file matters more than it sounds

The alternative format, .safetensors plus a directory of JSON, requires a Python environment with a matching transformers version to interpret it. That is fine for a workstation and unhelpful on a phone, in a container you did not build, or in five years.

Safetensors deserves its own note. It replaced an older serialisation format whose files were not plain data: loading one ran code embedded in it, so a weights file from an unknown uploader was effectively a program. A .safetensors file is a header plus raw tensors, and a GGUF likewise. If you download weights from strangers, and everybody does, preferring these two formats is a genuine security control rather than a preference.

Not every tensor gets the same treatment

A GGUF’s name describes the dominant scheme, not a uniform one. In Q4_K_M, most weights are 4-bit k-quant, but the attention value projections and the feed-forward down projections in half the layers are 6-bit, and the token embedding and output layers get more bits again. Dump the tensors and you can see it:

python3 llama.cpp/gguf-py/scripts/gguf_dump.py model.gguf | head -40

The reason is empirical. Some tensors are far more sensitive to rounding than others, and spending bits where they matter is worth more than spending them evenly. The output layer is the extreme case: an error there lands directly on the token probabilities with nothing downstream to absorb it.

Reading the suffix letters

The naming looks arbitrary and is not.

• The number is the nominal bits per weight.
• _0 and _1 are the original simple block schemes: _1 carries an offset as well as a scale, so it handles asymmetric distributions better and costs slightly more.
• _K means k-quant: super-blocks with hierarchical scales and mixed precision per tensor type.
• _S, _M, _L are small, medium and large variants of a k-quant, differing in how many tensors are promoted to a higher bit rate. M is the usual default.
• IQ means i-quant: a different scheme using lattice-style codebooks, designed for very low bit rates and requiring an importance matrix to be worth anything.

One practical warning about i-quants: they need more arithmetic per weight to reconstruct, so on a slow CPU they can generate more slowly than a larger _K quant despite being smaller. On a GPU that cost disappears. Measure rather than assuming smaller is faster.

Where the metadata gets edited

GGUF metadata can be changed without re-quantising, which is occasionally exactly what you need: fixing a wrong chat template, correcting a rope scaling factor, or setting a default context length:

python3 llama.cpp/gguf-py/scripts/gguf_new_metadata.py in.gguf out.gguf \
  --chat-template-config template.jinja

This matters because a surprising number of community quants ship with a template copied from a different model. The weights are fine; the file’s idea of how to talk to them is not. Being able to look and fix it, rather than concluding the model is bad, is most of the value of understanding the format at all.

Split files

Large models arrive as model-00001-of-00009.gguf. Point the loader at the first shard and it finds the rest. llama-gguf-split --merge will combine them if you would rather have one file. The split exists because some filesystems and some download tools still struggle above a few tens of gigabytes.