{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"inside-a-gguf-file","title":"Inside a GGUF file","url":"https://addaly.com/learn/running-models-yourself/inside-a-gguf-file","position":20,"of":82,"minutes":8,"answer":"A GGUF is one memory-mappable file carrying weights, tokeniser and chat template together, with different tensors quantised at different bit rates, and its metadata can be inspected and repaired without re-quantising anything.","keyIdea":"A GGUF is one memory-mappable file carrying weights, tokeniser and chat template together, with different tensors quantised at different bit rates, and its metadata can be inspected and repaired without re-quantising anything."},"body":{"format":"markdown","text":"## One file, everything in it\n\nGGUF is the container format used by llama.cpp and everything built on it. Its design goal was simple: a single file that a program can open and run with no accompanying Python, no config directory, no tokeniser package, and no code execution.\n\nThe layout is: a magic number and version, then a key-value metadata section, then a tensor index, then the tensor data itself, aligned so it can be memory-mapped directly.\n\nThe metadata is the interesting part. It carries the architecture name, layer counts, head counts, rope parameters, the full tokeniser (vocabulary, merges, special token ids) and the chat template as a Jinja string. That is why `llama-server -m model.gguf` needs nothing else, and why a GGUF from 2024 still runs today.\n\nInspect it:\n\n```bash\npython3 llama.cpp/gguf-py/scripts/gguf_dump.py model.gguf --no-tensors\n```\n\nYou will see entries such as `llama.context_length`, `llama.attention.head_count_kv`, `tokenizer.chat_template` and `general.quantization_version`. Every number you need for the memory sum is there, on your own disk, with no network call.\n\n## Why one file matters more than it sounds\n\nThe alternative format, `.safetensors` plus a directory of JSON, requires a Python environment with a matching transformers version to interpret it. That is fine for a workstation and unhelpful on a phone, in a container you did not build, or in five years.\n\nSafetensors deserves its own note. It replaced an older serialisation format whose files were not plain data: loading one ran code embedded in it, so a weights file from an unknown uploader was effectively a program. A `.safetensors` file is a header plus raw tensors, and a GGUF likewise. If you download weights from strangers, and everybody does, preferring these two formats is a genuine security control rather than a preference.\n\n## Not every tensor gets the same treatment\n\nA GGUF's name describes the dominant scheme, not a uniform one. In `Q4_K_M`, most weights are 4-bit k-quant, but the attention value projections and the feed-forward down projections in half the layers are 6-bit, and the token embedding and output layers get more bits again. Dump the tensors and you can see it:\n\n```bash\npython3 llama.cpp/gguf-py/scripts/gguf_dump.py model.gguf | head -40\n```\n\nThe reason is empirical. Some tensors are far more sensitive to rounding than others, and spending bits where they matter is worth more than spending them evenly. The output layer is the extreme case: an error there lands directly on the token probabilities with nothing downstream to absorb it.\n\n## Reading the suffix letters\n\nThe naming looks arbitrary and is not.\n\n- The number is the nominal bits per weight.\n- `_0` and `_1` are the original simple block schemes: `_1` carries an offset as well as a scale, so it handles asymmetric distributions better and costs slightly more.\n- `_K` means k-quant: super-blocks with hierarchical scales and mixed precision per tensor type.\n- `_S`, `_M`, `_L` are small, medium and large variants of a k-quant, differing in how many tensors are promoted to a higher bit rate. `M` is the usual default.\n- `IQ` means i-quant: a different scheme using lattice-style codebooks, designed for very low bit rates and requiring an importance matrix to be worth anything.\n\nOne practical warning about i-quants: they need more arithmetic per weight to reconstruct, so on a slow CPU they can generate more slowly than a larger `_K` quant despite being smaller. On a GPU that cost disappears. Measure rather than assuming smaller is faster.\n\n## Where the metadata gets edited\n\nGGUF metadata can be changed without re-quantising, which is occasionally exactly what you need: fixing a wrong chat template, correcting a rope scaling factor, or setting a default context length:\n\n```bash\npython3 llama.cpp/gguf-py/scripts/gguf_new_metadata.py in.gguf out.gguf \\\n  --chat-template-config template.jinja\n```\n\nThis matters because a surprising number of community quants ship with a template copied from a different model. The weights are fine; the file's idea of how to talk to them is not. Being able to look and fix it, rather than concluding the model is bad, is most of the value of understanding the format at all.\n\n## Split files\n\nLarge models arrive as `model-00001-of-00009.gguf`. Point the loader at the first shard and it finds the rest. `llama-gguf-split --merge` will combine them if you would rather have one file. The split exists because some filesystems and some download tools still struggle above a few tens of gigabytes."},"check":{"question":"A community GGUF of a well-regarded model gives rambling answers that ignore the system prompt, while the same model from another uploader behaves correctly. Both are Q4_K_M and the same file size. What is the most likely cause and the cheapest fix?","options":[{"text":"The bad quant was produced without an importance matrix, so rare instruction-following weights were damaged; re-download an imatrix build.","correct":false,"tempting":"Names a real quality mechanism, but imatrix differences are subtle at 4 bits and would not produce a system prompt being ignored outright."},{"text":"The tensor data is corrupt from an interrupted upload; verify the checksum and re-download.","correct":false,"tempting":"Corruption of tensor data almost never produces coherent-but-wrong behaviour. It produces gibberish or a load failure."},{"text":"The file carries a wrong chat template in its metadata, which can be dumped to confirm and replaced without re-quantising.","correct":true},{"text":"The two files use different k-quant variants despite the same name, so one has fewer promoted tensors; switch to the _L variant.","correct":false,"tempting":"Assumes a naming inconsistency that would change file size, and attributes a formatting failure to a bit-allocation difference."}]},"prev":{"slug":"quantisation-properly","title":"Quantisation, properly","url":"https://addaly.com/learn/running-models-yourself/quantisation-properly"},"next":{"slug":"k-quants-and-importance-matrices","title":"K-quants and importance matrices","url":"https://addaly.com/learn/running-models-yourself/k-quants-and-importance-matrices"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Inside a GGUF file', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/inside-a-gguf-file (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Inside a GGUF file","courseTitle":"Running Models Yourself","module":{"index":3,"slug":"quantisation-and-formats","title":"Quantisation and model formats"},"answer":null,"keyIdea":"A GGUF is one memory-mappable file carrying weights, tokeniser and chat template together, with different tensors quantised at different bit rates, and its metadata can be inspected and repaired without re-quantising anything.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"one-file-everything-in-it","runs":[{"text":"One file, everything in it"}]},{"t":"p","runs":[{"text":"GGUF is the container format used by llama.cpp and everything built on it. Its design goal was simple: a single file that a program can open and run with no accompanying Python, no config directory, no tokeniser package, and no code execution."}]},{"t":"p","runs":[{"text":"The layout is: a magic number and version, then a key-value metadata section, then a tensor index, then the tensor data itself, aligned so it can be memory-mapped directly."}]},{"t":"p","runs":[{"text":"The metadata is the interesting part. It carries the architecture name, layer counts, head counts, rope parameters, the full tokeniser (vocabulary, merges, special token ids) and the chat template as a Jinja string. That is why "},{"text":"llama-server -m model.gguf","code":true},{"text":" needs nothing else, and why a GGUF from 2024 still runs today."}]},{"t":"p","runs":[{"text":"Inspect it:"}]},{"t":"code","lang":"bash","label":"Bash","text":"python3 llama.cpp/gguf-py/scripts/gguf_dump.py model.gguf --no-tensors"},{"t":"p","runs":[{"text":"You will see entries such as "},{"text":"llama.context_length","code":true},{"text":", "},{"text":"llama.attention.head_count_kv","code":true},{"text":", "},{"text":"tokenizer.chat_template","code":true},{"text":" and "},{"text":"general.quantization_version","code":true},{"text":". Every number you need for the memory sum is there, on your own disk, with no network call."}]},{"t":"h2","id":"why-one-file-matters-more-than-it-sounds","runs":[{"text":"Why one file matters more than it sounds"}]},{"t":"p","runs":[{"text":"The alternative format, "},{"text":".safetensors","code":true},{"text":" plus a directory of JSON, requires a Python environment with a matching transformers version to interpret it. That is fine for a workstation and unhelpful on a phone, in a container you did not build, or in five years."}]},{"t":"p","runs":[{"text":"Safetensors deserves its own note. It replaced an older serialisation format whose files were not plain data: loading one ran code embedded in it, so a weights file from an unknown uploader was effectively a program. A "},{"text":".safetensors","code":true},{"text":" file is a header plus raw tensors, and a GGUF likewise. If you download weights from strangers, and everybody does, preferring these two formats is a genuine security control rather than a preference."}]},{"t":"h2","id":"not-every-tensor-gets-the-same-treatment","runs":[{"text":"Not every tensor gets the same treatment"}]},{"t":"p","runs":[{"text":"A GGUF’s name describes the dominant scheme, not a uniform one. In "},{"text":"Q4_K_M","code":true},{"text":", most weights are 4-bit k-quant, but the attention value projections and the feed-forward down projections in half the layers are 6-bit, and the token embedding and output layers get more bits again. Dump the tensors and you can see it:"}]},{"t":"code","lang":"bash","label":"Bash","text":"python3 llama.cpp/gguf-py/scripts/gguf_dump.py model.gguf | head -40"},{"t":"p","runs":[{"text":"The reason is empirical. Some tensors are far more sensitive to rounding than others, and spending bits where they matter is worth more than spending them evenly. The output layer is the extreme case: an error there lands directly on the token probabilities with nothing downstream to absorb it."}]},{"t":"h2","id":"reading-the-suffix-letters","runs":[{"text":"Reading the suffix letters"}]},{"t":"p","runs":[{"text":"The naming looks arbitrary and is not."}]},{"t":"ul","items":[[{"text":"The number is the nominal bits per weight."}],[{"text":"_0","code":true},{"text":" and "},{"text":"_1","code":true},{"text":" are the original simple block schemes: "},{"text":"_1","code":true},{"text":" carries an offset as well as a scale, so it handles asymmetric distributions better and costs slightly more."}],[{"text":"_K","code":true},{"text":" means k-quant: super-blocks with hierarchical scales and mixed precision per tensor type."}],[{"text":"_S","code":true},{"text":", "},{"text":"_M","code":true},{"text":", "},{"text":"_L","code":true},{"text":" are small, medium and large variants of a k-quant, differing in how many tensors are promoted to a higher bit rate. "},{"text":"M","code":true},{"text":" is the usual default."}],[{"text":"IQ","code":true},{"text":" means i-quant: a different scheme using lattice-style codebooks, designed for very low bit rates and requiring an importance matrix to be worth anything."}]]},{"t":"p","runs":[{"text":"One practical warning about i-quants: they need more arithmetic per weight to reconstruct, so on a slow CPU they can generate more slowly than a larger "},{"text":"_K","code":true},{"text":" quant despite being smaller. On a GPU that cost disappears. Measure rather than assuming smaller is faster."}]},{"t":"h2","id":"where-the-metadata-gets-edited","runs":[{"text":"Where the metadata gets edited"}]},{"t":"p","runs":[{"text":"GGUF metadata can be changed without re-quantising, which is occasionally exactly what you need: fixing a wrong chat template, correcting a rope scaling factor, or setting a default context length:"}]},{"t":"code","lang":"bash","label":"Bash","text":"python3 llama.cpp/gguf-py/scripts/gguf_new_metadata.py in.gguf out.gguf \\\n  --chat-template-config template.jinja"},{"t":"p","runs":[{"text":"This matters because a surprising number of community quants ship with a template copied from a different model. The weights are fine; the file’s idea of how to talk to them is not. Being able to look and fix it, rather than concluding the model is bad, is most of the value of understanding the format at all."}]},{"t":"h2","id":"split-files","runs":[{"text":"Split files"}]},{"t":"p","runs":[{"text":"Large models arrive as "},{"text":"model-00001-of-00009.gguf","code":true},{"text":". Point the loader at the first shard and it finds the rest. "},{"text":"llama-gguf-split --merge","code":true},{"text":" will combine them if you would rather have one file. The split exists because some filesystems and some download tools still struggle above a few tens of gigabytes."}]}],"check":{"question":[{"text":"A community GGUF of a well-regarded model gives rambling answers that ignore the system prompt, while the same model from another uploader behaves correctly. Both are Q4_K_M and the same file size. What is the most likely cause and the cheapest fix?"}],"options":[{"text":[{"text":"The bad quant was produced without an importance matrix, so rare instruction-following weights were damaged; re-download an imatrix build."}],"correct":false,"tempting":[{"text":"Names a real quality mechanism, but imatrix differences are subtle at 4 bits and would not produce a system prompt being ignored outright."}]},{"text":[{"text":"The tensor data is corrupt from an interrupted upload; verify the checksum and re-download."}],"correct":false,"tempting":[{"text":"Corruption of tensor data almost never produces coherent-but-wrong behaviour. It produces gibberish or a load failure."}]},{"text":[{"text":"The file carries a wrong chat template in its metadata, which can be dumped to confirm and replaced without re-quantising."}],"correct":true,"tempting":null},{"text":[{"text":"The two files use different k-quant variants despite the same name, so one has fewer promoted tensors; switch to the _L variant."}],"correct":false,"tempting":[{"text":"Assumes a naming inconsistency that would change file size, and attributes a formatting failure to a bit-allocation difference."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"quantisation-properly","title":"Quantisation, properly","minutes":9},"next":{"slug":"k-quants-and-importance-matrices","title":"K-quants and importance matrices","minutes":9}}}