{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"weight-formats-and-conversion","title":"Formats, and what conversion costs you","url":"https://addaly.com/learn/open-models/weight-formats-and-conversion","position":29,"of":71,"minutes":8,"answer":"Conversion between weight formats is where chat templates, stop tokens and unrecognised architectures get silently mangled, so prefer official conversions and verify a converted file against the original on the same prompts.","keyIdea":"Conversion between weight formats is where chat templates, stop tokens and unrecognised architectures get silently mangled, so prefer official conversions and verify a converted file against the original on the same prompts."},"body":{"format":"markdown","text":"## Five formats you will meet\n\n**safetensors** is the canonical distribution format: a JSON header of tensor names, shapes and byte offsets, then the raw data. No code, memory-mappable, sharded across files for large models. This is what a repository publishes and what `transformers`, vLLM and most training code load.\n\n**GGUF** is llama.cpp's single-file format. It holds the weights, the quantisation scheme, the tokenizer and the chat template in one file, which is why an Ollama or llama.cpp model is a single download with no dependencies. It is the format of local inference on CPUs and consumer hardware.\n\n**MLX** is Apple's array framework format, for Apple Silicon. It uses unified memory well and is usually the fastest option on a Mac.\n\n**ONNX** is a portable computation graph, used to run models in environments without PyTorch: browsers via WebAssembly, mobile runtimes, and various embedded stacks.\n\n**ExecuTorch** and **TensorRT-LLM engines** are ahead-of-time compiled artefacts for phones and Nvidia servers respectively. They are built for one target and one configuration, and they are not portable.\n\nThere are also packed quantised checkpoints (GPTQ, AWQ, and bitsandbytes-quantised saves) which are safetensors files containing already-quantised weights plus scale tensors, loaded by a matching kernel.\n\n## Converting, in practice\n\nThe common direction is safetensors to GGUF:\n\n```bash\ngit clone https://github.com/ggml-org/llama.cpp && cd llama.cpp\npip install -r requirements.txt\n\npython convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf --outtype f16\n./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M\n```\n\nTwo stages, deliberately: a lossless conversion to a 16-bit GGUF, then quantisation from it. Keep the intermediate if you have the disk, because producing a different quantisation later is then minutes rather than a re-conversion.\n\nFor Apple Silicon:\n\n```bash\npip install mlx-lm\nmlx_lm.convert --hf-path Qwen/Qwen3-8B -q --q-bits 4 --mlx-path ./qwen3-8b-mlx\n```\n\n## What conversion breaks\n\nConversion is where silent damage enters the ecosystem, and it is worth knowing the three ways.\n\n**The chat template.** GGUF stores the template in its metadata. If the conversion script does not understand a newer template, it can write a default one, and every user of that file then gets slightly wrong formatting with no indication. This is the most common defect in third-party GGUF uploads. Check it:\n\n```bash\n./llama-server -m model.gguf --verbose 2>&1 | grep -i \"chat template\"\n```\n\n**Special tokens and stop conditions.** A model with several end-of-turn tokens can lose one in conversion, which produces generation that will not stop. The next lesson is about diagnosing exactly this.\n\n**Architecture support.** A converter that does not recognise an architecture may map it onto the nearest one it knows. The file loads, produces text, and is subtly wrong. If the output is fluent but the model seems dumber than its reputation, this is a candidate.\n\nThe safe habit is to prefer conversions published by the model's own organisation, and where you use a third party, use one whose uploads are widely used and who states the source revision. When you convert yourself, verify: run the same five prompts through the original with `transformers` at temperature 0 and through the converted file at temperature 0, and read both. They will not be token-identical (different kernels, different arithmetic order) but they should be recognisably the same model.\n\n## Which format for which job\n\n- Training or fine-tuning: safetensors, full precision.\n- Serving on a GPU to many users: safetensors with vLLM, or an AWQ or fp8 quantised checkpoint.\n- One person on a laptop or a CPU server: GGUF.\n- A Mac: MLX first, GGUF second.\n- A phone or a browser: ONNX or ExecuTorch, and expect to spend real time on it.\n\nYou will end up with more than one copy of the same model in different formats. That is normal and it is why the arithmetic on download sizes earlier in the course was worth doing."},"check":{"question":"A third-party GGUF of a well-regarded model produces noticeably worse output than the same model served from safetensors, though it loads and generates fluently. What should you check first?","options":[{"text":"The quantisation level, since Q4 always costs a large amount of quality on instruction following.","correct":false,"tempting":"Worth checking, but a Q4 gap on ordinary tasks is usually small; a large gap points at something structural."},{"text":"The chat template and special tokens stored in the GGUF metadata, which conversion scripts can write incorrectly without any error.","correct":true},{"text":"The context length, since GGUF files default to a shorter window than the model supports.","correct":false,"tempting":"A short default context truncates long inputs but does not degrade short-prompt quality."},{"text":"The hardware, since CPU inference produces lower-quality output than GPU inference.","correct":false,"tempting":"Arithmetic order differs slightly between backends, but the same weights and sampler give equivalent quality on either."}]},"prev":{"slug":"multimodal-open-models","title":"Models that can see","url":"https://addaly.com/learn/open-models/multimodal-open-models"},"next":{"slug":"stop-conditions","title":"Why it will not stop talking","url":"https://addaly.com/learn/open-models/stop-conditions"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Formats, and what conversion costs you', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/weight-formats-and-conversion (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Formats, and what conversion costs you","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"Conversion between weight formats is where chat templates, stop tokens and unrecognised architectures get silently mangled, so prefer official conversions and verify a converted file against the original on the same prompts.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"five-formats-you-will-meet","runs":[{"text":"Five formats you will meet"}]},{"t":"p","runs":[{"text":"safetensors","b":true},{"text":" is the canonical distribution format: a JSON header of tensor names, shapes and byte offsets, then the raw data. No code, memory-mappable, sharded across files for large models. This is what a repository publishes and what "},{"text":"transformers","code":true},{"text":", vLLM and most training code load."}]},{"t":"p","runs":[{"text":"GGUF","b":true},{"text":" is llama.cpp’s single-file format. It holds the weights, the quantisation scheme, the tokenizer and the chat template in one file, which is why an Ollama or llama.cpp model is a single download with no dependencies. It is the format of local inference on CPUs and consumer hardware."}]},{"t":"p","runs":[{"text":"MLX","b":true},{"text":" is Apple’s array framework format, for Apple Silicon. It uses unified memory well and is usually the fastest option on a Mac."}]},{"t":"p","runs":[{"text":"ONNX","b":true},{"text":" is a portable computation graph, used to run models in environments without PyTorch: browsers via WebAssembly, mobile runtimes, and various embedded stacks."}]},{"t":"p","runs":[{"text":"ExecuTorch","b":true},{"text":" and "},{"text":"TensorRT-LLM engines","b":true},{"text":" are ahead-of-time compiled artefacts for phones and Nvidia servers respectively. They are built for one target and one configuration, and they are not portable."}]},{"t":"p","runs":[{"text":"There are also packed quantised checkpoints (GPTQ, AWQ, and bitsandbytes-quantised saves) which are safetensors files containing already-quantised weights plus scale tensors, loaded by a matching kernel."}]},{"t":"h2","id":"converting-in-practice","runs":[{"text":"Converting, in practice"}]},{"t":"p","runs":[{"text":"The common direction is safetensors to GGUF:"}]},{"t":"code","lang":"bash","label":"Bash","text":"git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp\npip install -r requirements.txt\n\npython convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf --outtype f16\n./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M"},{"t":"p","runs":[{"text":"Two stages, deliberately: a lossless conversion to a 16-bit GGUF, then quantisation from it. Keep the intermediate if you have the disk, because producing a different quantisation later is then minutes rather than a re-conversion."}]},{"t":"p","runs":[{"text":"For Apple Silicon:"}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install mlx-lm\nmlx_lm.convert --hf-path Qwen/Qwen3-8B -q --q-bits 4 --mlx-path ./qwen3-8b-mlx"},{"t":"h2","id":"what-conversion-breaks","runs":[{"text":"What conversion breaks"}]},{"t":"p","runs":[{"text":"Conversion is where silent damage enters the ecosystem, and it is worth knowing the three ways."}]},{"t":"p","runs":[{"text":"The chat template.","b":true},{"text":" GGUF stores the template in its metadata. If the conversion script does not understand a newer template, it can write a default one, and every user of that file then gets slightly wrong formatting with no indication. This is the most common defect in third-party GGUF uploads. Check it:"}]},{"t":"code","lang":"bash","label":"Bash","text":"./llama-server -m model.gguf --verbose 2>&1 | grep -i \"chat template\""},{"t":"p","runs":[{"text":"Special tokens and stop conditions.","b":true},{"text":" A model with several end-of-turn tokens can lose one in conversion, which produces generation that will not stop. The next lesson is about diagnosing exactly this."}]},{"t":"p","runs":[{"text":"Architecture support.","b":true},{"text":" A converter that does not recognise an architecture may map it onto the nearest one it knows. The file loads, produces text, and is subtly wrong. If the output is fluent but the model seems dumber than its reputation, this is a candidate."}]},{"t":"p","runs":[{"text":"The safe habit is to prefer conversions published by the model’s own organisation, and where you use a third party, use one whose uploads are widely used and who states the source revision. When you convert yourself, verify: run the same five prompts through the original with "},{"text":"transformers","code":true},{"text":" at temperature 0 and through the converted file at temperature 0, and read both. They will not be token-identical (different kernels, different arithmetic order) but they should be recognisably the same model."}]},{"t":"h2","id":"which-format-for-which-job","runs":[{"text":"Which format for which job"}]},{"t":"ul","items":[[{"text":"Training or fine-tuning: safetensors, full precision."}],[{"text":"Serving on a GPU to many users: safetensors with vLLM, or an AWQ or fp8 quantised checkpoint."}],[{"text":"One person on a laptop or a CPU server: GGUF."}],[{"text":"A Mac: MLX first, GGUF second."}],[{"text":"A phone or a browser: ONNX or ExecuTorch, and expect to spend real time on it."}]]},{"t":"p","runs":[{"text":"You will end up with more than one copy of the same model in different formats. That is normal and it is why the arithmetic on download sizes earlier in the course was worth doing."}]}],"check":{"question":[{"text":"A third-party GGUF of a well-regarded model produces noticeably worse output than the same model served from safetensors, though it loads and generates fluently. What should you check first?"}],"options":[{"text":[{"text":"The quantisation level, since Q4 always costs a large amount of quality on instruction following."}],"correct":false,"tempting":[{"text":"Worth checking, but a Q4 gap on ordinary tasks is usually small; a large gap points at something structural."}]},{"text":[{"text":"The chat template and special tokens stored in the GGUF metadata, which conversion scripts can write incorrectly without any error."}],"correct":true,"tempting":null},{"text":[{"text":"The context length, since GGUF files default to a shorter window than the model supports."}],"correct":false,"tempting":[{"text":"A short default context truncates long inputs but does not degrade short-prompt quality."}]},{"text":[{"text":"The hardware, since CPU inference produces lower-quality output than GPU inference."}],"correct":false,"tempting":[{"text":"Arithmetic order differs slightly between backends, but the same weights and sampler give equivalent quality on either."}]}],"order":[2,3,1,0],"why":null},"prev":{"slug":"multimodal-open-models","title":"Models that can see","minutes":9},"next":{"slug":"stop-conditions","title":"Why it will not stop talking","minutes":8}}}