Formats, and what conversion costs you

Five formats you will meet

safetensors is the canonical distribution format: a JSON header of tensor names, shapes and byte offsets, then the raw data. No code, memory-mappable, sharded across files for large models. This is what a repository publishes and what transformers, vLLM and most training code load.

GGUF is llama.cpp’s single-file format. It holds the weights, the quantisation scheme, the tokenizer and the chat template in one file, which is why an Ollama or llama.cpp model is a single download with no dependencies. It is the format of local inference on CPUs and consumer hardware.

MLX is Apple’s array framework format, for Apple Silicon. It uses unified memory well and is usually the fastest option on a Mac.

ONNX is a portable computation graph, used to run models in environments without PyTorch: browsers via WebAssembly, mobile runtimes, and various embedded stacks.

ExecuTorch and TensorRT-LLM engines are ahead-of-time compiled artefacts for phones and Nvidia servers respectively. They are built for one target and one configuration, and they are not portable.

There are also packed quantised checkpoints (GPTQ, AWQ, and bitsandbytes-quantised saves) which are safetensors files containing already-quantised weights plus scale tensors, loaded by a matching kernel.

Converting, in practice

The common direction is safetensors to GGUF:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
pip install -r requirements.txt

python convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf --outtype f16
./llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

Two stages, deliberately: a lossless conversion to a 16-bit GGUF, then quantisation from it. Keep the intermediate if you have the disk, because producing a different quantisation later is then minutes rather than a re-conversion.

For Apple Silicon:

pip install mlx-lm
mlx_lm.convert --hf-path Qwen/Qwen3-8B -q --q-bits 4 --mlx-path ./qwen3-8b-mlx

What conversion breaks

Conversion is where silent damage enters the ecosystem, and it is worth knowing the three ways.

The chat template. GGUF stores the template in its metadata. If the conversion script does not understand a newer template, it can write a default one, and every user of that file then gets slightly wrong formatting with no indication. This is the most common defect in third-party GGUF uploads. Check it:

./llama-server -m model.gguf --verbose 2>&1 | grep -i "chat template"

Special tokens and stop conditions. A model with several end-of-turn tokens can lose one in conversion, which produces generation that will not stop. The next lesson is about diagnosing exactly this.

Architecture support. A converter that does not recognise an architecture may map it onto the nearest one it knows. The file loads, produces text, and is subtly wrong. If the output is fluent but the model seems dumber than its reputation, this is a candidate.

The safe habit is to prefer conversions published by the model’s own organisation, and where you use a third party, use one whose uploads are widely used and who states the source revision. When you convert yourself, verify: run the same five prompts through the original with transformers at temperature 0 and through the converted file at temperature 0, and read both. They will not be token-identical (different kernels, different arithmetic order) but they should be recognisably the same model.

Which format for which job

• Training or fine-tuning: safetensors, full precision.
• Serving on a GPU to many users: safetensors with vLLM, or an AWQ or fp8 quantised checkpoint.
• One person on a laptop or a CPU server: GGUF.
• A Mac: MLX first, GGUF second.
• A phone or a browser: ONNX or ExecuTorch, and expect to spend real time on it.

You will end up with more than one copy of the same model in different formats. That is normal and it is why the arithmetic on download sizes earlier in the course was worth doing.