Every number in config.json

Twelve numbers that decide everything

Here is a real configuration, lightly trimmed, from a Llama-3.1-8B repository:

{
  "hidden_size": 4096,
  "intermediate_size": 14336,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "vocab_size": 128256,
  "max_position_embeddings": 131072,
  "rope_theta": 500000.0,
  "rms_norm_eps": 1e-05,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "architectures": ["LlamaForCausalLM"]
}

Everything you need to size, serve and budget for this model is in that block. Learning to read it takes twenty minutes and pays for itself the first time somebody claims a model will fit on a machine where it will not.

Deriving the parameter count

Work per layer, then multiply.

Attention. Four projection matrices. The query projection is hidden_size × hidden_size = 4096 × 4096 = 16.8M. The output projection is the same, 16.8M. The key and value projections are smaller, because num_key_value_heads is 8 rather than 32: each is hidden_size × (num_key_value_heads × head_dim) where head_dim = hidden_size / num_attention_heads = 128. So 4096 × 1024 = 4.2M each, 8.4M for the pair.

Feed-forward. Three matrices in a gated design: gate, up and down, each hidden_size × intermediate_size = 4096 × 14336 = 58.7M. Together 176.2M.

Per layer: 16.8 + 16.8 + 8.4 + 176.2 = 218.2M. Times 32 layers = 6.98 billion.

Embeddings. The input table is vocab_size × hidden_size = 128256 × 4096 = 525M. Because tie_word_embeddings is false, the output head is a separate matrix of the same size, another 525M.

Total: 6.98 + 0.53 + 0.53 = 8.04 billion, which is why it is called an 8B. At bfloat16, two bytes each, that is about 16 GB of weights.

Notice what the feed-forward layers did: 176 of the 218 million parameters per layer, roughly 80% of the model. Most of a transformer is the feed-forward stack, not the attention everybody talks about.

What each number tells you operationally

num_key_value_heads less than num_attention_heads means grouped-query attention, which shrinks the KV cache by the ratio: here by a factor of four. This single field is the difference between a model you can serve to twenty users and one you cannot.

vocab_size matters far more at small sizes than at large ones. A 128k vocabulary costs 525M parameters at hidden size 4096. In a 1B model with hidden size 2048, a 256k vocabulary costs 524M of embeddings alone: half the model spent before any layer exists. That is the trade a large multilingual vocabulary makes, and it is why very small models often have small vocabularies and tokenise other languages badly.

max_position_embeddings is the maximum the architecture will accept, not a promise about quality. rope_theta at 500,000 rather than the older 10,000 is the signal that the positional encoding was set up for long context.

intermediate_size divided by hidden_size is usually between 2.7 and 4. An unusual ratio means an unusual architecture, and is worth a moment’s attention before you assume standard tooling will load it.

torch_dtype tells you what the weights are stored as, and architectures tells your loader which class to build. An architecture name your library has never heard of is the situation that leads people to enable remote code execution, so it is worth noticing here rather than at the traceback.

Do it yourself in one command

python - <<'EOF'
import json, urllib.request
u = "https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json"
c = json.load(urllib.request.urlopen(u))
h, L = c["hidden_size"], c["num_hidden_layers"]
kv, heads = c["num_key_value_heads"], c["num_attention_heads"]
hd = h // heads
attn = 2*h*h + 2*h*(kv*hd)
mlp  = 3*h*c["intermediate_size"]
emb  = c["vocab_size"]*h * (1 if c.get("tie_word_embeddings") else 2)
print(f"params ~ {(L*(attn+mlp)+emb)/1e9:.2f}B")
print(f"KV bytes/token @fp16 = {2*L*kv*hd*2:,}")
EOF

The second line is the number people most often get wrong and the subject of a later lesson. For this 8B model it is 131,072 bytes (128 KB per token) so a 32,000-token conversation carries about 4.2 GB of cache alongside the 16 GB of weights.

The habit worth forming: before you download anything large, fetch the 2 KB config file and do this arithmetic. It costs nothing and it has stopped many pointless downloads.