{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"reading-config-json","title":"Every number in config.json","url":"https://addaly.com/learn/open-models/reading-config-json","position":22,"of":71,"minutes":10,"answer":"config.json contains everything needed to derive a model's parameter count, its weight memory and its KV cache per token, and most of the parameters live in the feed-forward matrices rather than in attention.","keyIdea":"config.json contains everything needed to derive a model's parameter count, its weight memory and its KV cache per token, and most of the parameters live in the feed-forward matrices rather than in attention."},"body":{"format":"markdown","text":"## Twelve numbers that decide everything\n\nHere is a real configuration, lightly trimmed, from a Llama-3.1-8B repository:\n\n```json\n{\n  \"hidden_size\": 4096,\n  \"intermediate_size\": 14336,\n  \"num_hidden_layers\": 32,\n  \"num_attention_heads\": 32,\n  \"num_key_value_heads\": 8,\n  \"vocab_size\": 128256,\n  \"max_position_embeddings\": 131072,\n  \"rope_theta\": 500000.0,\n  \"rms_norm_eps\": 1e-05,\n  \"tie_word_embeddings\": false,\n  \"torch_dtype\": \"bfloat16\",\n  \"architectures\": [\"LlamaForCausalLM\"]\n}\n```\n\nEverything you need to size, serve and budget for this model is in that block. Learning to read it takes twenty minutes and pays for itself the first time somebody claims a model will fit on a machine where it will not.\n\n## Deriving the parameter count\n\nWork per layer, then multiply.\n\n**Attention.** Four projection matrices. The query projection is `hidden_size × hidden_size` = 4096 × 4096 = 16.8M. The output projection is the same, 16.8M. The key and value projections are smaller, because `num_key_value_heads` is 8 rather than 32: each is `hidden_size × (num_key_value_heads × head_dim)` where `head_dim = hidden_size / num_attention_heads = 128`. So 4096 × 1024 = 4.2M each, 8.4M for the pair.\n\n**Feed-forward.** Three matrices in a gated design: gate, up and down, each `hidden_size × intermediate_size` = 4096 × 14336 = 58.7M. Together 176.2M.\n\n**Per layer:** 16.8 + 16.8 + 8.4 + 176.2 = 218.2M. Times 32 layers = **6.98 billion**.\n\n**Embeddings.** The input table is `vocab_size × hidden_size` = 128256 × 4096 = 525M. Because `tie_word_embeddings` is false, the output head is a separate matrix of the same size, another 525M.\n\n**Total:** 6.98 + 0.53 + 0.53 = **8.04 billion**, which is why it is called an 8B. At bfloat16, two bytes each, that is about 16 GB of weights.\n\nNotice what the feed-forward layers did: 176 of the 218 million parameters per layer, roughly 80% of the model. Most of a transformer is the feed-forward stack, not the attention everybody talks about.\n\n> Figure: Where the parameters are, per layer, in Llama-3.1-8B. Bars (million parameters in one layer): Feed-forward: gate, up and down 176.2; Query projection 16.8; Output projection 16.8; Key and value projections together 8.4. Two hundred and eighteen million per layer, of which the feed-forward stack is about eighty per cent. Most of a transformer is the part nobody talks about. The key and value projections are small because num_key_value_heads is 8 against 32 attention heads, and that same field makes the cache four times smaller.\n\n## What each number tells you operationally\n\n`num_key_value_heads` **less than** `num_attention_heads` means grouped-query attention, which shrinks the KV cache by the ratio: here by a factor of four. This single field is the difference between a model you can serve to twenty users and one you cannot.\n\n`vocab_size` matters far more at small sizes than at large ones. A 128k vocabulary costs 525M parameters at hidden size 4096. In a 1B model with hidden size 2048, a 256k vocabulary costs 524M of embeddings alone: half the model spent before any layer exists. That is the trade a large multilingual vocabulary makes, and it is why very small models often have small vocabularies and tokenise other languages badly.\n\n`max_position_embeddings` is the maximum the architecture will accept, not a promise about quality. `rope_theta` at 500,000 rather than the older 10,000 is the signal that the positional encoding was set up for long context.\n\n`intermediate_size` divided by `hidden_size` is usually between 2.7 and 4. An unusual ratio means an unusual architecture, and is worth a moment's attention before you assume standard tooling will load it.\n\n`torch_dtype` tells you what the weights are stored as, and `architectures` tells your loader which class to build. An architecture name your library has never heard of is the situation that leads people to enable remote code execution, so it is worth noticing here rather than at the traceback.\n\n## Do it yourself in one command\n\n```bash\npython - <<'EOF'\nimport json, urllib.request\nu = \"https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json\"\nc = json.load(urllib.request.urlopen(u))\nh, L = c[\"hidden_size\"], c[\"num_hidden_layers\"]\nkv, heads = c[\"num_key_value_heads\"], c[\"num_attention_heads\"]\nhd = h // heads\nattn = 2*h*h + 2*h*(kv*hd)\nmlp  = 3*h*c[\"intermediate_size\"]\nemb  = c[\"vocab_size\"]*h * (1 if c.get(\"tie_word_embeddings\") else 2)\nprint(f\"params ~ {(L*(attn+mlp)+emb)/1e9:.2f}B\")\nprint(f\"KV bytes/token @fp16 = {2*L*kv*hd*2:,}\")\nEOF\n```\n\nThe second line is the number people most often get wrong and the subject of a later lesson. For this 8B model it is 131,072 bytes (128 KB per token) so a 32,000-token conversation carries about 4.2 GB of cache alongside the 16 GB of weights.\n\n> The habit worth forming: before you download anything large, fetch the 2 KB config file and do this arithmetic. It costs nothing and it has stopped many pointless downloads."},"check":{"question":"Two 1B models are identical except that one has vocab_size 32000 and the other 256000, both with hidden_size 2048 and untied embeddings. What is the practical consequence?","options":[{"text":"The large-vocabulary model has far more transformer capacity, since a bigger vocabulary means more expressive layers.","correct":false,"tempting":"Reverses the relationship: at a fixed total size, embedding parameters are taken away from the layers, not added to them."},{"text":"The large-vocabulary model spends about a billion parameters on embedding tables, leaving much less for the layers, in exchange for tokenising more languages efficiently.","correct":true},{"text":"Vocabulary size affects only the tokenizer file, so the two models have the same parameter budget.","correct":false,"tempting":"Treats the vocabulary as tokenizer-only, but the embedding and output matrices are both sized by it and live in the weights."},{"text":"The larger vocabulary increases the KV cache proportionally, so long contexts cost four times as much.","correct":false,"tempting":"KV cache depends on layers, key-value heads and head dimension; vocabulary size does not appear in it."}]},"prev":{"slug":"base-instruct-reasoning","title":"Base, Instruct, Reasoning","url":"https://addaly.com/learn/open-models/base-instruct-reasoning"},"next":{"slug":"tokenizers-across-families","title":"Why the same sentence costs different amounts","url":"https://addaly.com/learn/open-models/tokenizers-across-families"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Every number in config.json', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/reading-config-json (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Every number in config.json","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"config.json contains everything needed to derive a model’s parameter count, its weight memory and its KV cache per token, and most of the parameters live in the feed-forward matrices rather than in attention.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"million parameters in one layer","title":"Where the parameters are, per layer, in Llama-3.1-8B","series":[{"label":"Feed-forward: gate, up and down","value":176.2,"highlight":true},{"label":"Query projection","value":16.8},{"label":"Output projection","value":16.8},{"label":"Key and value projections together","value":8.4}],"caption":"Two hundred and eighteen million per layer, of which the feed-forward stack is about eighty per cent. Most of a transformer is the part nobody talks about. The key and value projections are small because num_key_value_heads is 8 against 32 attention heads, and that same field makes the cache four times smaller."},"alt":"Figure: Where the parameters are, per layer, in Llama-3.1-8B. Bars (million parameters in one layer): Feed-forward: gate, up and down 176.2; Query projection 16.8; Output projection 16.8; Key and value projections together 8.4. Two hundred and eighteen million per layer, of which the feed-forward stack is about eighty per cent. Most of a transformer is the part nobody talks about. The key and value projections are small because num_key_value_heads is 8 against 32 attention heads, and that same field makes the cache four times smaller."}},"blocks":[{"t":"h2","id":"twelve-numbers-that-decide-everything","runs":[{"text":"Twelve numbers that decide everything"}]},{"t":"p","runs":[{"text":"Here is a real configuration, lightly trimmed, from a Llama-3.1-8B repository:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\n  \"hidden_size\": 4096,\n  \"intermediate_size\": 14336,\n  \"num_hidden_layers\": 32,\n  \"num_attention_heads\": 32,\n  \"num_key_value_heads\": 8,\n  \"vocab_size\": 128256,\n  \"max_position_embeddings\": 131072,\n  \"rope_theta\": 500000.0,\n  \"rms_norm_eps\": 1e-05,\n  \"tie_word_embeddings\": false,\n  \"torch_dtype\": \"bfloat16\",\n  \"architectures\": [\"LlamaForCausalLM\"]\n}"},{"t":"p","runs":[{"text":"Everything you need to size, serve and budget for this model is in that block. Learning to read it takes twenty minutes and pays for itself the first time somebody claims a model will fit on a machine where it will not."}]},{"t":"h2","id":"deriving-the-parameter-count","runs":[{"text":"Deriving the parameter count"}]},{"t":"p","runs":[{"text":"Work per layer, then multiply."}]},{"t":"p","runs":[{"text":"Attention.","b":true},{"text":" Four projection matrices. The query projection is "},{"text":"hidden_size × hidden_size","code":true},{"text":" = 4096 × 4096 = 16.8M. The output projection is the same, 16.8M. The key and value projections are smaller, because "},{"text":"num_key_value_heads","code":true},{"text":" is 8 rather than 32: each is "},{"text":"hidden_size × (num_key_value_heads × head_dim)","code":true},{"text":" where "},{"text":"head_dim = hidden_size / num_attention_heads = 128","code":true},{"text":". So 4096 × 1024 = 4.2M each, 8.4M for the pair."}]},{"t":"p","runs":[{"text":"Feed-forward.","b":true},{"text":" Three matrices in a gated design: gate, up and down, each "},{"text":"hidden_size × intermediate_size","code":true},{"text":" = 4096 × 14336 = 58.7M. Together 176.2M."}]},{"t":"p","runs":[{"text":"Per layer:","b":true},{"text":" 16.8 + 16.8 + 8.4 + 176.2 = 218.2M. Times 32 layers = "},{"text":"6.98 billion","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"Embeddings.","b":true},{"text":" The input table is "},{"text":"vocab_size × hidden_size","code":true},{"text":" = 128256 × 4096 = 525M. Because "},{"text":"tie_word_embeddings","code":true},{"text":" is false, the output head is a separate matrix of the same size, another 525M."}]},{"t":"p","runs":[{"text":"Total:","b":true},{"text":" 6.98 + 0.53 + 0.53 = "},{"text":"8.04 billion","b":true},{"text":", which is why it is called an 8B. At bfloat16, two bytes each, that is about 16 GB of weights."}]},{"t":"p","runs":[{"text":"Notice what the feed-forward layers did: 176 of the 218 million parameters per layer, roughly 80% of the model. Most of a transformer is the feed-forward stack, not the attention everybody talks about."}]},{"t":"h2","id":"what-each-number-tells-you-operationally","runs":[{"text":"What each number tells you operationally"}]},{"t":"p","runs":[{"text":"num_key_value_heads","code":true},{"text":" "},{"text":"less than","b":true},{"text":" "},{"text":"num_attention_heads","code":true},{"text":" means grouped-query attention, which shrinks the KV cache by the ratio: here by a factor of four. This single field is the difference between a model you can serve to twenty users and one you cannot."}]},{"t":"p","runs":[{"text":"vocab_size","code":true},{"text":" matters far more at small sizes than at large ones. A 128k vocabulary costs 525M parameters at hidden size 4096. In a 1B model with hidden size 2048, a 256k vocabulary costs 524M of embeddings alone: half the model spent before any layer exists. That is the trade a large multilingual vocabulary makes, and it is why very small models often have small vocabularies and tokenise other languages badly."}]},{"t":"p","runs":[{"text":"max_position_embeddings","code":true},{"text":" is the maximum the architecture will accept, not a promise about quality. "},{"text":"rope_theta","code":true},{"text":" at 500,000 rather than the older 10,000 is the signal that the positional encoding was set up for long context."}]},{"t":"p","runs":[{"text":"intermediate_size","code":true},{"text":" divided by "},{"text":"hidden_size","code":true},{"text":" is usually between 2.7 and 4. An unusual ratio means an unusual architecture, and is worth a moment’s attention before you assume standard tooling will load it."}]},{"t":"p","runs":[{"text":"torch_dtype","code":true},{"text":" tells you what the weights are stored as, and "},{"text":"architectures","code":true},{"text":" tells your loader which class to build. An architecture name your library has never heard of is the situation that leads people to enable remote code execution, so it is worth noticing here rather than at the traceback."}]},{"t":"h2","id":"do-it-yourself-in-one-command","runs":[{"text":"Do it yourself in one command"}]},{"t":"code","lang":"bash","label":"Bash","text":"python - <<'EOF'\nimport json, urllib.request\nu = \"https://huggingface.co/Qwen/Qwen3-8B/raw/main/config.json\"\nc = json.load(urllib.request.urlopen(u))\nh, L = c[\"hidden_size\"], c[\"num_hidden_layers\"]\nkv, heads = c[\"num_key_value_heads\"], c[\"num_attention_heads\"]\nhd = h // heads\nattn = 2*h*h + 2*h*(kv*hd)\nmlp  = 3*h*c[\"intermediate_size\"]\nemb  = c[\"vocab_size\"]*h * (1 if c.get(\"tie_word_embeddings\") else 2)\nprint(f\"params ~ {(L*(attn+mlp)+emb)/1e9:.2f}B\")\nprint(f\"KV bytes/token @fp16 = {2*L*kv*hd*2:,}\")\nEOF"},{"t":"p","runs":[{"text":"The second line is the number people most often get wrong and the subject of a later lesson. For this 8B model it is 131,072 bytes (128 KB per token) so a 32,000-token conversation carries about 4.2 GB of cache alongside the 16 GB of weights."}]},{"t":"quote","runs":[{"text":"The habit worth forming: before you download anything large, fetch the 2 KB config file and do this arithmetic. It costs nothing and it has stopped many pointless downloads."}]}],"check":{"question":[{"text":"Two 1B models are identical except that one has vocab_size 32000 and the other 256000, both with hidden_size 2048 and untied embeddings. What is the practical consequence?"}],"options":[{"text":[{"text":"The large-vocabulary model has far more transformer capacity, since a bigger vocabulary means more expressive layers."}],"correct":false,"tempting":[{"text":"Reverses the relationship: at a fixed total size, embedding parameters are taken away from the layers, not added to them."}]},{"text":[{"text":"The large-vocabulary model spends about a billion parameters on embedding tables, leaving much less for the layers, in exchange for tokenising more languages efficiently."}],"correct":true,"tempting":null},{"text":[{"text":"Vocabulary size affects only the tokenizer file, so the two models have the same parameter budget."}],"correct":false,"tempting":[{"text":"Treats the vocabulary as tokenizer-only, but the embedding and output matrices are both sized by it and live in the weights."}]},{"text":[{"text":"The larger vocabulary increases the KV cache proportionally, so long contexts cost four times as much."}],"correct":false,"tempting":[{"text":"KV cache depends on layers, key-value heads and head dimension; vocabulary size does not appear in it."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"base-instruct-reasoning","title":"Base, Instruct, Reasoning","minutes":8},"next":{"slug":"tokenizers-across-families","title":"Why the same sentence costs different amounts","minutes":9}}}