{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"the-shape-of-a-model","title":"Counting the parameters yourself","url":"https://addaly.com/learn/how-llms-work/the-shape-of-a-model","position":8,"of":78,"minutes":9,"answer":"Parameter count, memory footprint and cost per token all follow from four numbers on a config page, and doing the arithmetic once makes model comparisons concrete rather than vibes.","keyIdea":"Parameter count, memory footprint and cost per token all follow from four numbers on a config page, and doing the arithmetic once makes model comparisons concrete rather than vibes."},"body":{"format":"markdown","text":"## Four numbers, one calculation\n\nOpen any open-weights model on Hugging Face and click `config.json`. You will find something like this:\n\n```json\n{\n  \"hidden_size\": 4096,\n  \"num_hidden_layers\": 32,\n  \"num_attention_heads\": 32,\n  \"num_key_value_heads\": 8,\n  \"intermediate_size\": 14336,\n  \"vocab_size\": 128256\n}\n```\n\nFrom those you can derive the model's size, its memory needs, and roughly what it will cost to run. Nobody teaches this and it takes ten minutes.\n\n## The embedding table\n\n`vocab_size × hidden_size` = 128,256 × 4,096 ≈ **525 million** parameters. The output layer that converts the final vector back into one score per vocabulary entry has the same shape, and in this model it is a separate matrix, so that is another 525 million. Some architectures *tie* the two (using one matrix for both) which saves the memory at a small cost in quality, and is common in small models where the vocabulary is a large fraction of the total.\n\nAlready, in an 8-billion-parameter model, over a billion parameters are spent on the vocabulary.\n\n## Attention, per layer\n\nFour projections: query, key, value and output. If all were full width, that would be 4 × hidden² = 4 × 4,096² ≈ 67 million per layer.\n\nBut look at `num_key_value_heads: 8` against `num_attention_heads: 32`. That is grouped-query attention: 32 query heads share 8 sets of keys and values. So the key and value projections are a quarter of the width. The count becomes hidden² for the query, hidden² for the output, and 2 × hidden² / 4 for key and value: about 2.5 × 4,096² ≈ **42 million** per layer.\n\n## The feed-forward half, per layer\n\n`intermediate_size: 14336` against a width of 4,096: roughly 3.5× rather than the classic 4×, because this architecture uses a gated activation which needs three matrices instead of two. Up-projection, gate-projection and down-projection: 3 × 4,096 × 14,336 ≈ **176 million** per layer.\n\nNote the ratio. The feed-forward half is four times the attention half. That is the general rule: most parameters are not in attention.\n\n## Totting up\n\nPer layer: 42 + 176 ≈ 218 million. Times 32 layers ≈ **7.0 billion**. Plus the two vocabulary matrices, about 1.05 billion. Total ≈ **8.0 billion**, which is what the model is called.\n\n> Figure: Where the eight billion parameters go. Bars (billions of parameters): Embedding table (128,256 × 4,096) 0.53; Output projection, untied 0.53; Attention, all 32 layers 1.34; Feed-forward, all 32 layers 5.63. Every number here comes from six lines of config.json and multiplication. The feed-forward half is four times the attention half, and over a billion parameters go on the vocabulary alone, which is why small models often tie the two vocabulary matrices together.\n\nDo this once for a model you use, and the phrase \"8B\" stops being a brand and becomes a shape you can reason about.\n\n## From parameters to memory\n\nMultiply by the bytes per parameter:\n\n- **fp32**, 4 bytes: 32 GB. Nobody serves models this way.\n- **bf16 or fp16**, 2 bytes: **16 GB**. The standard for serving on a GPU.\n- **int8**, 1 byte: 8 GB.\n- **4-bit**, about 0.5 bytes plus overhead: **4.5 to 5 GB**, which is why an 8B model runs on a laptop with 8 GB of free RAM.\n\nThe rule of thumb worth memorising: **parameters in billions × 2 = gigabytes at fp16, and × 0.6 = gigabytes at 4-bit.** Then add the key-value cache, which grows with your context length and is covered in module seven.\n\n## From parameters to compute\n\nTwo more rules that let you sanity-check any claim:\n\n- A forward pass costs roughly **2 × N floating-point operations per token**, where N is the parameter count. So one token through an 8B model is about 16 billion operations.\n- Training costs roughly **6 × N × D**, where D is the number of training tokens: the factor of 6 comes from one forward and one backward pass. An 8B model on 15 trillion tokens is about 7 × 10²³ operations. On hardware delivering a few hundred teraflops sustained, that is millions of GPU-hours, which is why nobody retrains a model to fix a typo in its knowledge.\n\n## Why this changes decisions\n\nWhen somebody proposes fine-tuning a 70B model on a laptop, you now know it needs 140 GB just to hold the weights at fp16, and roughly three times that for optimiser state during full training. When a vendor quotes a price per million tokens, you can estimate whether it is plausible. When a model claims a large context, you can compute the cache cost.\n\nNone of it requires linear algebra. It requires four numbers and multiplication, and it moves you from consumer to practitioner faster than almost anything else in this course."},"check":{"question":"An 8-billion-parameter model at fp16 needs about 16 GB for its weights. Someone wants to serve 30 simultaneous users each with a 32,000-token conversation on a single 24 GB GPU. What does the arithmetic say?","options":[{"text":"The weights fit, but each user's key-value cache also occupies memory that grows with context length, and thirty long conversations will exhaust the remaining 8 GB.","correct":true},{"text":"It fits comfortably, because the weights are loaded once and shared across all users.","correct":false,"tempting":"Weight sharing is real and is the reason batching works at all, so stopping the analysis there feels complete."},{"text":"It will not fit, because each user needs their own copy of the model weights.","correct":false,"tempting":"Per-process copies are a genuine problem in naive deployments, which makes this a believable operational worry."},{"text":"It fits, since context length affects compute time but not memory.","correct":false,"tempting":"Longer prompts obviously take longer, so tying context to time rather than memory is an easy substitution to make."}]},"prev":{"slug":"position-and-order","title":"How the model knows what order the words were in","url":"https://addaly.com/learn/how-llms-work/position-and-order"},"next":{"slug":"what-is-in-a-model-file","title":"What is actually inside a model you download","url":"https://addaly.com/learn/how-llms-work/what-is-in-a-model-file"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Counting the parameters yourself', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/the-shape-of-a-model (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Counting the parameters yourself","courseTitle":"How a Language Model Actually Works","module":{"index":1,"slug":"from-characters-to-vectors","title":"From characters to vectors"},"answer":null,"keyIdea":"Parameter count, memory footprint and cost per token all follow from four numbers on a config page, and doing the arithmetic once makes model comparisons concrete rather than vibes.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"billions of parameters","title":"Where the eight billion parameters go","series":[{"label":"Embedding table (128,256 × 4,096)","value":0.53},{"label":"Output projection, untied","value":0.53},{"label":"Attention, all 32 layers","value":1.34},{"label":"Feed-forward, all 32 layers","value":5.63,"highlight":true}],"caption":"Every number here comes from six lines of config.json and multiplication. The feed-forward half is four times the attention half, and over a billion parameters go on the vocabulary alone, which is why small models often tie the two vocabulary matrices together."},"alt":"Figure: Where the eight billion parameters go. Bars (billions of parameters): Embedding table (128,256 × 4,096) 0.53; Output projection, untied 0.53; Attention, all 32 layers 1.34; Feed-forward, all 32 layers 5.63. Every number here comes from six lines of config.json and multiplication. The feed-forward half is four times the attention half, and over a billion parameters go on the vocabulary alone, which is why small models often tie the two vocabulary matrices together."}},"blocks":[{"t":"h2","id":"four-numbers-one-calculation","runs":[{"text":"Four numbers, one calculation"}]},{"t":"p","runs":[{"text":"Open any open-weights model on Hugging Face and click "},{"text":"config.json","code":true},{"text":". You will find something like this:"}]},{"t":"code","lang":"json","label":"JSON","text":"{\n  \"hidden_size\": 4096,\n  \"num_hidden_layers\": 32,\n  \"num_attention_heads\": 32,\n  \"num_key_value_heads\": 8,\n  \"intermediate_size\": 14336,\n  \"vocab_size\": 128256\n}"},{"t":"p","runs":[{"text":"From those you can derive the model’s size, its memory needs, and roughly what it will cost to run. Nobody teaches this and it takes ten minutes."}]},{"t":"h2","id":"the-embedding-table","runs":[{"text":"The embedding table"}]},{"t":"p","runs":[{"text":"vocab_size × hidden_size","code":true},{"text":" = 128,256 × 4,096 ≈ "},{"text":"525 million","b":true},{"text":" parameters. The output layer that converts the final vector back into one score per vocabulary entry has the same shape, and in this model it is a separate matrix, so that is another 525 million. Some architectures "},{"text":"tie","i":true},{"text":" the two (using one matrix for both) which saves the memory at a small cost in quality, and is common in small models where the vocabulary is a large fraction of the total."}]},{"t":"p","runs":[{"text":"Already, in an 8-billion-parameter model, over a billion parameters are spent on the vocabulary."}]},{"t":"h2","id":"attention-per-layer","runs":[{"text":"Attention, per layer"}]},{"t":"p","runs":[{"text":"Four projections: query, key, value and output. If all were full width, that would be 4 × hidden² = 4 × 4,096² ≈ 67 million per layer."}]},{"t":"p","runs":[{"text":"But look at "},{"text":"num_key_value_heads: 8","code":true},{"text":" against "},{"text":"num_attention_heads: 32","code":true},{"text":". That is grouped-query attention: 32 query heads share 8 sets of keys and values. So the key and value projections are a quarter of the width. The count becomes hidden² for the query, hidden² for the output, and 2 × hidden² / 4 for key and value: about 2.5 × 4,096² ≈ "},{"text":"42 million","b":true},{"text":" per layer."}]},{"t":"h2","id":"the-feed-forward-half-per-layer","runs":[{"text":"The feed-forward half, per layer"}]},{"t":"p","runs":[{"text":"intermediate_size: 14336","code":true},{"text":" against a width of 4,096: roughly 3.5× rather than the classic 4×, because this architecture uses a gated activation which needs three matrices instead of two. Up-projection, gate-projection and down-projection: 3 × 4,096 × 14,336 ≈ "},{"text":"176 million","b":true},{"text":" per layer."}]},{"t":"p","runs":[{"text":"Note the ratio. The feed-forward half is four times the attention half. That is the general rule: most parameters are not in attention."}]},{"t":"h2","id":"totting-up","runs":[{"text":"Totting up"}]},{"t":"p","runs":[{"text":"Per layer: 42 + 176 ≈ 218 million. Times 32 layers ≈ "},{"text":"7.0 billion","b":true},{"text":". Plus the two vocabulary matrices, about 1.05 billion. Total ≈ "},{"text":"8.0 billion","b":true},{"text":", which is what the model is called."}]},{"t":"p","runs":[{"text":"Do this once for a model you use, and the phrase “8B” stops being a brand and becomes a shape you can reason about."}]},{"t":"h2","id":"from-parameters-to-memory","runs":[{"text":"From parameters to memory"}]},{"t":"p","runs":[{"text":"Multiply by the bytes per parameter:"}]},{"t":"ul","items":[[{"text":"fp32","b":true},{"text":", 4 bytes: 32 GB. Nobody serves models this way."}],[{"text":"bf16 or fp16","b":true},{"text":", 2 bytes: "},{"text":"16 GB","b":true},{"text":". The standard for serving on a GPU."}],[{"text":"int8","b":true},{"text":", 1 byte: 8 GB."}],[{"text":"4-bit","b":true},{"text":", about 0.5 bytes plus overhead: "},{"text":"4.5 to 5 GB","b":true},{"text":", which is why an 8B model runs on a laptop with 8 GB of free RAM."}]]},{"t":"p","runs":[{"text":"The rule of thumb worth memorising: "},{"text":"parameters in billions × 2 = gigabytes at fp16, and × 0.6 = gigabytes at 4-bit.","b":true},{"text":" Then add the key-value cache, which grows with your context length and is covered in module seven."}]},{"t":"h2","id":"from-parameters-to-compute","runs":[{"text":"From parameters to compute"}]},{"t":"p","runs":[{"text":"Two more rules that let you sanity-check any claim:"}]},{"t":"ul","items":[[{"text":"A forward pass costs roughly "},{"text":"2 × N floating-point operations per token","b":true},{"text":", where N is the parameter count. So one token through an 8B model is about 16 billion operations."}],[{"text":"Training costs roughly "},{"text":"6 × N × D","b":true},{"text":", where D is the number of training tokens: the factor of 6 comes from one forward and one backward pass. An 8B model on 15 trillion tokens is about 7 × 10²³ operations. On hardware delivering a few hundred teraflops sustained, that is millions of GPU-hours, which is why nobody retrains a model to fix a typo in its knowledge."}]]},{"t":"h2","id":"why-this-changes-decisions","runs":[{"text":"Why this changes decisions"}]},{"t":"p","runs":[{"text":"When somebody proposes fine-tuning a 70B model on a laptop, you now know it needs 140 GB just to hold the weights at fp16, and roughly three times that for optimiser state during full training. When a vendor quotes a price per million tokens, you can estimate whether it is plausible. When a model claims a large context, you can compute the cache cost."}]},{"t":"p","runs":[{"text":"None of it requires linear algebra. It requires four numbers and multiplication, and it moves you from consumer to practitioner faster than almost anything else in this course."}]}],"check":{"question":[{"text":"An 8-billion-parameter model at fp16 needs about 16 GB for its weights. Someone wants to serve 30 simultaneous users each with a 32,000-token conversation on a single 24 GB GPU. What does the arithmetic say?"}],"options":[{"text":[{"text":"The weights fit, but each user’s key-value cache also occupies memory that grows with context length, and thirty long conversations will exhaust the remaining 8 GB."}],"correct":true,"tempting":null},{"text":[{"text":"It fits comfortably, because the weights are loaded once and shared across all users."}],"correct":false,"tempting":[{"text":"Weight sharing is real and is the reason batching works at all, so stopping the analysis there feels complete."}]},{"text":[{"text":"It will not fit, because each user needs their own copy of the model weights."}],"correct":false,"tempting":[{"text":"Per-process copies are a genuine problem in naive deployments, which makes this a believable operational worry."}]},{"text":[{"text":"It fits, since context length affects compute time but not memory."}],"correct":false,"tempting":[{"text":"Longer prompts obviously take longer, so tying context to time rather than memory is an easy substitution to make."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"position-and-order","title":"How the model knows what order the words were in","minutes":8},"next":{"slug":"what-is-in-a-model-file","title":"What is actually inside a model you download","minutes":8}}}