Counting the parameters yourself

Four numbers, one calculation

Open any open-weights model on Hugging Face and click config.json. You will find something like this:

{
  "hidden_size": 4096,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "intermediate_size": 14336,
  "vocab_size": 128256
}

From those you can derive the model’s size, its memory needs, and roughly what it will cost to run. Nobody teaches this and it takes ten minutes.

The embedding table

vocab_size × hidden_size = 128,256 × 4,096 ≈ 525 million parameters. The output layer that converts the final vector back into one score per vocabulary entry has the same shape, and in this model it is a separate matrix, so that is another 525 million. Some architectures tie the two (using one matrix for both) which saves the memory at a small cost in quality, and is common in small models where the vocabulary is a large fraction of the total.

Already, in an 8-billion-parameter model, over a billion parameters are spent on the vocabulary.

Attention, per layer

Four projections: query, key, value and output. If all were full width, that would be 4 × hidden² = 4 × 4,096² ≈ 67 million per layer.

But look at num_key_value_heads: 8 against num_attention_heads: 32. That is grouped-query attention: 32 query heads share 8 sets of keys and values. So the key and value projections are a quarter of the width. The count becomes hidden² for the query, hidden² for the output, and 2 × hidden² / 4 for key and value: about 2.5 × 4,096² ≈ 42 million per layer.

The feed-forward half, per layer

intermediate_size: 14336 against a width of 4,096: roughly 3.5× rather than the classic 4×, because this architecture uses a gated activation which needs three matrices instead of two. Up-projection, gate-projection and down-projection: 3 × 4,096 × 14,336 ≈ 176 million per layer.

Note the ratio. The feed-forward half is four times the attention half. That is the general rule: most parameters are not in attention.

Totting up

Per layer: 42 + 176 ≈ 218 million. Times 32 layers ≈ 7.0 billion. Plus the two vocabulary matrices, about 1.05 billion. Total ≈ 8.0 billion, which is what the model is called.

Do this once for a model you use, and the phrase “8B” stops being a brand and becomes a shape you can reason about.

From parameters to memory

Multiply by the bytes per parameter:

• fp32, 4 bytes: 32 GB. Nobody serves models this way.
• bf16 or fp16, 2 bytes: 16 GB. The standard for serving on a GPU.
• int8, 1 byte: 8 GB.
• 4-bit, about 0.5 bytes plus overhead: 4.5 to 5 GB, which is why an 8B model runs on a laptop with 8 GB of free RAM.

The rule of thumb worth memorising: parameters in billions × 2 = gigabytes at fp16, and × 0.6 = gigabytes at 4-bit. Then add the key-value cache, which grows with your context length and is covered in module seven.

From parameters to compute

Two more rules that let you sanity-check any claim:

• A forward pass costs roughly 2 × N floating-point operations per token, where N is the parameter count. So one token through an 8B model is about 16 billion operations.
• Training costs roughly 6 × N × D, where D is the number of training tokens: the factor of 6 comes from one forward and one backward pass. An 8B model on 15 trillion tokens is about 7 × 10²³ operations. On hardware delivering a few hundred teraflops sustained, that is millions of GPU-hours, which is why nobody retrains a model to fix a typo in its knowledge.

Why this changes decisions

When somebody proposes fine-tuning a 70B model on a laptop, you now know it needs 140 GB just to hold the weights at fp16, and roughly three times that for optimiser state during full training. When a vendor quotes a price per million tokens, you can estimate whether it is plausible. When a model claims a large context, you can compute the cache cost.

None of it requires linear algebra. It requires four numbers and multiplication, and it moves you from consumer to practitioner faster than almost anything else in this course.