Doing a matrix multiplication yourself, once

Do it once and you never fear it again

Matrix multiplication looks like a rule to memorise. It is one sentence: entry (i, j) of the answer is the dot product of row i of the left matrix with column j of the right matrix. Everything else follows from that, including the shape rule and the fact that order matters.

Here is a full worked example. Take

A = [ 1  2  3 ]        B = [ 7   8 ]
    [ 4  5  6 ]            [ 9  10 ]
                           [ 11 12 ]

A is 2×3, B is 3×2. Work out each of the four answer entries.

(1,1): row 1 of A . column 1 of B = 1*7 + 2*9 + 3*11 = 7 + 18 + 33  = 58
(1,2): row 1 of A . column 2 of B = 1*8 + 2*10 + 3*12 = 8 + 20 + 36 = 64
(2,1): row 2 of A . column 1 of B = 4*7 + 5*9 + 6*11 = 28 + 45 + 66 = 139
(2,2): row 2 of A . column 2 of B = 4*8 + 5*10 + 6*12 = 32 + 50 + 72 = 154

So

AB = [ 58   64 ]
     [ 139 154 ]

Do that once, slowly, with a pen. It takes four minutes and it converts a rule into a thing you understand.

The shape rule, and why it is what it is

An (m × n) matrix times an (n × p) matrix gives an (m × p) matrix. Write the shapes next to each other:

(2 x 3) (3 x 2)  ->  (2 x 2)
     ^---^  the inner numbers must match; they vanish
   ^---------^     the outer numbers survive

The inner numbers must match because each answer entry is a dot product, and a dot product needs two lists of equal length. Row i of A has n entries; column j of B has n entries. If they differ, there is nothing to compute. That is the whole explanation for the error message you will see most often in your first month.

Order matters, and it is not a technicality

AB and BA are different operations, and frequently only one of them is even legal. With the A and B above, BA is (3×2)(2×3) = 3×3 — a different size of answer entirely. Even when both are square and both legal, the results usually differ:

[1 1] [1 0]   =  [1 0]        [1 0] [1 1]   =  [1 1]
[0 1] [1 1]      [1 1]        [1 1] [0 1]      [1 2]

Two legal products, two different answers. This matters in practice because it is why a stack of layers is a sequence, not a set: W3 W2 W1 x applies W1 first. Swapping two layers changes the function.

Reading it as columns instead of dots

There is a second reading, and it is the one that makes deep learning make sense. Instead of "dot products", think of a matrix times a vector as a weighted sum of the matrix's columns:

[ 1  2 ] [ 3 ]  =  3 * [1]  +  4 * [2]  =  [ 3 + 8  ]  =  [ 11 ]
[ 4  5 ] [ 4 ]         [4]         [5]     [ 12 + 20]     [ 32 ]

Every output of a linear layer is a mixture of the columns of the weight matrix, with the input supplying the mixing amounts. The set of everything reachable this way is called the column space, and it is exactly what the layer can produce. Anything outside it is unreachable at any input, which is a limitation you will meet again under the word rank.

Cost, in operations you can count

Multiplying an (m × n) by an (n × p) matrix requires m * n * p multiply-add operations, because there are m * p output entries and each costs n multiply-adds. Put numbers in it: a single 4096 × 4096 weight matrix applied to one 4096-vector costs

4096 * 4096 * 1 = 16.7 million multiply-adds

which is why people count a matrix multiplication as roughly 2 * m * n * p floating-point operations — one multiply and one add each. A transformer layer does several of these per token, and a model has dozens of layers. That arithmetic, and nothing more mysterious, is where the electricity goes.

The two habits worth forming

Write the shapes. Before writing any model code, write the shape of every tensor as a comment: (batch, seq, d_model). Most bugs in a first model are shape bugs, and most shape bugs are visible in the comments before the code runs.

Verify small. When you are unsure what an operation does, run it on a 2×3 and a 3×2 you can check by hand. In free NumPy:

import numpy as np
A = np.array([[1, 2, 3], [4, 5, 6]])
B = np.array([[7, 8], [9, 10], [11, 12]])
print(A @ B)          # the 58 / 64 / 139 / 154 above
print((A @ B).shape)  # (2, 2)

Three lines, no GPU, no account. Any claim in this module can be checked the same way, and checking it yourself is the difference between knowing the rule and believing it.