The directions a matrix does not turn

A question with a surprisingly useful answer

A matrix generally turns vectors: put an arrow in, get an arrow out pointing somewhere else. But for most matrices there are a few special directions that come out pointing exactly where they went in, only longer or shorter. Those are the eigenvectors, and the stretch factor for each is its eigenvalue.

Written down:

A v = lambda v

Read it as: applying A to v does the same thing as multiplying v by a single number.

Worked, with a matrix simple enough to check:

A = [ 3  1 ]
    [ 0  2 ]

try v = [1, 0]:   A v = [3, 0] = 3 * [1, 0]     eigenvalue 3
try v = [1, -1]:  A v = [3 - 1, -2] = [2, -2] = 2 * [1, -1]   eigenvalue 2
try v = [1, 1]:   A v = [4, 2]                  not a multiple of [1, 1]

Two eigenvectors, eigenvalues 3 and 2. Note that eigenvectors have no fixed length — [2, 0] works as well as [1, 0] — so implementations return unit-length ones by convention.

Why repeated application is the point

Eigenvectors matter because they tell you what happens when you apply a matrix many times, and many things in computing are exactly that.

Write any vector as a mixture of eigenvectors. Apply A once and each part scales by its eigenvalue. Apply it n times and each part scales by λ^n. With eigenvalues 3 and 2:

after 1 step:  3   and 2       ratio 1.5
after 10 steps: 59,049 and 1,024   ratio 58
after 30 steps: 2.06e14 and 1.07e9  ratio 192,000

The largest eigenvalue takes over completely. Whatever direction it belongs to is where any repeatedly-multiplied vector ends up pointing. That is the whole mechanism behind:

• The power method, which finds the top eigenvector by multiplying a random vector by A repeatedly and normalising. Five lines of code, no library.
• PageRank, which is the top eigenvector of a link matrix. Google's original algorithm was, in its arithmetic, this lesson.
• Whether a recurrent network's memory decays or explodes. If the largest eigenvalue of the recurrent weight matrix is below 1, signals shrink towards nothing over time steps; above 1, they blow up. That number is called the spectral radius, and the difficulty of keeping it near 1 is the reason LSTMs and later transformers exist.

import numpy as np
v = np.random.randn(A.shape[0])
for _ in range(50):
    v = A @ v
    v /= np.linalg.norm(v)
print(v, v @ A @ v)     # top eigenvector and eigenvalue

What eigenvalues mean when they are strange

Complex eigenvalues mean rotation. A pure 90-degree rotation matrix has no real eigenvector at all, which makes sense: it turns every direction, so no direction survives unturned. Its eigenvalues are i and −i.

Negative eigenvalues mean the direction is preserved but flipped.

Zero eigenvalues mean that direction is annihilated — the matrix is singular, and the count of zero eigenvalues is how many dimensions were destroyed.

Symmetric matrices behave beautifully. If A = A^T, all eigenvalues are real and the eigenvectors are mutually perpendicular. Covariance matrices, Hessians and graph Laplacians are all symmetric, which is why the tidy version of this theory covers most of the cases you meet. Use np.linalg.eigh for those, not eig: it is faster and returns sorted real results.

Where they are and are not used

Eigen-decomposition sits under PCA (eigenvectors of the covariance matrix), spectral clustering (eigenvectors of a graph Laplacian), and the analysis of optimisation landscapes (eigenvalues of the Hessian give the curvature in each direction, and their ratio is the condition number that decides how badly plain gradient descent will zigzag).

It is not used inside transformer training, and it is worth being clear about that so you do not go looking for it. There is no eigen-decomposition in a forward pass. Its role is analytical: understanding why training behaves as it does, and building the classical methods that surround the deep model.

The honest limitation

Eigen-decomposition applies only to square matrices, and not even to all of those — some square matrices cannot be diagonalised at all. Most of the matrices you actually care about, weight matrices between layers of different widths, are not square.

The generalisation that works for every matrix without exception is the singular value decomposition, which is the next lesson. If you only have room to keep one of the two, keep that one: SVD covers every matrix, is numerically well behaved, and reduces to the eigen-decomposition for symmetric matrices anyway.

The rule to keep

Eigenvectors answer "what does this matrix do if I apply it forever". When you see a stability question — does this recur, does this converge, does this blow up — the largest eigenvalue is usually the number that decides it.