The same vector, written two different ways

Numbers are a description, not the thing

Point at a spot on a table and say where it is. You might say "30 centimetres from the left edge, 20 from the front". Somebody standing on the other side of the table would describe the same spot with different numbers. The spot did not move. The description changed, because the axes changed.

A vector works the same way. When we write [3, 4] we mean 3 lots of one reference direction plus 4 lots of another. Those reference directions are called a basis. The default is the standard basis: e1 = [1, 0] and e2 = [0, 1], so [3, 4] = 3·e1 + 4·e2. That is so automatic that most people never notice a choice was made.

Rewriting in another basis

Choose a different pair of directions, say u1 = [1, 1]/√2 and u2 = [-1, 1]/√2 — the same axes rotated 45 degrees, still unit length and still perpendicular. Since they are orthonormal, the new coordinates are just dot products:

a = [3, 4]
a . u1 = (3 + 4)/sqrt(2) = 7/1.414 = 4.95
a . u2 = (-3 + 4)/sqrt(2) = 1/1.414 = 0.707

So the same arrow is [3, 4] in one basis and [4.95, 0.707] in the other. Check that nothing physical changed: its length in the new basis is sqrt(4.95² + 0.707²) = sqrt(24.5 + 0.5) = 5, exactly the length it had before. Lengths, angles and dot products are all preserved by an orthonormal change of basis. Only the numbers move.

Why this matters for embeddings

People open an embedding, look at dimension 42, and try to work out what it means. Usually it means nothing.

Most training objectives are invariant to rotation of the representation space. If you rotate every embedding by the same orthogonal matrix and rotate the next layer's weights back by the inverse, every dot product, every distance and every output is identical, so the loss is identical. Training has no reason to prefer one rotation over another, so the axes it lands on are arbitrary. A single coordinate is a projection onto an axis nobody chose.

This is why the honest way to inspect an embedding space is through relationships — distances, neighbourhoods, directions between pairs — rather than through individual coordinates. The relationships survive a change of basis; the coordinates do not.

The important exception

There are situations where individual dimensions do mean something, and it is worth knowing which.

After PCA. Principal component analysis chooses a specific basis, ordered by how much variance each direction explains. Component 1 is not arbitrary: it is defined as the direction of greatest spread. Reading it is legitimate.

Where a non-linearity breaks the symmetry. A ReLU acts coordinate by coordinate — it zeroes negative entries in the basis it is given. That destroys rotation invariance for the layer it follows, which is why some interpretability work does find meaningful individual neurons in the hidden layers of a network with element-wise activations, while pure embedding tables show much less of it.

When a sparse autoencoder has been fitted. A recent line of work trains a wide, sparse layer whose job is to find a basis in which features are individually meaningful. Early results look encouraging and the approach is under active debate — how many features are found, whether they are the model's features or the autoencoder's, and how to evaluate any of it are all open. It is a promising research direction, not a solved tool.

The practical consequences

Three things follow directly.

1. Do not compare dimension k of one model with dimension k of another. Two models trained on the same data with different seeds will have unrelated bases. Their vectors are not interchangeable in any way, which is why you must re-embed your whole corpus when you change embedding model, rather than mixing old and new vectors in one index.

1. Do not average embeddings from different models. The arithmetic runs; the result is meaningless, because you are adding coordinates measured against different axes.

1. Concatenating is different from averaging, and is sometimes fine. Sticking a 768-dimension vector from one model next to a 384-dimension vector from another gives a 1,152-dimension vector whose dot product is the sum of two separate dot products. That is a defensible hybrid, provided you scale each half deliberately rather than by accident.

The rule to keep

Coordinates are relative to a basis. Distances, angles and neighbourhoods are not. Build everything you rely on out of the second kind.