Superposition: more features than dimensions

The puzzle left over from module two

The feed-forward lesson reported that individual neurons respond to unrelated collections of things (legal citations, chess notation, a kind of German compound) and named the leading explanation, superposition, in a paragraph. This module needs more than a paragraph, because every method in it depends on what superposition implies about where to look.

What a feature is

A feature is a property of the input that the model tracks: this text is code, this token is a person’s name, this sentence is in French, the statement just made was negated, the previous word was an adjective. Operationally, a feature is a direction in the space of activations. Project the residual-stream vector onto that direction and you get a number that reads as how strongly the feature is present.

That definition carries an assumption, usually called the linear representation hypothesis: that features are directions and their strength is a projection, so that adding two features’ directions represents both at once. It is an assumption with a great deal of evidence behind it and some known exceptions, and this module treats it as a working model rather than a law.

Why a model would pack in more features than it has room for

The residual stream of an 8-billion-parameter model has 4,096 dimensions. The model tracks far more than 4,096 distinguishable things about text. So features cannot each have a dimension to themselves.

The way out is geometry. In 4,096 dimensions you can place enormously many vectors that are nearly orthogonal: pairwise dot products small but not zero. If features are sparse, meaning that any given input activates only a few of them, then storing them along nearly orthogonal directions works: the interference between features that are rarely active together is small noise, and the model gains capacity in exchange. That arrangement is superposition, and it means a neuron, one basis direction, is crossed by many features, and each feature is spread across many neurons.

The toy model

A 2022 paper made this reproducible. Train a tiny network to reconstruct n sparse features through a bottleneck of m dimensions, with m smaller than n, and vary how sparse the features are.

# sketch: n features, m < n dimensions, sparsity s
x = (torch.rand(batch, n) < s) * torch.rand(batch, n)   # sparse inputs
h = x @ W                                                # n -> m
x_hat = torch.relu(h @ W.T + b)                          # m -> n
loss = ((x - x_hat) ** 2 * importance).sum()

When features are dense, active most of the time, the network keeps exactly m of them, one per dimension, and drops the rest. As sparsity increases it starts packing more than m features in, arranging their directions into regular geometric shapes (pairs, triangles, pentagons) that minimise interference. Nothing told it to do this; it is the optimal solution to “represent as much as possible with limited room and rare co-occurrence”. The whole experiment runs in a free notebook in minutes, and it is worth running once, because it turns a slogan into something you have watched happen.

Four consequences

1. Neurons are the wrong unit. A neuron is one axis of the coordinate system. Features live along other directions. Asking what a neuron does is asking what several unrelated features have in common, which is why the answer is “legal citations and chess”.
2. Interference is a source of error. When two features that share directions are active together, each reads out slightly wrong. This is a candidate mechanism for some of the strange, input-specific mistakes of module six: the model’s own bookkeeping is lossy by design.
3. Interpretation means finding directions. The next three lessons are three ways of doing that: ask the activations a labelled question, learn a dictionary of directions without labels, and intervene to see which directions matter.
4. Width helps, partly. A larger residual stream can hold more features with less crowding, which is one reason larger models are cleaner to interpret per feature. It does not remove the effect, because the number of features the model wants to track grows too.

The evidence, and the edges of it

The support is now considerable. Linear probes find features as directions, and the same directions turn up in different models. Sparse dictionaries recover thousands of clean, single-meaning features from activations that looked hopeless at the neuron level. Adding a direction to the activations changes behaviour in the way the feature’s meaning predicts. Each of those is a lesson to come.

The exceptions are real. In 2024 features were found that are not single directions but small circular structures: the days of the week arranged in a ring, so that “Tuesday plus two” lands on Thursday. Some features appear to be represented in low-dimensional subspaces rather than lines. The hypothesis that everything is a direction is therefore incomplete, and the honest position is that most of what has been found is linear, some of it is not, and nobody knows the proportions.

Why this matters outside research

Because the rest of this module is only usable if you have this picture. When a later lesson says “the refusal direction” or “a truth probe” or “amplify the feature”, it means a direction in activation space that a model packed in among thousands of others. That is what you are reading, what you are steering, and what you would have to remove to unlearn something. Every claim about what a model “knows” that you will meet from here on is a claim about such directions, and how well they can be found.