Here’s my take on the ongoing superposition debate that followed...
Pierre Beckmann@BeckmannPierre
12 views
Oct 22, 2026
~3 min read
Advertisement
1
Here’s my take on the ongoing superposition debate that followed from this great release! I think it helps to distinguish structure-driven superposition from space-saving superposition. 1/
Nick@nickcammarata
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com
2
Start with the null hypothesis: if there were no superposition at all, every feature would get its own orthogonal direction. Latent spaces would act as ledgers where each feature is a wholly separate entry. 2/
3
In real NNs, two genuinely distinct features (atoms that light up for different inputs and push toward different output sequences) are almost never exactly orthogonal, and related features often sit at small angles. I think two different pressures explain this. 3/
4
The first pressure is is to exploit structure/relations. Even with unlimited dimensions, a model would (imo) still place related features at small angles, because this lets it encode similarity and a form of composition. I’ll call this structure-driven superposition. 4/
5
First, similarity. Take the coffee feature: it encodes dispositions about what to say next. If the espresso feature is placed at a small angle to it, it can inherit the desired dispositions (and others can be added to it). 5/
6
Now composition. Coffee can be seen as [hot + brown + liquid + big cup], and espresso as [hot + brown + liquid + small cup]. Since they share many constituents, their directions partly overlap. 6/
7
Is this really superposition, or just composition? If coffee and espresso were combinations of nearly orthogonal base features, we could call it composition. But it’s not clear that such a base exists, so I’ll go with the less demanding notion of structure-driven superposition 7/
8
The second pressure is limited space. Neural networks often have a very constraining width, so features get pushed closer together than structure alone would require. I’ll call this space-saving superposition. 8/
9
Networks can deal with the limited-space pressure in two ways. One is to go a bit further in exploiting relations and structure, e.g. by merging features or putting them at even closer angles. 9/
10
The other is to use geometry tricks: putting unrelated features together because they rarely show up at the same time, e.g. “quantum physics” on the opposite side of “cat.” This kind of packing disrupts elegant structure-based organisation. 10/
11
One big motivation for believing in geometry-tricks was InceptionV1 neurons that seemed to mix unrelated things, like cat faces and car fronts. Nick shows that such a neuron actually detects a shared V-pattern 11/
12
He also shows how seemingly unrelated features can get grouped by training history. So with these two cases, he argues that there might be much less space-driven and geometry-trick superposition than we thought. 12/
13
Nick’s analyses are mostly at the neuron level, e.g. right after an activation function. Activation functions act on each neuron separately (e.g. ReLU cuts off negative values), so the network has a reason to align features with neurons. This is called a privileged basis. 13/
14
In transformers, this means that MLP neurons might be more interpretable than we thought, and that we can build attribution graphs directly on them. For example, Nick traces a medical-diagnosis circuit in Qwen3-32B using only MLP neurons. 14/
15
Things might be different in the residual stream, though. Every att head and MLP reads from and writes to it, so it may need to carry a lot of information. So space-saving superposition and geometry tricks could be more common there. 15/
16
Overall, Nick gives evidence that there are fewer geometry tricks than we thought, at least for MLP neurons. More widely, this points toward superposition being driven more by structure than by geometry tricks. If true, latent spaces are indeed cleaner than prev. expected. 16/
17
Still, I don’t think this is a paradigm shift for mech interp. We’ll still need diff-means, probes and SAEs to find features (in particular for the res stream). But I’m excited to see more attribution graphs built directly on MLP neurons! 17/