You replace part of a language model with three lines of code. The...

Now try zero lines. π§΅
- take an attention head or MLP
- harvest its inputs and outputs
- fit a small typed program to mimic its behaviour
- replace the reference
If loss rises by < 0.005 nats/token, we have a winner.
Most of the ones that "work" are empty. And this has big implications.
Replacements that kept the model working AND matched the original's output within 5%: 0.
But the shortest program is `return 0`. Wherever a model tolerates ablation, Occam's razor picks deletion.
Description length plus a loss budget measures redundancy. It does not hand you mechanism.
We planted ground truth. Compiled Tracr (courtesy: @GoogleDeepMind) programs inside trained transformers.
There, the same pipeline recovers 120/144 branches faithfully.
Before a replacement can be called an explanation, it must beat deleting the component.
Case in point: DistilGPT2 layer 1, head 7: a rank-2 map with output error 29% adds ~0.0009 nats.
The rest of the networks reads only some directions of what a head writes. Error at the head and error in the final predictions are different quantities.
So what does this mean?
Published work in the most critical fields rely on this test.
Interpretability: supposed to vouch for models, as in, "we isolated the deception circuit, removed it and the model still works."
Compression: "average score unchanged" is supposed to assure you that the model hasn't changed much.
None of the papers do the tests required to tackle all failure modes.
- Circuit papers (11 of 12) test parts together, but only 1 checks the part's own output.
- SAE papers check the output (8 of 12), but none tests parts together
- 36 of the most-cited compression papers, 0 measured whether answers have changed.
_/// Circuit papers that don't check the part's own output (11 of 12)
The circuit meets task metrics but its components compute something different from what the paper describes. This means that roles assigned to components (e.g. "this controls deception") are at risk.
_/// Papers with no deletion comparison (10 of 24 with none at all; 4 with it consistently)
"Our explanation captures the behaviour and here's a recovery score."
Deleting the same components or a random set of the same size, might score about as well. Then the score measures how little those parts were needed.
_/// SAE papers that never test layers together (0 of 12 do)
"Per layer reconstruction quality justifies using SAEs across many layers at once"
Each layer's error passes alone, but the errors compound.
_/// Compression papers with no agreement measure (36 of 36)
"behaves like the original model"
Two methods with the same loss can change very different sets of answers, so emergent behaviour is still unknown.
A lab safety-tests its model at full precision. Then the world downloads a 4-bit copy. Does the copy still refuse what the original refused? Nobody checked.
_/// Compression papers with no uncertainty (26 of 36)
"method A beats method B"
The ranking may be noise.





