You replace part of a language model with three lines of code. The...

@latakian_
Jackie@latakian_
7 views Nov 18, 2026 ~4 min read
Advertisement
1
You replace part of a language model with three lines of code. The model still works. Looks like you figured out what that bit does.

Now try zero lines. 🧡
2
What exactly happens if you replace parts of a language model with tiny (human-readable) programs?

- take an attention head or MLP
- harvest its inputs and outputs
- fit a small typed program to mimic its behaviour
- replace the reference

If loss rises by < 0.005 nats/token, we have a winner.

Most of the ones that "work" are empty. And this has big implications.
Media image
3
We did this for EVERY component. Not just the promising ones. 790 heads and MLPs across Pythia, DistilGPT2, and OLMo-1B.

Replacements that kept the model working AND matched the original's output within 5%: 0.
4
Perplexingly, plenty did pass the loss test: 70 of 544 in OLMo-1B

48 of the 70 in OLMo were deletions. The model did not need the head, so it had no need of our program either.
Media image
5
We chose the shortest program within a loss budget.

But the shortest program is `return 0`. Wherever a model tolerates ablation, Occam's razor picks deletion.

Description length plus a loss budget measures redundancy. It does not hand you mechanism.
6
Is this just a broken method we have? I wish it was.

We planted ground truth. Compiled Tracr (courtesy: @GoogleDeepMind) programs inside trained transformers.

There, the same pipeline recovers 120/144 branches faithfully.
7
This brings you to the "where's your baseline" reprimand. Always run the deletion baseline. On OLMo, plain deletion is accepted 37 times. Our selected programs were accepted 36 times. A dense rank-8 linear fit got 60.

Before a replacement can be called an explanation, it must beat deleting the component.
Media image
8
Tolerance does not compose. If you combine multiple replacements, the formerly harmless replacements fail get along with the loss budget like a live lobster in your trousers.

On OLMo-1B, 45 swaps that looked harmless during selection cost 0.14 nats/token together.

That's 28x our budget.
Media image
9
The annoying thing is that a singular head can be 29% wrong and nothing downstream notices.

Case in point: DistilGPT2 layer 1, head 7: a rank-2 map with output error 29% adds ~0.0009 nats.

The rest of the networks reads only some directions of what a head writes. Error at the head and error in the final predictions are different quantities.
Media image
10
If the loss takes an "improvement" without complaint, you mustn't mistake good manners for an endorsement.

So what does this mean?

Published work in the most critical fields rely on this test.

Interpretability: supposed to vouch for models, as in, "we isolated the deception circuit, removed it and the model still works."

Compression: "average score unchanged" is supposed to assure you that the model hasn't changed much.
11
We audited 60 of the most heavily-cited papers (2022-26) that rely on "the model still works." 24 that explain models. 36 that compress them.

None of the papers do the tests required to tackle all failure modes.

- Circuit papers (11 of 12) test parts together, but only 1 checks the part's own output.
- SAE papers check the output (8 of 12), but none tests parts together
- 36 of the most-cited compression papers, 0 measured whether answers have changed.
Media image
12
These effects can interact catastrophically.

_/// Circuit papers that don't check the part's own output (11 of 12)

The circuit meets task metrics but its components compute something different from what the paper describes. This means that roles assigned to components (e.g. "this controls deception") are at risk.

_/// Papers with no deletion comparison (10 of 24 with none at all; 4 with it consistently)

"Our explanation captures the behaviour and here's a recovery score."

Deleting the same components or a random set of the same size, might score about as well. Then the score measures how little those parts were needed.

_/// SAE papers that never test layers together (0 of 12 do)

"Per layer reconstruction quality justifies using SAEs across many layers at once"

Each layer's error passes alone, but the errors compound.

_/// Compression papers with no agreement measure (36 of 36)

"behaves like the original model"

Two methods with the same loss can change very different sets of answers, so emergent behaviour is still unknown.

A lab safety-tests its model at full precision. Then the world downloads a 4-bit copy. Does the copy still refuse what the original refused? Nobody checked.

_/// Compression papers with no uncertainty (26 of 36)

"method A beats method B"

The ranking may be noise.
13
Builds on @pmichelX's attention-head pruning work & @banburismus_'s Hydra Effect.
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement