So how can Astra read neural network weights? A thread 🧵 Weights of...

10 views Nov 04, 2026 ~2 min read

So how can Astra read neural network weights? A thread 🧵

Weights of a deep NN are famously opaque, with one exception: the first and last layers, which map to known inputs and outputs. Is that what Astra is reading?
Post by @rikiyatakehi et al: [1/10]

Rikiya Takehi@rikiyatakehi

We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.

We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weight-sp…

                            2
                        
The Weight-Space Programming post by Takehi, Teoh, Zhang & Khattab went viral. I agree with others that its most impressive part is the “zero-shot” reading of NN weights. But hidden units can be shuffled freely, so their weights have no fixed meaning. I was suspicious 🤔. [2/10]

The weights of a *linear* network, like a perceptron trained on MNIST, can be read directly in input space: each digit’s weights look like a blurry template of that digit. For deep networks, this isn’t necessarily true, but the first layer is often still interpretable. [3/10]

                            4
                        
So, can the task solved in Exp 1 (guess which game a network was trained on, from its weights alone) be reduced to inspecting the first-layer weights? [4/10]

                            5
                        
In Exp 1, they trained MLPs to play games on a 10x10 grid. The network “sees” each square via 5 yes/no channels: agent (is the thing I control here?), moving object (anything moving?), reward (worth points?), wall (blocked?), last frame (was something here a step ago?) [5/10]

For each cell & channel, we plot the norm of its weights into the 32 hidden units. A cell that isn’t used during play gets no gradient, so a large norm means the network uses that cell. Given these illustrations, performance is not as mysterious [6/10]

Similarly for the output side: we linearize the network by multiplying W1·W2·W3 (ignoring the tanhs), which gives each cell a weight onto each action, then plot the norm per action. Breakout lights up only left/right, Pong only up/down. [7/10]

                            8
                        
So the game layout is quite transparent in these weights. It’s still super impressive that Astra spots it *without* tools, but this *isn’t* general weight-space reading, and it likely won’t carry over to nets whose inputs aren’t laid out like this. [8/10]

                            9
                        
Exp 3 (no input/output layers) is harder, but there tools *were* allowed, and Astra used coarse cues: lots of copying heads => code repeats names => Python; fewer distinct directions => smaller alphabet => romaji. And confidence was ~50%. [9/10]

                            10
                        
Kudos to the authors for releasing all weights & logs, which made this check possible. The end! [10/10]

So how can Astra read neural network weights? A thread 🧵 Weights of...

Actions

What You Can Do

Actions

What You Can Do

More in AI & Machine Learning