We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math....
Rikiya Takehi@rikiyatakehi
47 views
Nov 03, 2026
~2 min read
Advertisement
1
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weight-sp…
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weight-sp…
2
1/6 We find early signs that frontier LLMs are surprisingly good at *natively* reading, writing, and reasoning about neural network weights.
Instead of training, models may treat raw weights like readable, editable code: diagnosing failures and patching them directly.
Instead of training, models may treat raw weights like readable, editable code: diagnosing failures and patching them directly.
3
2/6 First, a look at understandability:
We fed Astra raw weights from 4 game policies one at a time. All four share the exact same architecture, inputs, and actions.
With no tools or options given, it correctly named all four: Breakout, Pong, Pac-Man, Freeway.
We fed Astra raw weights from 4 game policies one at a time. All four share the exact same architecture, inputs, and actions.
With no tools or options given, it correctly named all four: Breakout, Pong, Pac-Man, Freeway.
0:09
4
3/6 Next, fixing weights: A PPO car policy kept getting stuck on a ramp.
Given the weights, scene, and 1 failed trace, Astra wrote a patch in one reply with no tool-use.
By editing just 3.5% of the weights, it achieved 100% success.
Given the weights, scene, and 1 failed trace, Astra wrote a patch in one reply with no tool-use.
By editing just 3.5% of the weights, it achieved 100% success.
0:15
5
4/6 Writing from scratch: given MinAtar game rules and code, Astra wrote a Breakout network's weights in a single reply.
With zero training or tools, it scored 20× higher than PPO trained for 500M steps!
With zero training or tools, it scored 20× higher than PPO trained for 500M steps!
0:14
6
5/6 Can an LLM improve another LLM?
We simply asked Astra to make Qwen3.5-0.8B better at math.
Astra saw Qwen get stuck in reasoning loops, which is a known issue that Qwen’s own model card warns about (see their official model card). With a budget of 50 forward passes on practice problems, it located the cause and edited it out. This used no gradients or training.
Fixing these loops by editing just 0.016% of weights boosted other capabilities beyond math too!
We simply asked Astra to make Qwen3.5-0.8B better at math.
Astra saw Qwen get stuck in reasoning loops, which is a known issue that Qwen’s own model card warns about (see their official model card). With a budget of 50 forward passes on practice problems, it located the cause and edited it out. This used no gradients or training.
Fixing these loops by editing just 0.016% of weights boosted other capabilities beyond math too!
7
6/6 We think neural network weights will soon be natively optimized by LLMs that deliberately reason about the weights.
We are now building a benchmark to evaluate, and a harness to help LLMs explore and write weights.
rikiyat.github.io/blog/weight-sp…
We are now building a benchmark to evaluate, and a harness to help LLMs explore and write weights.
rikiyat.github.io/blog/weight-sp…
8
@a1zhang @lateinteraction @jayden_teoh_ The edited model is now on the Hub: huggingface.co/RikiyaT/Qwen3.…
Huge thanks to @akhaliq, who rebuilt it from our published logs, re-ran MATH-500 (51.0%).
They also seems to have a live demo: huggingface.co/spaces/ML-Inte…
Huge thanks to @akhaliq, who rebuilt it from our published logs, re-ran MATH-500 (51.0%).
They also seems to have a live demo: huggingface.co/spaces/ML-Inte…
