going to read and react while i wait for my agents. some really...
stochasm@stochasticchasm
31 views
Aug 28, 2026
~6 min read
Advertisement
1
going to read and react while i wait for my agents. some really nice stuff in here, i skimmed it earlier and i quite like the gated residual section. also somehow missed the gated rmsnorm that they reference in there
2
algorithmic progress indeed. much as i like a good architecture paper, this point seems like it should be at least followed by a discussion of training data changes or at least if there was a change in dataset (which there must have been since previous gen). would be nice to see a decoupling of impact here, though the overall comparison is nice as well
3
generally nice to see that the global attention layer seemingly can be replaced by a sparse attention layer with no downside. glm-5.3 flash did the same, with KDA + sparse attn. intuitively makes sense since linear attn has that local position bias and lacks in retrieval, and sparse attn seems to preserve retrieval capabilities, but nice to see in practice and scaled up
4
this is interesting and something that i'd expect to change with compute budget, much like normal tokenizer/vocab-size changes. another instance of the classic loss != downstream capabilities. second image from arxiv.org/abs/2605.01188
5
also in that section, some more interesting takeaways. the first sentence i probably should have highlighted above for loss != downstream capabilities etc. that later sentence is interesting though, especially given that kimi k3 (well, kimi linear) prefers nope on global
6
everyone's got their own ns iteration setup now
7
trained with fp8 activations? or i guess this is in the context of decode, so fp8 activations for inference then? or just for residual state
8
man what a great paragrpah here. i don't think they unfuse each attn head here and it's just qkv -> q,k,v etc.
for the scaling law, does this confirm they use the original kaplan law? since no modern scaling law work is cited? or do they have their own internal formulation?
for the scaling law, does this confirm they use the original kaplan law? since no modern scaling law work is cited? or do they have their own internal formulation?
9
also interesting to note that batch size warmup doesn't seem to do anything here. i doubt this is arch specific but there's a lot of mixed results on this, so maybe?
10
i like this. reminds me a lot of this classic arxiv.org/abs/2309.14322
11
random note but this is probably the cleanest GDN diagram i've ever seen
12
interesting results here. i wonder what makes some benchmarks vary so much more than others. seems like "reasoning" benchmarks here for lack of a better word (non-knowledge) get the highest uplift
13
another tilelang kernel library, classic
14
going to qwen sparse attention now, this seems like a refined version of minimax sparse attention, which i feel deserves at least a mention in the paper. (arxiv.org/abs/2606.13392) they're both block sparse, the only difference is that qwen's does the pooling before sdpa and minimax's does it afterwards. tbh makes a lot of sense to be doing this before sdpa to make sdpa way cheaper at long context
15
MQA makes sense, partial rope is interesting. tbh i haven't paid much attention to positional encodings in the indexer before, so it may be standard practice that i'm just realizing now
16
tbh i'm not a huge fan of this requiring two stages of training, and i wish they tested both from-scratch training as well as sparsification during CPT. minimax sparse attention was nicer here where they compare both sparsification during CPT to from-scratch training
17
also during the dense distillation phase, i wonder how it would perform if it was just teacher forced during the initial stage of training, instead of doing the freeze full network -> distill indexer -> unfreeze
18
hm this is actually introduced way later in the training process than i initially expected, if i'm reading this right. only 8k steps afterwards
19
these are two very interesting ablations. the first one because it's two techniques that i wouldn't really think to compare, and honestly could probably be composed? layer wise vs sequence wise indexer sharing essentially. i'd really like to see a composed ablation tbh
20
the second one because the number of query heads doesn't seem to have much of an impact at all after the first stage, even from single head to 4, and that's a lot more flops. but then it's way more noticeable after stage 2. i guess it makes sense that the network has to be trained to route retrieval through sparse blocks but yeah
21
oh man i have a meeting to go to
22
that's done, back to reading. gated residual section, my favorite part of the report. random thought, but has anyone tried shortconvs on the residual stream? i imagine this is just a worse version of attnres but would be interesting to see
23
this is a really nicely written section of the paper, and you can see how they arrived at the final design instead of just the final design itself. i think probably the more interesting things here is that H_res in HC doesn't seem to provide much of a benefit after a read/write are expressive enough. i'm a fan of this, since H_res is just a projection of the input residual anyway and any write should probably just write back in a way that accommodates the original residual stream. and i would guess this also keeps the geometry of the residual stream more consistent across layers, which probably helps for stuff like logit lensing
24
this is also an interesting finding. i suppose this allows for more fine-grained data dependence?
25
this part is really cool too, i missed this work previously. gated <X> is just a qwen tradition now
26
a couple points that i find interesting in the final design:
- this 2 in the residual write back op. seems to be for init dynamics, sigmoid(0)=0.5 and init is zero-centered. so for normal scale writeback you want 2 x sigmoid
- unless i'm missing something, \overline{R} isn't used anywhere? but they also call out group-RMSNorm as being important in the previous section
- this 2 in the residual write back op. seems to be for init dynamics, sigmoid(0)=0.5 and init is zero-centered. so for normal scale writeback you want 2 x sigmoid
- unless i'm missing something, \overline{R} isn't used anywhere? but they also call out group-RMSNorm as being important in the previous section
27
i wonder how more extreme depth would affect the rankings here. block attnres and full attnres are so close already, and maybe at 100+ layers the scales tip? since attnres can always full magnitude pull in early layers
28
i understand that there's no explicit H_res mixer here but i mean the blocks are taking in info from all branches and writing to all branches - feels like this should be counted as a form of mixing
29
i like this visualization, once again very interesting value-residuals/attn-residuals-like trend where early layer info seems to be the most important, here early layer GDN outputs seem to be read a lot later down the line
30
seemingly nothing new about the way they design their N-gram embeddings, but this seems to be an emerging pattern - letting it overlap with early layer execution
31
big fan of this call out, and also intuitively makes sense. with limited context in n-grams, there's only so much info you can pre-bake into a single embedding without stronger contextualization/computation through the backbone
32
classic moonlight scaling and polar express orthogonalization
33
interesting callout of eps 1e-14
34
this is really interesting, first open model to do this from what i can remember, i feel like i've seen some conflicting results here before
35
i was wrong earlier, seems like they do in fact do per-head orthogonalization following kimi and glm
36
not much to add but i really appreciate this section and them testing it out. this report has a lot of useful informatino
37
muon is just really stable huh
38
i'm getting incredibly gated-residual-pilled here. not only is the grad norm so much smoother but much more stable residual maxima without any sandwich norm or anything is really impressive
39
<|end_of_thread|>
