AI/ML thread by @stochasticchasm

1

good morning (reaction thread)

2

big headline of course is the causal encoder-decoder. honestly a bit of a confusing name, though i kinda think it's accurate. it's not really a decoder-decoder because you're not using the first half to decode latents to anything. even though it's causal

3

i've already seen a gazillion comparisons to YOCO, which does feel very similar. and they call it a decoder-decoder there, so maybe we should just stick with decoder-decoder? not that this is important either way.

                            4
                        
will probably get back to comparing to yoco once i land on that part of the tech report

5

before we get into the meat of the tech report, i think it's pretty crazy how high the benchmarks of this thing are. and going 4x smaller on the kv cache compared to dsv4-flash is also insane.

                            6
                        
i'm surprised that they're breaking the v = new pretrain convention, they were one of the ones who were still neatly dividing releases like this. though the commitment to this new architecture is really cool to see. it's neat how this seems like a simplification of v4 and they claim it trained much more stably as well

                            7
                        
as a side note it's hilarious to include deepseek v1 in the kv cache bar chart

8

seems similar in design philosophy to hysparse, nsa, and deepseek's own csa/hca from v4. combining a local sliding window branch with a sparse branch for retrieval seems like it's solidifying as a broad view of things

9

very very inference-informed design. cheap prefill, small KV cache, probably very friendly for PD disagg i'd imagine, seems like you'd get good utilization with larger batch prefill here

                            10
                        
it's somewhat interesting that inference efficiency is now the sole purpose of architectural modifications, in a sense

11

i'm a fan of "pure CSA2" over the alternating CSA/HCA setup from v4, it always felt weird that they had two frequencies of compression without any real explanation as to why. i'm sure there's a case to be made for multiple compression frequencies, but different designs felt odd

                            12
                        
FP4 KV cache with no degradation + compressing KVs via CSA, it's no wonder that it barely uses any KV cache. inference engineers are going to have a fun time with this one. i wonder how much kernel reduction order etc will affect perf here. cool that it's just straight up fp4

13

i think this makes sense. for a single request of course you have to use the kv cache in hbm/ram to fulfill the request, but between requests, if it takes up a ton of space, why not just re-prefill those bits. it's a negligible amount of flops at like 500K context anyway

14

engram finally made it in the model, plus an mHC simplification. both very expected additions. mega-mhc sounds fun, will need to look into that. does this mean they used dspark during pre-training? i guess we'll see the answer later in the paper

15

whenever a sparse attn version doesn't need a dense attention warmup, feels like a good sign. long-context pretraining here is probably enabled by the lower prefill flops, but this probably enables things like agentic traces much earlier in training as well. midtraining kinda being absorbed

16

makes sense. they already had a strong recipe + all the batch invariant kernels + low precision inference etc for stable RL, so data is what deserves attention at this stage. no critics yet though i feel like it's only a matter of time until they pull out some value modeling

17

alright onto the fun part. CED. when i read the initial blog i sort of assumed that the decoder would be able to attend to the KVs of the encoder section at multiple layers, but it doesn't seem that way at all. instead it's just a periodic read of the final encoder states. i guess this makes sense, those would be the richest and this also somewhat follows the original transformer. of course, this isn't cross attention, the self-attention just has prefilled KVs

18

cross attention would be only attending to the KVs of the encoder - which i'm sure they probably tried as one of the ablations, but its interesting that the decoder can attend to the encoder hidden states in context of its own sequence until that point directly. seems like a good choice. this also reminds me of something like MoDA - since it's sort of an allow-KVs-from-earlier-layers type of approach

19

i wonder why the first two layers are SWA only. two layers out of 20 doesn't seem like it's saving a ton on prefill flops but i guess the less global the cheaper it is. feels like there's a story here, or that it's something like first-k-dense for MoEs

20

gigantic engram size. most scaled up we've seen to date, qwen-3.8-flash-next had 51B n-gram params. granted the model itself is larger here

21

hysparse + indexshare type of thing here. sharing global kv + unique swa seems like a decent tradeoff of getting tiny kv cache but still getting each layer to see new states

22

man i mentally read this as "load-bearing"
modality specific experts have been a thing for a while, with modern models having it sort of emerge during training. so i think a load balancing term that aims to balance both text and image tokens is an interesting development

                            23
                        
do you want to let modality specific experts emerge? do you want to disincentivize them for more embedding space overlap between modalities? i'm sure someone has done work on this that i'm just not aware of yet

24

nothing crazy on how multimodal is introduced. just let the backbone handle most of it and give it visual tokens. 3x3 pixel unshuffle is more aggressive than others' 2x2 that we've seen

25

all very sensible choices here i think. removing convs for muon is an interesting motivation for the choice, given keller's original post did mention that muon works with convs (via flattening). maybe they tried this and didn't see it performing well

26

i'm not seeing why modality-specific imbalance is necessarily harmful unless it's like really really skewed and you're effectively stuck with a smaller model.

                            27
                        
gonna take a brunch break and get back to it

28

alright, CED time. they mention YoCo directly, not sure why i've seen so many comments saying that they didn't acknowledge it

29

i guess it should be YOCO and not YoCo? lol. anyway, going back to YOCO, it's a very simple design. the first half of the model is normal, and then you take the hidden states, project them to K/Vs once more, and then use that identical KV cache for every layer in the second half of the model

                            30
                        
in deepseek's CED, it's basically the same thing, applied only to the global branch of the CSA2 layers, and uniquely for each layer. 
so in the decoder every CSA2 layer has a global branch and a local swa branch where the swa branch is computed from current layer hidden states, and the global branch's kv cache is just a linear proj of the encoder hidden states. and they don't do this during prefill, since you can just get the relevant KVs for decode with a quick linear proj

31

i think this classification is a little funny, but a nice way to talk about this, the modes being named is easier than mentioning what they share every time. random part of me wishes the names were more intuitive, like what would you call a mode that re-uses indices but not KV like glm 5.2 here?

32

this part is kinda nice. you get perf benefits while still being flexible. it's basically blockwise scoring -> tokenwise scoring with shared block indices and not shared token indices within the blocks. unsure why this was introduced at post-training, probably because of longer sequences if i had to guess, though 64k is decently long at pretrain still

33

very heavy hysparse vibes, just had to mention it again

34

yeah i guess you can overlap it to make this more efficient. fair enough why not

35

some interesting details in the engram section:
- table sizes chosen to be distinct primes - don't recall this being important before
- 4-grams are also not something i think we've seen in other models that used these
- all in fp8? so fp8 lookup tables, i'm guessing this is inference

36

they killed MTP which is kinda sad. guess the performance uplift wasn't worth it at the end of the day?

37

ah this explains it. QAT for the kv cache as well. no wonder it performs better than other models at fp4 kv cache

38

i found this breakdown very interesting

39

another head wise muon shift. safe to say it's becoming the default

40

ah this is so sick. no adam at all. and also especially relevant with the gigantic engram table

41

nevermind they do still have some adam optimized parameters. but no input/output is super cool

42

no wd on lm head

43

also cool to see the adam-rms-matching for the sinkhorn update too

44

must have missed this earlier, but this is a big difference from k3 on vision encoders

45

i like this. never had to deal with image processing bottlenecks so far, so this is something i was wondering about for long sequences. for inference as well as training

46

what a fun name. it's kinda like ac where they just recompute during training

47

yeah i guess this is unavoidable since the receptive field kinda is n_win*L but man that's crazy

48

this is kinda vague. i wonder how you get "interactions" between corpora - surely this is just synthetic data

49

wow smolvlm mentioned

50

no instabilities, so i guess they solved whatever the issue was during v4 training. that's a good sign for this arch. also very interesting that they continue with WSD, and also extend context length for ~10T tokens. training at 1M tokens for that long is kinda wild

51

this gives us a second datapoint on top of MAI-thinking-1 as to how models are continuing RL beyond collapse. though this is absurd, merging checkpoints across different scaffolds? harnesses presumably?

52

jesus. i think k3 mentioned ~50M sandboxes over the full run, but millions of *concurrent* sandboxes is a lot

53

yep, here's the corresponding section of the k3 report

54

love reading novel reasoning effort control schemes. this seems similar to openai's juice value, which imo is nice because you can get much more fine-grained control. really interesting setup with grpo here too, in that each group is sampled at one effort level, and each task at multiple effort levels. the reference length "L_norm" seems to be similar to k3's calibrated per-problem reference length, though they don't say how they get this reference length at all

55

i assumed they were already doing this. i guess batch invariant + sync explains the batch invariant focus in v4. imagine how much more throughput they got now

56

i guess this makes sense. but also really funny to see colocated async RL becoming more popular with k3 doing the same thing

57

to end it off, i think the multi agent eval is really interesting. there's a ton of forms a multi agent harness could look like, and the optimal one is still very much undecided. training for it seems straightforwardly beneficial though and can scale better than single agent. great to see it.

good morning (reaction thread)

Actions

What You Can Do

Actions

What You Can Do

More in AI & Machine Learning

More from @stochasticchasm