Notes on DeepSeek V4.1 Flash. How obsessing over KV Cache...
wh@nrehiew_
22 views
Sep 11, 2026
~5 min read
Advertisement
1
Notes on DeepSeek V4.1 Flash.
How obsessing over KV Cache compression gets you a hyper-efficient frontier model
How obsessing over KV Cache compression gets you a hyper-efficient frontier model
2
The first architectural difference is the most 'normal' one: multimodality
- Pretrained on 45T image-text tokens
- Custom image encoder
- Load balancing at the modality level
The last point is reminiscent of the original Chameleon paper (second image) where tokens from different modalities may compete and interfere with each other
- Pretrained on 45T image-text tokens
- Custom image encoder
- Load balancing at the modality level
The last point is reminiscent of the original Chameleon paper (second image) where tokens from different modalities may compete and interfere with each other
3
Now the big one, 20 layer encoder, 20 layer decoder.
> Prefill is expensive
We can halve KV cache size and have the bottom half create the KV cache which is shared with the upper half layers (projected per layer). This can roughly be thought of as some state that is read individually by the later layers
This is of course inspired by YOCO which is why people have called it decoder-decoder, where both modules are masked causally.
This doesn't apply to the SWA layers
> Prefill is expensive
We can halve KV cache size and have the bottom half create the KV cache which is shared with the upper half layers (projected per layer). This can roughly be thought of as some state that is read individually by the later layers
This is of course inspired by YOCO which is why people have called it decoder-decoder, where both modules are masked causally.
This doesn't apply to the SWA layers
4
Their "global attention" is some updated compressed attention thing.
In all 3 variants below, there is some concept of an indexer operating over smaller Keys and Values, doing selection on the main KV, concatenating with a windowed KV and only then going through attention
The variants differ in:
- No reuse
- reuse KV from earlier layers and indexer K
- reuse index selection output
(claude slop diagram which ended up being pretty good imo)
In all 3 variants below, there is some concept of an indexer operating over smaller Keys and Values, doing selection on the main KV, concatenating with a windowed KV and only then going through attention
The variants differ in:
- No reuse
- reuse KV from earlier layers and indexer K
- reuse index selection output
(claude slop diagram which ended up being pretty good imo)
5
The indexer itself is also sparse.
The first full mode indexer selects candidates which are then used to score in later layers
The first full mode indexer selects candidates which are then used to score in later layers
6
The rest of these changes are:
1) Simplification over normal mHC where the input of the next block is a mixture between the current block and the previous block. The motivation of this is better explained in the paper but basically the input mixing operation cannot be fused in normal mHC
2) Engram of 196B params
3) DSpark
4) QAT for FP4 KV cache for a further reduction. It looks like this format is basically NVFP4 without the second scale
1) Simplification over normal mHC where the input of the next block is a mixture between the current block and the previous block. The motivation of this is better explained in the paper but basically the input mixing operation cannot be fused in normal mHC
2) Engram of 196B params
3) DSpark
4) QAT for FP4 KV cache for a further reduction. It looks like this format is basically NVFP4 without the second scale
7
Head wise muon as the optimizer including for the vision models.
The fact that sparse attention is trained from scratch without warmup is pretty cool
The interesting part is that they use Sinkhorn Balancing for the Engram params, token embedding and prediction head (the super large params) instead of Adam is it would be too large. My understanding of this is that we keep the momentum buffer, do sinkhorn iterations, RMSNorm and then apply this gradient update.
(not fully understanding tbh)
The fact that sparse attention is trained from scratch without warmup is pretty cool
The interesting part is that they use Sinkhorn Balancing for the Engram params, token embedding and prediction head (the super large params) instead of Adam is it would be too large. My understanding of this is that we keep the momentum buffer, do sinkhorn iterations, RMSNorm and then apply this gradient update.
(not fully understanding tbh)
8
Okay now on to infra stuff
1) Vision encoder is done Siglip style. The main insight is that the all gather of each modalities features can be done while the other modality is computing. They also apply CP for ultra-long multi-image sequences.
2) CSA 2 requires a bunch of stuff:
- Shadow indexers which only do computation and pull updated weights from a master replica. My intuition is that it is similar to inference workers in RL
- Something called pipeline payload extensions. I have no idea what they are talking about but it sounds like metadata across pp stages
3) Optimizer sharding across Engram tables + Engram being in the early layers allows for prefetchin while the vision encoder is running
1) Vision encoder is done Siglip style. The main insight is that the all gather of each modalities features can be done while the other modality is computing. They also apply CP for ultra-long multi-image sequences.
2) CSA 2 requires a bunch of stuff:
- Shadow indexers which only do computation and pull updated weights from a master replica. My intuition is that it is similar to inference workers in RL
- Something called pipeline payload extensions. I have no idea what they are talking about but it sounds like metadata across pp stages
3) Optimizer sharding across Engram tables + Engram being in the early layers allows for prefetchin while the vision encoder is running
9
For inference, the main insights are:
1) Tons of fused kernels. 15 kernels execute during prefill and 11 during decode
2) KV cache reduction from SWA. SWA is cached only at the end of every user/assitant message. The TTL of this cache is minute-scale and stored on DRAM.
If the cache is evicted, they have a replay scheme which bounds the max replay to some param S
For the encoder, replay the window but using the cached global KV (so its not fully identical) since we are replaying only the windowed tokens. The idea is that the global KV provides information beyond the window boundary. Similar for the decoder
(this is extremely cheap given window size of 128)
1) Tons of fused kernels. 15 kernels execute during prefill and 11 during decode
2) KV cache reduction from SWA. SWA is cached only at the end of every user/assitant message. The TTL of this cache is minute-scale and stored on DRAM.
If the cache is evicted, they have a replay scheme which bounds the max replay to some param S
For the encoder, replay the window but using the cached global KV (so its not fully identical) since we are replaying only the windowed tokens. The idea is that the global KV provides information beyond the window boundary. Similar for the decoder
(this is extremely cheap given window size of 128)
10
(skipping pretraining stuff, there is a bunch of hyperparam stuff and some general data stuff)
As many people have commented, their post-training philosophy is now completely data focused rather than some training algorithm research
As many people have commented, their post-training philosophy is now completely data focused rather than some training algorithm research
11
A large portion of their agent trajectories is inspired (for the first time explicitly) by real-world usage patterns from partners where data is created via multiple agents in a system synthetically
They also merge checkpoints from runs with different scaffolds (!) and configurations which led to (free) performance gains
They also merge checkpoints from runs with different scaffolds (!) and configurations which led to (free) performance gains
12
Now on to sandbox infra:
- They use a custom orchestrator to place sandboxes amongst compute nodes. Each node will also check locally for any capacity bottlenecks
- Each VM within their NUMA domain supporting 2500 concurrent sandboxes per node. To handle scheduling, they assign priority to tasks executing on the same machine
- Any hacking causes a "repercussion" signal;
- They use a custom orchestrator to place sandboxes amongst compute nodes. Each node will also check locally for any capacity bottlenecks
- Each VM within their NUMA domain supporting 2500 concurrent sandboxes per node. To handle scheduling, they assign priority to tasks executing on the same machine
- Any hacking causes a "repercussion" signal;
13
V4.1 is the first model where I have seen a numerical reasoning effort parameter which directly influences the length penalty
14
RL Infra time.
- nice dispatch strategy that gets rid of long tail stalls
- router replay from previous checkpoints
- this ends up causing shorter completions to impact early stages of training and off-policy. Their solution is to cap at the dataset level (so datasets with general shorter responses dont influence too much) and a discard scheme
- For offpolicyness, they bound the off-policy ratio and loss masking. so pretty standard here
- When a new checkpoint is updated, KVs and routers are persistent instead of being recomputed
At the final stage they do full vocab OPD on >40 teacher models
- nice dispatch strategy that gets rid of long tail stalls
- router replay from previous checkpoints
- this ends up causing shorter completions to impact early stages of training and off-policy. Their solution is to cap at the dataset level (so datasets with general shorter responses dont influence too much) and a discard scheme
- For offpolicyness, they bound the off-policy ratio and loss masking. so pretty standard here
- When a new checkpoint is updated, KVs and routers are persistent instead of being recomputed
At the final stage they do full vocab OPD on >40 teacher models
15
I dont usually talk too much about benchmarks but these numbers are insane. Sol/Opus level at this efficiency is crazy
16
Interesting that the reasoning-performance plot is not fully linear. I wonder why. I know FrontierCode had some code quality metrics which affected Opus 5 xhigh but i dont think any of these 3 benches here have similar penalties
The agent swarm plot looks alot cleaner. Interestingly, they do say they train Agent team mode explicitly using a RL reward "combining task performance, a
collaboration bonus that encourages delegation and inter-agent communication, and a derived-
latency penalty that promotes efficient coordination"
interesting stuff
The agent swarm plot looks alot cleaner. Interestingly, they do say they train Agent team mode explicitly using a RL reward "combining task performance, a
collaboration bonus that encourages delegation and inter-agent communication, and a derived-
latency penalty that promotes efficient coordination"
interesting stuff
17
All in all, yet another banger tech report from DeepSeek. I actually think this is alot cleaner than the HSA, CSA combination in v4.
Very clearly this was designed for inference in mind (which kinda makes sense since RL blurrs the line between inference and training). 890 bytes/token KV size for these benchmark score is just nuts. Eitherways, i would be pretty surprised if OAI/Ant are doing this sort of architectural frankenstein monstrosity thing given they have custom inference chips
Very clearly this was designed for inference in mind (which kinda makes sense since RL blurrs the line between inference and training). 890 bytes/token KV size for these benchmark score is just nuts. Eitherways, i would be pretty surprised if OAI/Ant are doing this sort of architectural frankenstein monstrosity thing given they have custom inference chips