Stronger Normalization-Free Transformers – new paper. We introduce...
Zhuang Liu@liuzhuang1234
77 views
Dec 14, 2025
~2 min read
Advertisement
1
Stronger Normalization-Free Transformers – new paper.
We introduce Derf (Dynamic erf), a simple point-wise layer that lets norm-free Transformers not only work, but actually outperform their normalized counterparts.
We introduce Derf (Dynamic erf), a simple point-wise layer that lets norm-free Transformers not only work, but actually outperform their normalized counterparts.
2
Earlier this year, in our "Transformers without Normalization" paper, we showed that a Dynamic tanh (DyT) function can replace norm layers in Transformers.
Derf pushes this idea further.
Derf pushes this idea further.
3
Like DyT, Derf is a statistics-free, point-wise layer that doesn’t rely on activation statistics.
It’s just a shifted and scaled Gauss error function with a few learnable parameters, dropped in wherever you’d normally use LayerNorm or RMSNorm.
It’s just a shifted and scaled Gauss error function with a few learnable parameters, dropped in wherever you’d normally use LayerNorm or RMSNorm.
4
Before designing Derf, we asked: what makes a point-wise function a good normalization replacement? What’s needed for stable convergence?
We identify 4 key properties:
- Zero-centeredness
- Boundedness
- Center sensitivity (responsive near 0)
- Monotonicity
We found functions satisfying all four conditions train stably and perform well.
We identify 4 key properties:
- Zero-centeredness
- Boundedness
- Center sensitivity (responsive near 0)
- Monotonicity
We found functions satisfying all four conditions train stably and perform well.
5
Guided by these properties, we run a expansive function search over many candidates that satisfy them (e.g., transformed erf, tanh, arcsinh, log-type functions, etc.).
Across ViT classification and DiT diffusion performance, erf with a learnable shift and scale consistently comes out on top.
We introduce it as 𝗗𝗲𝗿𝗳 (𝗗ynamic 𝗲𝗿𝗳).
Across ViT classification and DiT diffusion performance, erf with a learnable shift and scale consistently comes out on top.
We introduce it as 𝗗𝗲𝗿𝗳 (𝗗ynamic 𝗲𝗿𝗳).
6
Derf matches or outperforms normalization layers, and consistently beats DyT, with the same training recipe, across domains.
1. ImageNet - higher top-1 in ViT-B/L
2. Diffusion Transformers - lower FID across the DiT family
3. Genomics (HyenaDNA, Caduceus) - higher DNA classification accuracy
4. Speech (wav2vec 2.0) - lower validation loss
5. Language (GPT-2) - matches LayerNorm, clearly beats DyT.
A simple point-wise layer can make Transformers stronger, not just as good.
1. ImageNet - higher top-1 in ViT-B/L
2. Diffusion Transformers - lower FID across the DiT family
3. Genomics (HyenaDNA, Caduceus) - higher DNA classification accuracy
4. Speech (wav2vec 2.0) - lower validation loss
5. Language (GPT-2) - matches LayerNorm, clearly beats DyT.
A simple point-wise layer can make Transformers stronger, not just as good.
7
Is Derf just fitting better? Surprisingly, no.
When we measure training loss in eval mode on the training set:
• Norm-based models have the lowest train loss
• Derf has a higher train loss
• Yet Derf has better test performance
This suggests Derf’s gains mainly come from stronger generalization than norm layers.
When we measure training loss in eval mode on the training set:
• Norm-based models have the lowest train loss
• Derf has a higher train loss
• Yet Derf has better test performance
This suggests Derf’s gains mainly come from stronger generalization than norm layers.
8
To summarize: Derf is a simple, practical drop-in layer for stronger normalization-free Transformers.
code: github.com/zlab-princeton…
paper: arxiv.org/abs/2512.10938
Led by @mingzhic2004, and joint work with @TaimingLu, @JiachenAI, and @_mingjiesun.
code: github.com/zlab-princeton…
paper: arxiv.org/abs/2512.10938
Led by @mingzhic2004, and joint work with @TaimingLu, @JiachenAI, and @_mingjiesun.
