Muon’s speed arguably comes from approximate orthogonalization: a...

Egor Shulgin@egor_shulg
16 views
Oct 29, 2025
~1 min read
Advertisement
2
We bridge theory & practice: prior work studies an idealized SVD update. We analyze the implemented inexact (Newton–Schulz) iteration and show how approximation quality shifts the best LR & test on nanoGPT. With @SultanAlra60920 @bremen79 @peter_richtarik
arxiv.org/abs/2510.19933
arxiv.org/abs/2510.19933
3
Analysis builds on the great LMO (@tmpethick et al.) and trust‑region (@dakovalev1) frameworks for the idealized Muon update. We extend them by modeling orthogonalization with a realistic additive error δ: a proxy for the number of Newton–Schulz iterations.
3/n
3/n
6
Practical Muon takeaway: choose the number of Newton–Schulz iterations, then tune LR: fewer → lower LR, more → higher LR, and a broader stable region. This closes a key theory-practice gap and explains why better approximation methods (PolarExpress, CANS) help.
6/n
6/n


