Muon’s speed arguably comes from approximate orthogonalization: a...

@egor_shulg
Egor Shulgin@egor_shulg
16 views Oct 29, 2025 ~1 min read
Advertisement
1
Muon’s speed arguably comes from approximate orthogonalization: a few fast Newton–Schulz iterations instead of an expensive full SVD. But this makes the update inexact. So how should hyperparameters change as approximation quality varies?🧵
1/n
Media image
2
We bridge theory & practice: prior work studies an idealized SVD update. We analyze the implemented inexact (Newton–Schulz) iteration and show how approximation quality shifts the best LR & test on nanoGPT. With @SultanAlra60920 @bremen79 @peter_richtarik
arxiv.org/abs/2510.19933
3
Analysis builds on the great LMO (@tmpethick et al.) and trust‑region (@dakovalev1) frameworks for the idealized Muon update. We extend them by modeling orthogonalization with a realistic additive error δ: a proxy for the number of Newton–Schulz iterations.
3/n
4
Our theory predicts: better approximations (smaller δ) → better performance (faster convergence). In practice, we clearly see this: more PolarExpress iterations (by @noahamsel et al.) consistently improve validation loss, with gains saturating around ~5 iterations.
4/n
Media image
5
Key insight: the optimal learning rate couples with approximation quality. Fewer Newton-Schulz/PolarExpress iterations (larger δ) need a smaller LR. NanoGPT sweeps (2 vs 5 iterations) show: higher precision → higher optimal LR + wider stability, matching the prediction.
5/n
Media image
6
Practical Muon takeaway: choose the number of Newton–Schulz iterations, then tune LR: fewer → lower LR, more → higher LR, and a broader stable region. This closes a key theory-practice gap and explains why better approximation methods (PolarExpress, CANS) help.
6/n
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement