🎯 Just released a new preprint that proves LR transfer under μP. ->...
Soufiane Hayou@hayou_soufiane
19 views
Nov 05, 2025
~2 min read
Advertisement
1
🎯 Just released a new preprint that proves LR transfer under μP.
-> The Problem: When training large neural networks, one of the trickiest questions is: what learning rate should I use? [1/n]🧵
Link: arxiv.org/abs/2511.01734
-> The Problem: When training large neural networks, one of the trickiest questions is: what learning rate should I use? [1/n]🧵
Link: arxiv.org/abs/2511.01734
2
A technique called μP solve this— paramterize your network in a certain way, then, you can tune your learning rate on a small model, and it should work on larger models too. This property is called "learning rate transfer" [2/n]
3
Practitioners who train LLMs have been using μP for years, but surprisingly: no one had actually proven it works mathematically. It was more of an conjecture based on how features behave during training [3/n]
4
-> In this new paper: We provide the first rigorous proof that learning rate transfer happens under μP. Specifically, we prove that as neural networks get wider, the optimal learning rate converges to a stable, non-zero value - exactly what you need for transfer to work [4/n]
5
It was largely a conjecture: μP is associated with "maximal feature learning" (feature move by Theta(1) in width with each gradient step), making it the "optimal" feature learning parametrization. The convergence of optimal LR has been an open question since [5/n]
6
Why is it hard to prove LR transfer? To address this question, we should first formally define LR transfer.
- Optimal LR is defined as the argmin of the train loss over learning rates.
- LR transfer is when Optimal LR converges to a **non-zero** constant [6/n]
- Optimal LR is defined as the argmin of the train loss over learning rates.
- LR transfer is when Optimal LR converges to a **non-zero** constant [6/n]
7
Convergence to 0 or divergence ==> no LR transfer.
- So why is proving LR transfer is challenging?
It involves proving that the argmin of a random process converges as width → ∞! [7/n]
- So why is proving LR transfer is challenging?
It involves proving that the argmin of a random process converges as width → ∞! [7/n]
8
- How do we prove it here?
With linear MLPs, we express train loss at any step t as a polynomial function in the LR. The optimal LR is therefore a root of a polynomial (derivative of the loss), and we use this to establish convergence.
Check the paper for more details!
With linear MLPs, we express train loss at any step t as a polynomial function in the LR. The optimal LR is therefore a root of a polynomial (derivative of the loss), and we use this to establish convergence.
Check the paper for more details!