Distillation is a common way to prepare a model for RL where...
Base Labs@baselabs
8 views
Oct 30, 2026
~1 min read
Advertisement
1
Distillation is a common way to prepare a model for RL where learning occurs from a stronger model, and then improves through trial and error.
But how much does that better starting point help after RL? We conduct preliminary research on this question across model sizes and reasoning tasks. 🧵
But how much does that better starting point help after RL? We conduct preliminary research on this question across model sizes and reasoning tasks. 🧵
2
We ran all our experiments on Qwen3 (0.6B, 1.7B, 4B, 8B).
Throughout, smaller models kept their distillation advantage after RL, but with the same RL token budget this advantage faded at larger sizes. The largest model finished ahead without distillation.
Throughout, smaller models kept their distillation advantage after RL, but with the same RL token budget this advantage faded at larger sizes. The largest model finished ahead without distillation.
3
We further dissected how the type of reasoning task impacted these findings. Larger models (4B, 8B) did better with RL alone on tasks involving calculation (e.g. a simple integral) while distillation retained an advantage on structured tasks (e.g. reversing spelling).
Most importantly, a stronger starting model didn’t always become a stronger final model.
Most importantly, a stronger starting model didn’t always become a stronger final model.
4
We also investigated different amounts of distillation and the rate of accuracy improvement that distillation confers, amongst other results.
This is a first step toward understanding how different forms of distillation affect RL, and how to warm-start model training more effectively.
Full work: labs.baseten.co/articles/when-…
This is a first step toward understanding how different forms of distillation affect RL, and how to warm-start model training more effectively.
Full work: labs.baseten.co/articles/when-…
