There are competing views on whether RL can genuinely improve base...
Xiang Yue@xiangyue96
30 views
Dec 11, 2025
~2 min read
Advertisement
1
There are competing views on whether RL can genuinely improve base model's performance (e.g., pass@128). The answer is both yes and no, largely depending on the interplay between pre-training, mid-training, and RL. We trained a few hundreds of GPT-2 scale LMs on synthetic GSM-like reasoning data from scratch. Here are what we found: 🧵
2
Takeaway 1: RL helps extrapolation only when there's headroom left by pre-training and with the right RL data at the edge of the model's competence.
1) When RL is applied to very in-domain tasks, it never improves pass@k beyond the base model.
2) When RL targets the edge of the model's competence there are substantial extrapolations to even unseen OOD-hard tasks.
1) When RL is applied to very in-domain tasks, it never improves pass@k beyond the base model.
2) When RL targets the edge of the model's competence there are substantial extrapolations to even unseen OOD-hard tasks.
3
Takeaway 2: RL incentivizes contextual generalization only when the base model already contains the necessary primitives.
RL can incentivize new composition skills, but it cannot invent new basic skills. Minimal but well-placed pre-training exposure provides the necessary seed coverage—RL then unlocks strong transfer across contexts by combining these known primitives in novel ways.
RL can incentivize new composition skills, but it cannot invent new basic skills. Minimal but well-placed pre-training exposure provides the necessary seed coverage—RL then unlocks strong transfer across contexts by combining these known primitives in novel ways.
4
Takeaway 3: Under the same compute budget, mid-training + RL outperforms pure RL only. Mid-training is a powerful and overlooked lever in building reasoning LMs.
We additionally found with the same compute budget: 1) mid-training + light RL leads to better in-distribution performance; 2) mid-training + heavy RL leads to better OOD reasoning.
We additionally found with the same compute budget: 1) mid-training + light RL leads to better in-distribution performance; 2) mid-training + heavy RL leads to better OOD reasoning.
5
Takeaway 4: Process-verified outcome rewards mitigate reward hacking and enhance reasoning fidelity.
We find that incorporating process verification into outcome rewards delivers: 1) More truthful, error-resistant reasoning and 2) Better generalization on complex, multi-step tasks.
We find that incorporating process verification into outcome rewards delivers: 1) More truthful, error-resistant reasoning and 2) Better generalization on complex, multi-step tasks.
6
All of the experiments were done in strictly controlled settings. No contamination. Answers can be strictly verified for both process and outcome.
The data was built on the Physics of LMs and GSM-Infinite data generation framework to create a testbed with precise control over reasoning structure, complexity, and context.
The data was built on the Physics of LMs and GSM-Infinite data generation framework to create a testbed with precise control over reasoning structure, complexity, and context.
7
Resources:
Paper: arxiv.org/abs/2512.07783
Github: github.com/Interplay-LM-R…
HF: huggingface.co/Interplay-LM-R…
Paper: arxiv.org/abs/2512.07783
Github: github.com/Interplay-LM-R…
HF: huggingface.co/Interplay-LM-R…
8
This work was done with @gneubig and Charlie Zhang, who was an intern working with me at CMU and is now actively looking for Fall 2026 PhD positions (chariezhang0106@gmail.com)!
