Demystifying Long CoT Reasoning in LLMs arxiv.org/pdf/2502.03373...
Xiang Yue@xiangyue96
96 views
Feb 06, 2025
~5 min read
Advertisement
1
Demystifying Long CoT Reasoning in LLMs
arxiv.org/pdf/2502.03373
Reasoning models like R1 / O1 / O3 have gained massive attention, but their training dynamics remain a mystery. We're taking a first deep dive into understanding long CoT reasoning in LLMs!
11 Major Takeaways👇(long threads)
arxiv.org/pdf/2502.03373
Reasoning models like R1 / O1 / O3 have gained massive attention, but their training dynamics remain a mystery. We're taking a first deep dive into understanding long CoT reasoning in LLMs!
11 Major Takeaways👇(long threads)
2
Takeaway 1:
(a) SFT with long CoT can scale up to a higher performance upper limit than short CoT.
(b) SFT with long CoTs makes further RL improvement easier, while short CoTs do not.
(a) SFT with long CoT can scale up to a higher performance upper limit than short CoT.
(b) SFT with long CoTs makes further RL improvement easier, while short CoTs do not.
3
Takeaway 2:
SFT initialization matters: high-quality, emergent long CoT patterns from a large model (QwQ-32B) lead to significantly better generalization and RL gains compared with constructed long CoT patterns from an action prompting framework.
SFT initialization matters: high-quality, emergent long CoT patterns from a large model (QwQ-32B) lead to significantly better generalization and RL gains compared with constructed long CoT patterns from an action prompting framework.
4
Takeaway 3:
CoT length does not always scale up in a stable fashion. We observed that models increased their CoT length during RL training, eventually reaching the context window limit. This led to a decline in training accuracy due to CoTs exceeding the allowable window size.
CoT length does not always scale up in a stable fashion. We observed that models increased their CoT length during RL training, eventually reaching the context window limit. This led to a decline in training accuracy due to CoTs exceeding the allowable window size.
5
Takeaway 4:
Reward shaping can be used to stabilize and control CoT length while improving accuracy. We designed a reward function (Cosine Reward) to use CoT length as an additional input to stabilize emergent length scaling.
Reward shaping can be used to stabilize and control CoT length while improving accuracy. We designed a reward function (Cosine Reward) to use CoT length as an additional input to stabilize emergent length scaling.
6
Takeaway 5:
Length rewards will be hacked with enough compute, but this can be mitigated using a repetition penalty.
Length rewards will be hacked with enough compute, but this can be mitigated using a repetition penalty.
7
Takeaway 6:
Models might need more training samples to learn to utilize larger context window sizes. We found that the model with a context window size of 8K performed better than the model with 4K, as expected. However, we observed performance was better under 8K than 16K.
Models might need more training samples to learn to utilize larger context window sizes. We found that the model with a context window size of 8K performed better than the model with 4K, as expected. However, we observed performance was better under 8K than 16K.
8
Takeaway 7:
Verifiable reward signals based on ground-truth answers are essential for stabilizing long CoT RL for reasoning tasks. As an attempt to counter this, we explore using WebInstruct from our previous work which is more available despite more noise, like reasoning-related QA pairs extracted from web corpora. Adding noisy but diverse verifiable data leads to better performance in more general reasoning beyond math.
Verifiable reward signals based on ground-truth answers are essential for stabilizing long CoT RL for reasoning tasks. As an attempt to counter this, we explore using WebInstruct from our previous work which is more available despite more noise, like reasoning-related QA pairs extracted from web corpora. Adding noisy but diverse verifiable data leads to better performance in more general reasoning beyond math.
Xiang Yue@xiangyue96
🚀 Scaling 10M naturally existing instructions from the Web! 🌐
Introducing 🦣MAmmoTH2 (tiger-ai-lab.github.io/MAmmoTH2/), a family of open-source models that unlocks unprecedented performance gains in reasoning tasks by leveraging two key insights:
1️⃣ Scaling is still all you need, even in the instruction tuning phase! 📈
2️⃣ High-quality, diverse instructions naturally exist on the web and can be efficiently harvested without costly human annotation. 💸
Our 3-step pipeline mines 10M instruction-response pairs from web data:
1️⃣ Recall relevant docs from Common Crawl using fastText 🔍
2️⃣ Extract instruction-response pairs from docs using open-source LLMs 🪄
3️⃣ Refine the extracted pairs using open-source LLMs to fix formatting & add missing rationales ✨
By fine-tuning base LLMs on this dataset, MAmmoTH2 models achieve huge gains on reasoning benchmarks without seeing in-domain data, e.g.:
- MAmmoTH2-7B: Mistral-7B MATH 11% → 34% 🧮
- MAmmoTH2-8B: Llama-3-8B TheoremQA 20% → 30% 🎓
Further fine-tuning MAmmoTH2 on open-source instruction data, including OpenHermes 2.5, Code Feedback, and Math-Aug (our newly constructed), leads to MAmmoTH2-Plus.
🌟 MAmmoTH2-8×7B-Plus (Mixtral MoE base) matches 110B Qwen's reasoning performance with only 13B params while achieving impressive chat abilities, scoring 33.8 on AlpacaEval 2.0 and 32.6 on ArenaHard. 🏆
🌟MAmmoTH2-8B-Plus and Llama-3-8B-Instruct, both derived from Llama-3-base and tuned on 10M-sized datasets, provide an apples-to-apples comparison. Despite Llama-3-8B-Instruct using human-annotated data and MAmmoTH2-8B-Plus not, our model surpasses Llama-3-8B-Instruct by 6 points in reasoning tasks, showcasing the power of our annotation-free approach. 💡
Our work demonstrates a new paradigm for building powerful instruction-tuned models by efficiently harvesting large-scale, high-quality data from the web without costly human annotation or GPT-4 distillation. 💸🧠
An online demo (for reasoning and general chat; Yes, our model is not just intended for reasoning!) is available at: huggingface.co/spaces/TIGER-L…
Kudos to the co-lead: @WenhuChen and the team: Tuney Zheng @GeZhang86038849!
Introducing 🦣MAmmoTH2 (tiger-ai-lab.github.io/MAmmoTH2/), a family of open-source models that unlocks unprecedented performance gains in reasoning tasks by leveraging two key insights:
1️⃣ Scaling is still all you need, even in the instruction tuning phase! 📈
2️⃣ High-quality, diverse instructions naturally exist on the web and can be efficiently harvested without costly human annotation. 💸
Our 3-step pipeline mines 10M instruction-response pairs from web data:
1️⃣ Recall relevant docs from Common Crawl using fastText 🔍
2️⃣ Extract instruction-response pairs from docs using open-source LLMs 🪄
3️⃣ Refine the extracted pairs using open-source LLMs to fix formatting & add missing rationales ✨
By fine-tuning base LLMs on this dataset, MAmmoTH2 models achieve huge gains on reasoning benchmarks without seeing in-domain data, e.g.:
- MAmmoTH2-7B: Mistral-7B MATH 11% → 34% 🧮
- MAmmoTH2-8B: Llama-3-8B TheoremQA 20% → 30% 🎓
Further fine-tuning MAmmoTH2 on open-source instruction data, including OpenHermes 2.5, Code Feedback, and Math-Aug (our newly constructed), leads to MAmmoTH2-Plus.
🌟 MAmmoTH2-8×7B-Plus (Mixtral MoE base) matches 110B Qwen's reasoning performance with only 13B params while achieving impressive chat abilities, scoring 33.8 on AlpacaEval 2.0 and 32.6 on ArenaHard. 🏆
🌟MAmmoTH2-8B-Plus and Llama-3-8B-Instruct, both derived from Llama-3-base and tuned on 10M-sized datasets, provide an apples-to-apples comparison. Despite Llama-3-8B-Instruct using human-annotated data and MAmmoTH2-8B-Plus not, our model surpasses Llama-3-8B-Instruct by 6 points in reasoning tasks, showcasing the power of our annotation-free approach. 💡
Our work demonstrates a new paradigm for building powerful instruction-tuned models by efficiently harvesting large-scale, high-quality data from the web without costly human annotation or GPT-4 distillation. 💸🧠
An online demo (for reasoning and general chat; Yes, our model is not just intended for reasoning!) is available at: huggingface.co/spaces/TIGER-L…
Kudos to the co-lead: @WenhuChen and the team: Tuney Zheng @GeZhang86038849!
9
Takeaway 8:
To obtain reward signals from the noisy verifiable data WebInstruct, the ruled-based verifier after filtering the prompt set for short-form answers works the best.
To obtain reward signals from the noisy verifiable data WebInstruct, the ruled-based verifier after filtering the prompt set for short-form answers works the best.
10
Takeaway 9:
Commonly used pre-training data contain content that shares similar properties (e.g., branching and error validation) to long CoT. The base model might already acquire such skills during pre-training. We use MiniHash to search over OpenWebMath and Perplexity to search over the internet. We found many examples containing such patterns. Here is one example (discourse.mc-stan.org/t/interpretati…)
Commonly used pre-training data contain content that shares similar properties (e.g., branching and error validation) to long CoT. The base model might already acquire such skills during pre-training. We use MiniHash to search over OpenWebMath and Perplexity to search over the internet. We found many examples containing such patterns. Here is one example (discourse.mc-stan.org/t/interpretati…)
11
Takeaway 10:
However, incentivizing these behaviors from the base model may require strict conditions, such as a strong base model.
We greatly thank and appreciate the replication study efforts by @junxian_he However, we found such a seemingly emergent aha moment might need to be further carefully recognized. Sometimes such behaviors already exist in the base model’s output and RL does not substantially enhance them even though the accuracy is boosted.
We think the reasons are likely caused by:
- The base model, being relatively small (7B parameters), may lack the capacity to quickly develop such complex abilities when incentivized.
- The model might have been overexposed to MATH-like short instruction data during (continual) pre-training and annealing, leading to overfitting and hindering the development of long CoT behaviors.
However, incentivizing these behaviors from the base model may require strict conditions, such as a strong base model.
We greatly thank and appreciate the replication study efforts by @junxian_he However, we found such a seemingly emergent aha moment might need to be further carefully recognized. Sometimes such behaviors already exist in the base model’s output and RL does not substantially enhance them even though the accuracy is boosted.
We think the reasons are likely caused by:
- The base model, being relatively small (7B parameters), may lack the capacity to quickly develop such complex abilities when incentivized.
- The model might have been overexposed to MATH-like short instruction data during (continual) pre-training and annealing, leading to overfitting and hindering the development of long CoT behaviors.
Junxian He@junxian_he
We replicated the DeepSeek-R1-Zero and DeepSeek-R1 training on 7B model with only 8K examples, the results are surprisingly strong.
🚀 Starting from Qwen2.5-Math-7B (base model), we perform RL on it directly. No SFT, no reward model, just 8K MATH examples for verification, the resultant model achieves (pass@1) 33.3% on AIME, 62.5% on AMC, and 77.2% on MATH, outperforming Qwen2.5-math-7B-instruct and being comparable to PRIME and rStar-MATH that use >50x more data and more complicated components.
🚀 Increased CoT length and self-reflection emerge
We share the details and our findings in the blog:
hkust-nlp.notion.site/simplerl-reason
Training code and implementation details here: github.com/hkust-nlp/simp…
🚀 Starting from Qwen2.5-Math-7B (base model), we perform RL on it directly. No SFT, no reward model, just 8K MATH examples for verification, the resultant model achieves (pass@1) 33.3% on AIME, 62.5% on AMC, and 77.2% on MATH, outperforming Qwen2.5-math-7B-instruct and being comparable to PRIME and rStar-MATH that use >50x more data and more complicated components.
🚀 Increased CoT length and self-reflection emerge
We share the details and our findings in the blog:
hkust-nlp.notion.site/simplerl-reason
Training code and implementation details here: github.com/hkust-nlp/simp…
12
Takeaway 11:
We compared the RL from the Base Model and RL from Long CoT SFT and found that RL initialized from the long CoT SFT model outperforms RL from the base model.
We compared the RL from the Base Model and RL from Long CoT SFT and found that RL initialized from the long CoT SFT model outperforms RL from the base model.
13
We finally discuss the future directions.
[Discussion 1]:
We believe that the model size is the primary factor limiting the emergence of the behavior as also highlighted by @hwchung27 in his recent talk
[Discussion 1]:
We believe that the model size is the primary factor limiting the emergence of the behavior as also highlighted by @hwchung27 in his recent talk
Hyung Won Chung@hwchung27
Here is my talk at @MIT (after some delay😅)
I made this talk last year when I was thinking about a paradigm shift. This delayed posting is timely as we just released o1, which I believe is a new paradigm.
It's a good time to zoom out for high level thinking.
(1/11)
I made this talk last year when I was thinking about a paradigm shift. This delayed posting is timely as we just released o1, which I believe is a new paradigm.
It's a good time to zoom out for high level thinking.
(1/11)
14
[Discussion 2]:
However, scaling up with model size for long CoT with open-source RL Infrastructure still faces challenges. We observe that open-source RL frameworks (e.g., OpenRLHF) often coordinate multiple systems optimized for different training and inference workloads, leading to multiple copies of model parameters being stored in memory. Additionally, algorithms like PPO alternate between these workloads synchronously and sequentially, further limiting efficiency.
However, scaling up with model size for long CoT with open-source RL Infrastructure still faces challenges. We observe that open-source RL frameworks (e.g., OpenRLHF) often coordinate multiple systems optimized for different training and inference workloads, leading to multiple copies of model parameters being stored in memory. Additionally, algorithms like PPO alternate between these workloads synchronously and sequentially, further limiting efficiency.
15
[Discussion 3]:
We also explored REINFORCE++ as a faster alternative to PPO for scaling up data. However, we found it to be more unstable than PPO, leading to lower training accuracies in the long CoT setting. As this instability may be due to an untuned setup, we refrain from making general claims about the algorithm. We present this as an observation that may be useful to the community.
We also explored REINFORCE++ as a faster alternative to PPO for scaling up data. However, we found it to be more unstable than PPO, leading to lower training accuracies in the long CoT setting. As this instability may be due to an untuned setup, we refrain from making general claims about the algorithm. We present this as an observation that may be useful to the community.
16
[Discussion 4]:
While our findings demonstrate that combining rule-based verifiers with prompt set filtering is highly effective, designing such rules and curating prompt sets across different domains remains labor-intensive.
However, as highlighted in The Bitter Lesson, manually encoding human intuition tends to be an inefficient long-term strategy. This raises an intriguing question: how can verification signals be scaled effectively? Is there an equivalent of pre-training in the context of designing RL environments? We look forward to future research on silver supervision signals and the potential for self-supervised approaches in RL verification.
While our findings demonstrate that combining rule-based verifiers with prompt set filtering is highly effective, designing such rules and curating prompt sets across different domains remains labor-intensive.
However, as highlighted in The Bitter Lesson, manually encoding human intuition tends to be an inefficient long-term strategy. This raises an intriguing question: how can verification signals be scaled effectively? Is there an equivalent of pre-training in the context of designing RL environments? We look forward to future research on silver supervision signals and the potential for self-supervised approaches in RL verification.
17
[Discussion 5]:
Reasoning is a latent capability in base models that has only recently been unlocked. Our analysis suggests that one possible source of this emergent thinking is human dialogue on Internet discussion forums. This raises a broader question: what other abilities exist, waiting to be elicited from the vast reservoir of human knowledge and experience embedded in pre-training data? We look forward to more detailed analyses tracing model behaviors back to their data origins.
Reasoning is a latent capability in base models that has only recently been unlocked. Our analysis suggests that one possible source of this emergent thinking is human dialogue on Internet discussion forums. This raises a broader question: what other abilities exist, waiting to be elicited from the vast reservoir of human knowledge and experience embedded in pre-training data? We look forward to more detailed analyses tracing model behaviors back to their data origins.
18
This project was jointly led by @eddy_data3 @tongyx361, and collaborated with @sirius_ctrl and @gneubig. We started the project after o1 was released in Sep 2024. It took a really long journey to try different approaches and conduct rigorous studies to understand the long CoT behaviors in a resource-limited academic environment. It is also great to see that we independently adopted a similar technical route (RL with rule-based outcome reward) as Deepseek R1.
19
We would also thank the Deepseek R1 and the Kimi k1.5 team for releasing their technical report that inspired us to conduct additional experiments. We would also especially thank @junxian_he and @jiayi_pirate for their replication study SimpleRL and TinyZero. The private discussions with them also helped the verification of part of the paper’s claims. We are also grateful to many others who have helped but are not mentioned here.
20
We release every experimental setting in our code repo to facilitate the community understanding of the long CoT reasoning in LLMs! github.com/eddycmu/demyst…
