🔥 Why Agentic RL for LLMs is harder (and more practical) than you...
Zhihu Frontier@ZhihuFrontier
38 views
Dec 08, 2025
~2 min read
Advertisement
1
🔥 Why Agentic RL for LLMs is harder (and more practical) than you think — sharing from Zhihu contributor skydownacai
These are hands-on takeaways from months of running agent RL (search agents, data-analysis agents) across dense & MoE models, single-source and multi-source data. Precise, practical, and opinionated — useful if you run RL at scale. 👇
• Stability first.
In production RL, stability > everything. A stable pipeline that can train long and reliably is the precondition for scaling; unstable runs burn time and money. Recent fixes (train/eval mismatch checks, PPO-EWMA variants, etc.) have become default in our stacks. ⚠️
• Environment matters most for agents.
In reasoning RL: Algorithm > Reward > Environment.
In agentic RL: Environment > Reward > Algorithm.
How your tools behave matters more than fancy tricks.
• Tooling reliability is critical.
If your environment can't robustly support large-scale tool calls (APIs, DBs, subprocesses), you'll pay heavy experiment and performance costs. We always fix tool reliability before training a new agent scenario. 🔧
• Watch out for reward hacking with LLM judges.
When there's no ground-truth (math/code), using an LLM "judge" to score can lead to reward-hacking: sudden test jumps that are bogus. We saw multiple false positives in a month. Design and iterate reward pipelines to prevent gaming. 🕵️♂️
• Reward design should evolve.
Hand-crafted rewards drift fast. Continuous iteration is mandatory.
• Eval ≠ Train environment.
If the eval environment uses different tools or constraints, fix them first—otherwise you'll misinterpret training progress. 🧰
• Only ablate after infra+monitoring are ready.
Don't chase algorithmic tweaks until environment & reward systems pass stress tests and monitoring.
• When lots of GPU is available, scale RL compute.
PPO-EWMA + larger batch/group sizes can beat pure on-policy tricks — invest in RL compute to gain learning & exploration efficiency. 🚀
• RL grokking is real.
We observed grokking phenomena in RL (esp. on-policy) — worth studying further.
• Tool-usage monitoring = signal of learning.
Track which tools/files agents actually call; underused tools often cap capability. Usage patterns inform data & interface fixes.
• Big models generalize faster.
Many small-model RL tricks don't transfer; larger models need fewer hacks and learn more robustly.
• Don't overforce entropy tricks.
In continual RL, even with low initial entropy, heavy-handed entropy/clipping tricks aren't always needed — good data selection can let the model naturally increase exploration💡
📖 Full post (CN): zhuanlan.zhihu.com/p/197991342411…
#Agent #LLM #RL #AIInfra #AI #Tech
These are hands-on takeaways from months of running agent RL (search agents, data-analysis agents) across dense & MoE models, single-source and multi-source data. Precise, practical, and opinionated — useful if you run RL at scale. 👇
• Stability first.
In production RL, stability > everything. A stable pipeline that can train long and reliably is the precondition for scaling; unstable runs burn time and money. Recent fixes (train/eval mismatch checks, PPO-EWMA variants, etc.) have become default in our stacks. ⚠️
• Environment matters most for agents.
In reasoning RL: Algorithm > Reward > Environment.
In agentic RL: Environment > Reward > Algorithm.
How your tools behave matters more than fancy tricks.
• Tooling reliability is critical.
If your environment can't robustly support large-scale tool calls (APIs, DBs, subprocesses), you'll pay heavy experiment and performance costs. We always fix tool reliability before training a new agent scenario. 🔧
• Watch out for reward hacking with LLM judges.
When there's no ground-truth (math/code), using an LLM "judge" to score can lead to reward-hacking: sudden test jumps that are bogus. We saw multiple false positives in a month. Design and iterate reward pipelines to prevent gaming. 🕵️♂️
• Reward design should evolve.
Hand-crafted rewards drift fast. Continuous iteration is mandatory.
• Eval ≠ Train environment.
If the eval environment uses different tools or constraints, fix them first—otherwise you'll misinterpret training progress. 🧰
• Only ablate after infra+monitoring are ready.
Don't chase algorithmic tweaks until environment & reward systems pass stress tests and monitoring.
• When lots of GPU is available, scale RL compute.
PPO-EWMA + larger batch/group sizes can beat pure on-policy tricks — invest in RL compute to gain learning & exploration efficiency. 🚀
• RL grokking is real.
We observed grokking phenomena in RL (esp. on-policy) — worth studying further.
• Tool-usage monitoring = signal of learning.
Track which tools/files agents actually call; underused tools often cap capability. Usage patterns inform data & interface fixes.
• Big models generalize faster.
Many small-model RL tricks don't transfer; larger models need fewer hacks and learn more robustly.
• Don't overforce entropy tricks.
In continual RL, even with low initial entropy, heavy-handed entropy/clipping tricks aren't always needed — good data selection can let the model naturally increase exploration💡
📖 Full post (CN): zhuanlan.zhihu.com/p/197991342411…
#Agent #LLM #RL #AIInfra #AI #Tech
