Thoughts on Agentic RL - Reading Cameron Wolfe's Agentic RL: Frameworks and Best Practices

I spent some time relaxing yesterday and reading Cameron Wolfe’s @cwolferesearch writing on Agentic RL: Frameworks and Best Practices. I actually printed it out to read it by the pool and disconnect from devices for a while.

I loved it and learned a ton. I'm sharing my brain dump and notes below. Looking forward to reading more related things and would love to hear recommendations below.

Here’s my brain dump:

I really recommend reading this post for anyone who wants to get a foundation on Agentic RL. Cameron walks through a foundational overview of the topic to ground the reader before diving into a few recent papers in depth, providing his insights along the way. I’ve been trying to take a step back and go back to the basics of post training and RL from first principles so that I can get better intuition for it and have a baseline to go deeper from. Definitely recommend this post as a way to do that - each paper Cameron covers reveals one or more foundational insights about Agentic RL that were good to reinforce, and a few times things clicked for me that I didn’t have a good intuition for before.

The more I read, the more I realize that at its core, the concepts of post training are simple and at least to me, pretty intuitive. While there’s obviously been significant progress rapidly in the space that drive both open and closed model improvements like GLM 5.2 or Claude Fable 5, it’s still pretty new and everyone is figuring things out as they go. This is encouraging to me because it indicates that:

We have not even come close to the ceiling and there’s still incredible potential for improvement to models

I have the opportunity to make significant advancements in the space

Some things I thought were interesting that I noted when reading:

Infrastructure & Systems

Agentic RL has a systems and infrastructure bottleneck, just as much as a model research one. The agentic use of environments plus long horizon tasks that could take hours to complete means the time between a rollout and policy update is both highly variable even within the same batch and could take a very long time. Async rollouts help resolve the variability problem to some degree but as long running tasks become longer the need to optimize async rollouts will become more important to reduce off-policy bias. It feels like there’s a lot of opportunity here for improvements and new ways of thinking about this problem.

Rewards

There’s so much opportunity for breakthroughs in reward building. It feels that a lot of the research has been naive in reward design compared to what feels possible. From experience building agents to perform long running tasks in insurance systems, the criteria for success can be situational, hard to define, and requires judgement from someone with domain knowledge. At times adherence to specific guidelines regarding “how” something should be done, specifically the process that’s taken to get to the final outcome, is just as important as the outcome itself.

I know that Anthropic does evals well. Their blogs on building evals for long running agents are really impressive, and good evals naturally turn into good rewards. The more granular information you can derive from your traces through good evals, the more feedback the model gets during the training process. There’s costs and speed tradeoffs here too that make it harder to justify in some cases, but I think part of the reason Anthropic and OpenAI are leaders are because of reward design.

I’m looking forward to reading Cameron’s blog on Agentic Evals, because I’m sure he has a lot of ideas on the subject.

Training methodology

Stepping up the interaction limit over training time seems to make improvements, and I think there’s a lot of opportunity to develop techniques that follow a “training curriculum”, with a simple example of ScalingInter-RL increasing the task horizon over training time.

Task advantage normalization and similar concepts are good tactics to ensure certain task types or domains don’t get more weight on the overall policy update.

Environments and World models

There are many papers that build their own “world models” or Agentic RL frameworks that define how environments should be designed and how the agent interacts with them. The approaches vary in implementation between papers but the core ideas are the same and have similar shape.

It’s interesting here to me that everyone reached the same conclusion - SFT should be used to train the model on syntax and harness use, RL for training the model on reasoning tasks, instruction following, and process level thinking.

Clearly and obviously it matters that models are trained on the tools that they’ll be using and the harness they sit within. There’s not one framework or world model concept to me that’s been determined to be the “right one”, and we’re all figuring out what works the best depending on what kind of work you do and in what situations the agents will encounter.

The nice part is we that generally speaking, it’s been successful to train models to do tons of different things at the same time, and training a model to be good at one task doesn’t mean it will get worse at others.

Synthetic Data

For the same reason that good rewards are both super important and hard to design, synthetic data generation for post training feels like a cheat code. Especially as models get more capable of complex reasoning, I can see a future where simulated data is as good or better for training general “AGI” capabilities than anything else. It allows you to build complex verifiable rewards for complex tasks that in real life would be extremely hard to verify.

AutoForge paper proposes a pretty cool way of building synthetic data for RL environments.

Looking forward to reading NVIDIA’s paper on the Nemotron Super training process. I know their team loves synthetic data and I’ve read good things about NVIDIA data designer. https://arxiv.org/abs/2512.20856

Data Engineering

We’re just scratching the surface for how data can be prepared for LLM post training (including synthetic data engineering). I thought the tactics used for filtering data with low batch variance was really interesting. It makes sense that we want to make sure tasks are actually teaching the model something new instead of reinforcing something it already knows.

Thanks for reading! Would love to hear anyone's thoughts and get recommendations on what else to read