LLM RL code does not need to be complicated! Here is a minimal...

The repo is designed to be hackable and prioritize ease-of-understanding for research: github.com/kvfrans/lmpo
The only thing I couldn't figure out how to replicate is the HF tokenizer, if anyone knows how, please let me know...
- Multi-host and FSDP sharding works out of the box (single-host works fine too)
- Only bandit environments for now.
