Recently had a good chat with @tamaybes. He thinks we aren’t yet in...

OpenAI’s RFT team, Thinking Machines, and Applied Compute are all pitching customers on RL as a service.
So what happens if o5 zero-shots all customer tasks?
But of course humans are far more sample efficient at it than current RL—maybe that’s emergent generalization?
I do think LLMs are getting good enough at code to implement novel environments (got a sanskrit poetry env ~90% right).
Alternatively, methods for turning pretraining web data into RL tasks.
Verifiers are only useful to identify the small % of good answers, and if o5 doesn’t improve with custom RL, that % must already be high.
One can always contrive a task with e.g. the exact same prompts as one in the training distribution, but inverse rewards.
For example, I wouldn’t be surprised if a model RL’d on sufficiently many computer-use tasks etc. generalized to all standard white-collar workflows.
This may be bad news for some YC startups.
The ideal RL task for a startup is not just verifiable & high value, but also idiosyncratic.
