Recently had a good chat with @tamaybes. He thinks we aren’t yet in...

@khoomeik
Rohan Pandey@khoomeik
27 views Jul 01, 2025 ~2 min read
Advertisement
1
Recently had a good chat with @tamaybes. He thinks we aren’t yet in the GPT-3 era of RL and as it scales, cross-task OOD generalization will emerge.

It’s difficult to empirically study this at current scale, but let’s take it as true—what does this mean for custom RL plays? 🧵
Media image
2
RL is attractive for solving domain-specific tasks because all you need is a small problem set and a verifier.

OpenAI’s RFT team, Thinking Machines, and Applied Compute are all pitching customers on RL as a service.

So what happens if o5 zero-shots all customer tasks?
3
First, I’m not sure I believe this will happen. Humans do seem to do some form of RL when learning a new skill or job, even with thousands of “tasks” in our brains.

But of course humans are far more sample efficient at it than current RL—maybe that’s emergent generalization?
4
To scale multiple OOMs of task diversity, we’ll need novel methods for task creation.

I do think LLMs are getting good enough at code to implement novel environments (got a sanskrit poetry env ~90% right).

Alternatively, methods for turning pretraining web data into RL tasks.
5
If o5 does generalize so well that custom RL has negligible effects, even using a verifier during search wouldn’t improve performance much.

Verifiers are only useful to identify the small % of good answers, and if o5 doesn’t improve with custom RL, that % must already be high.
6
Frankly, the idea of a model that performs *every* task (even ones it has never seen) with such a high pass-rate seems implausible to me.

One can always contrive a task with e.g. the exact same prompts as one in the training distribution, but inverse rewards.
7
But o5 needn’t solve *every* contrived OOD task to tilt the field back to big labs.

For example, I wouldn’t be surprised if a model RL’d on sufficiently many computer-use tasks etc. generalized to all standard white-collar workflows.

This may be bad news for some YC startups.
8
So I think my advice is: if you’re a startup trying to build a moat around a task with custom RL, ask yourself how easily an experienced white collar worker (read: “o5”) could do it.

The ideal RL task for a startup is not just verifiable & high value, but also idiosyncratic.
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement