Claims in the system card indicate Opus 4.5 as a possibly big jump...
Eli Lifland@eli_lifland
17 views
Dec 01, 2025
~2 min read
Advertisement
1
Claims in the system card indicate Opus 4.5 as a possibly big jump in AI R&D and coding capabilities, with employees saying it gives them a 2x boost and Opus performing better than humans on 4-8 hour tasks. Looking forward to third-party evals and field reports.
More in thread.
More in thread.
Claude@claudeai
Introducing Claude Opus 4.5: the best model in the world for coding, agents, and computer use.
Opus 4.5 is a step forward in what AI systems can do, and a preview of larger changes to how work gets done.
Opus 4.5 is a step forward in what AI systems can do, and a preview of larger changes to how work gets done.
2
On Anthropic's AI R&D suite 1, Opus 4.5 is similar to or better than human baselines on the majority of tasks. Summing up the results:
(a) Roughly equivalent to human given 8 hours
(b) A bit better than human given 4 hours
(c) Close to human given 8 hours
(d) Better than human given 4-8 hours
(e) Somewhat better than human given 4 hours
(f) Worse than human given 40 hours
So based on these very few data points maybe an 8-24 hour 50% time horizon on this task distribution?
(a) Roughly equivalent to human given 8 hours
(b) A bit better than human given 4 hours
(c) Close to human given 8 hours
(d) Better than human given 4-8 hours
(e) Somewhat better than human given 4 hours
(f) Worse than human given 40 hours
So based on these very few data points maybe an 8-24 hour 50% time horizon on this task distribution?
3
On AI R&D suite 2, less details are given but Opus 4.5 also surpasses the rule-out threshold on this suite.
4
Perhaps the most interesting thing is that researchers give a 2x median for how much it speeds up their labor (!!). Compare this to 1.25x for Sonnet 4.5, albeit Sonnet had a smaller sample size (6 instead of 18).
5
Overall, Anthropic says they can't rule out ASL-4 AI R&D capabilities based on their benchmark results. The rule-out instead comes from Anthropic employees saying that it can't fully automate entry-level remote-only roles.
They say: "We might build more tasks to test capabilities in this regime, but evaluations of model capabilities might also need to depend on other, novel measures, as well as expert judgments."
They say: "We might build more tasks to test capabilities in this regime, but evaluations of model capabilities might also need to depend on other, novel measures, as well as expert judgments."
6
They separately say:
> Given the same information and access as a remote-only research engineer, we believe the
model would face serious struggles. For example, we believe Claude Opus 4.5 would fail to problem-solve, investigate, communicate, and collaborate in the way a junior researcher could; it would likely fail to have the coherence and self-guided success over multiple
weeks we expect of a junior researcher; and it would lack the broad situational judgment and necessary collaborative ability that characterizes long-term human work. That said, we think it is plausible that models equipped with highly effective scaffolding may not be very far away from this AI R&D-4 threshold.
> Given the same information and access as a remote-only research engineer, we believe the
model would face serious struggles. For example, we believe Claude Opus 4.5 would fail to problem-solve, investigate, communicate, and collaborate in the way a junior researcher could; it would likely fail to have the coherence and self-guided success over multiple
weeks we expect of a junior researcher; and it would lack the broad situational judgment and necessary collaborative ability that characterizes long-term human work. That said, we think it is plausible that models equipped with highly effective scaffolding may not be very far away from this AI R&D-4 threshold.
7
We also see huge improvements on Cybench, a benchmark that tests AIs' cybersecurity abilities.
8
And the cherry on top, 4.5 Opus is also 3x cheaper than 4.1
9
It looks like 4.5 Opus might be a big jump in AI R&D and coding capabilities, in which case it would shorten my timelines at least a little. I await third-party evaluations, uplift studies, and reports from researchers+engineers regarding performance on real-world tasks, to get a better sense of whether this is the case.
10
Link to system card: assets.anthropic.com/m/64823ba74853…
11
I didn't even catch the attached quote at first from the blog post anthropic.com/news/claude-op… re: Opus crushing an engineering hiring take-home exam.
