When will AI systems be able to carry out long projects...
METR@METR_Evals
44 views
Mar 20, 2025
~2 min read
Advertisement
1
When will AI systems be able to carry out long projects independently?
In new research, we find a kind of “Moore’s Law for AI agents”: the length of tasks that AIs can do is doubling about every 7 months.
In new research, we find a kind of “Moore’s Law for AI agents”: the length of tasks that AIs can do is doubling about every 7 months.
2
At a high level, our method is simple:
1. We ask both skilled humans and AI systems to attempt tasks in similar conditions.
2. We measure how long the humans take.
3. We then measure how AI success rates vary depending on how long the humans took to do those tasks.
1. We ask both skilled humans and AI systems to attempt tasks in similar conditions.
2. We measure how long the humans take.
3. We then measure how AI success rates vary depending on how long the humans took to do those tasks.
3
We measure human and AI performance on a variety of software tasks, some sourced from existing METR benchmarks like HCAST and some brand new.
Human completion times on these tasks range from 1 second to 16 hours.
Human completion times on these tasks range from 1 second to 16 hours.
david rein@idavidrein
I’m excited to share details about HCAST (Human-Calibrated Autonomy Software Tasks), a benchmark we’ve been developing at METR for the past year to measure the abilities of frontier AI systems to complete diverse software tasks autonomously.
4
We then fit a curve that predicts the success rate of an AI based on how long it took humans to do each task. This curve characterizes how capable an AI is at different task lengths. We then summarize the curve with the task length at which a model’s success rate is 50%.
5
This metric - the 50% task completion time horizon - gives us a way to track progress in model autonomy over time.
Plotting the historical trend of 50% time horizons across frontier AI systems shows exponential growth.
Plotting the historical trend of 50% time horizons across frontier AI systems shows exponential growth.
6
These results appear robust. Although our model could be wrong, we are relatively confident about its fit to the data. While our initial data only covered the most recent systems, we found we could retrodict back to GPT-2.
7
We ran experiments on SWE-bench Verified and found a similar trend. We also ran a small experiment on internal METR pull requests, and found results consistent with our other datasets. We are excited for researchers to extend this and measure time horizons on other benchmarks.
8
We are fairly confident in the rough trend of 1-4 doublings in horizon length per year. That is fast! Measures like these help make the notion of “degrees of autonomy” more concrete and let us quantify when AI abilities may rise above specific useful (or dangerous) thresholds.
Richard Ngo@RichardMCNgo
Instead of treating AGI as a binary threshold, I prefer to treat it as a continuous spectrum defined by comparison to time-limited humans.
I call a system a t-AGI if, on most cognitive tasks, it beats most human experts who are given time t to perform the task.
More details:
I call a system a t-AGI if, on most cognitive tasks, it beats most human experts who are given time t to perform the task.
More details:
9
We give more high-level information about these results and what they might imply on the METR blog: metr.org/blog/2025-03-1…
10
For the details, read “Measuring AI Ability to Complete Long Tasks,” now on available on arXiv: arxiv.org/abs/2503.14499
11
If you are interested in contributing to more research like this, on quantitative evaluation of frontier AI capabilities: METR is hiring!
hiring.metr.org
hiring.metr.org
