METR’s analysis of this experiment is wildly misleading. The...

@eshear
Emmett Shear@eshear
24 views Jul 16, 2025 ~3 min read
Advertisement
1
METR’s analysis of this experiment is wildly misleading. The results indicate that people who have ~never used AI tools before are less productive while learning to use the tools, and say ~nothing about experienced AI tool users. Let's take a look at why.
@METR_Evals
METR@METR_Evals
At the beginning of the study, developers forecasted that they would get sped up by 24%. After actually doing the work, they estimated that they had been sped up by 20%. But it turned out that they were actually slowed down by 19%.
Media image
2
I immediately found the claim suspect because it didn't jibe with my own experience working w people using coding assistants, but sometimes there are surprising results so I dug in. The first question: who were these developers in the study getting such poor results?
3
“We recruited 16 experienced open-source developers to work on 246 real tasks in their own repositories (avg 22k+ stars, 1M+ lines of code).” So they sound like reasonably experienced software devs.
"Developers have a range of experience using AI tools: 93% have prior experience with tools like ChatGPT, but only 44% have experience using Cursor." Uh oh. So they haven't actually used AI coding tools, they've like tried prompting an LLM to write code for them. But that's an entirely different kind of experience, as anyone who has used these tools can tell you.
They claim "a range of experience using AI tools", yet only a single developer of their sixteen had more than a single week of experience using Cursor. They make it look like a range by breaking "less than a week" into <1 hr, 1-10hrs, 10-30hrs, and 30-50hrs of experience. Given the long steep learning curve for effectively using these new AI tools well, this division betrays what I hope is just grossly negligent ignorance about that reality, rather than intentional deception.
Of course, the one developer who did have more than a week of experience was 20% faster instead of 20% slower. The authors note this fact, but then say “We are underpowered to draw strong conclusions from this analysis” and bury it in a figure’s description in an appendix.
If the authors of the paper had made the claim, "We tested experienced developers using AI tools for the first time, and found that at least during the first week they were slower rather than faster" that would have been a modestly interesting finding and true. Alas, that is not the claim they made.
Media image
4
METR published and promoted this paper with a provocative, misleading headline and summary and body. They buried the fact that the single experienced developer DID have significant gains, contradicting the headline. I hope that the authors withdraw this paper or at the very least update it to limit the claims.
The study does appear mostly well designed in other respects, although I would want to audit it carefully before accepting anything at face value. It does seem that developers and experts overestimate how much value a developer will get from using an AI tool during the first week they use it.
Media image
5
A more interesting study would include inexperienced devs, and those who use the tools already, and those who have invested work into optimizing usage of the tools. Perhaps someone with experience using the tools would be able to design such a future study?
6
It is clear that the source of disagreement is that I think using Cursor effectively is a distinct skill from talking to ChatGPT while you program and expect fairly low transfer, and the authors think it's the similar skill and expect much higher.

@idavidrein
david rein@idavidrein
@eshear @ohabryka Devs had roughly the following prior LLM experience:
- 7/16 had >100s of hours
- 7/16 had 10-100 hours
- 2/16 had 1-10 hours

We think describing this as "moderate AI experience" is fair, my guess is we'll have to agree to disagree, but appreciate the feedback!
7
I think conflating the two completely invalidates the study's headline and summary results. I suppose the future will tell if this is the case. I'm glad to have found the underlying disagreement.
8
Apparently many of the users with less than 50 hours of experience in the paper actually had more? If true, this basically invalidates all findings in the paper, since it indicates flaws in the data collection process.

@joel_bkr
Joel Becker@joel_bkr
@eshear @paravn gently noting that it is not quite a single exception x.com/ruben_bloom/st…
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement