METR’s analysis of this experiment is wildly misleading. The...


"Developers have a range of experience using AI tools: 93% have prior experience with tools like ChatGPT, but only 44% have experience using Cursor." Uh oh. So they haven't actually used AI coding tools, they've like tried prompting an LLM to write code for them. But that's an entirely different kind of experience, as anyone who has used these tools can tell you.
They claim "a range of experience using AI tools", yet only a single developer of their sixteen had more than a single week of experience using Cursor. They make it look like a range by breaking "less than a week" into <1 hr, 1-10hrs, 10-30hrs, and 30-50hrs of experience. Given the long steep learning curve for effectively using these new AI tools well, this division betrays what I hope is just grossly negligent ignorance about that reality, rather than intentional deception.
Of course, the one developer who did have more than a week of experience was 20% faster instead of 20% slower. The authors note this fact, but then say “We are underpowered to draw strong conclusions from this analysis” and bury it in a figure’s description in an appendix.
If the authors of the paper had made the claim, "We tested experienced developers using AI tools for the first time, and found that at least during the first week they were slower rather than faster" that would have been a modestly interesting finding and true. Alas, that is not the claim they made.
The study does appear mostly well designed in other respects, although I would want to audit it carefully before accepting anything at face value. It does seem that developers and experts overestimate how much value a developer will get from using an AI tool during the first week they use it.

- 7/16 had >100s of hours
- 7/16 had 10-100 hours
- 2/16 had 1-10 hours
We think describing this as "moderate AI experience" is fair, my guess is we'll have to agree to disagree, but appreciate the feedback!




