AI can make work faster, but a fear is that relying on it may make it harder to learn new skills on the job.
We ran an experiment with software engineers to learn more. Coding with AI led to a decrease in mastery—but this depended on how people used it.
anthropic.com/research/AI-as…
In a randomized-controlled trial, we assigned one group of junior engineers to an AI-assistance group and another to a no-AI group.
Both groups completed a coding task using a Python library they’d never seen before. Then they took a quiz covering concepts they’d just used.
Both groups completed a coding task using a Python library they’d never seen before. Then they took a quiz covering concepts they’d just used.
Participants in the AI group finished faster by about two minutes (although this wasn’t statistically significant).
But on average, the AI group also scored significantly worse on the quiz—17% lower, or roughly two letter grades.
But on average, the AI group also scored significantly worse on the quiz—17% lower, or roughly two letter grades.
However, some in the AI group still scored highly while using AI assistance.
When we looked at the ways they completed the task, we saw they asked conceptual and clarifying questions to understand the code they were working with—rather than delegating or relying on AI.
When we looked at the ways they completed the task, we saw they asked conceptual and clarifying questions to understand the code they were working with—rather than delegating or relying on AI.
We were particularly interested in coding because as software engineering grows more automated, humans will still need the skills to catch AI errors, guide its output, and ultimately provide oversight for AI deployed in high-stakes environments.
These results have broader implications—on how to design AI products that facilitate learning, and how workplaces should approach AI policies.
As we also continue to release more capable AI tools, we’re continuing to study their impact on work—at Anthropic, and more broadly.
As we also continue to release more capable AI tools, we’re continuing to study their impact on work—at Anthropic, and more broadly.
For more details on this research, see the full paper: arxiv.org/abs/2601.20245
Generated by Thread Navigator
Press ⌘ + S to quick-export