Do AI coding tools actually speed up experienced developers?

In a randomized controlled trial, allowing AI tools made experienced open-source developers slower: completion time rose 19 percent. Sixteen developers completed 246 tasks, each randomly assigned to allow or disallow early-2025 AI tools, mostly Cursor Pro with Claude 3.5/3.7 Sonnet. Before the tasks they forecast a 24 percent reduction in completion time, and afterward they estimated a 20 percent reduction. The 19 percent is the trial's measurement. The forecasts and estimates are the developers' own, along with economics and ML experts' forecasts of 39 and 38 percent reductions, and the excerpt's authors say both groups "drastically overestimate the usefulness of AI on developer productivity, even after they have spent many hours using the tools."

The design runs on real work. Each developer lists real issues from repositories they regularly contribute to, which average 23,000 stars and 1,100,000 lines of code. Issues are randomized by a simulated coin flip, and the outcome is time to completion, not lines of code or pull requests. The authors note that earlier field studies used those outcomes, though AI can move them without productivity rising. To explain the result, the authors hand-label 143 hours of screen recordings (29 percent of developers' total hours), pull source-code statistics, and survey and interview participants. Of 21 candidate factors, they find evidence that 5 contribute, mixed or no evidence for 10, and evidence against 6. The likely contributors include over-optimism about AI usefulness, high developer familiarity with the repositories, large and complex codebases, low AI reliability (developers accept under 44 percent of generations and spend 9 percent of their time reviewing and cleaning AI output), and implicit repository context.

The result cuts against the speedup figures elsewhere in the library. A randomized trial of 50 designers and 50 product managers links Does Figma Make speed up design task completion? to shorter completion times, so the two trials measure similar outcomes and point in opposite directions. The excerpt's own account of where AI helps is narrow: the slowdown is tied to mature repositories and developers' familiarity with them, and the authors expect that "small greenfield projects or development in unfamiliar codebases" may see substantial speedup. Anthropic's cited speedup of about 3x to about 52x, discussed in Is AI development already being handed to AI systems?, is a different kind of evidence. The excerpt argues for "field experiments with robust outcome measures, compared to relying solely on expert forecasts or developer surveys," and that standard does not reach the lab's own figure. The excerpt also points at where the time goes, decomposing it at about 10-second resolution, but it does not report that decomposition, so it cannot confirm Does AI really save time, or just change how we spend it?.

The excerpt does not establish a general productivity loss. The authors say the slowdown "does not imply that current AI tools do not often improve developer's productivity," and they allow that future models, better scaffolding or domain fine-tuning could change the sign. The evidence is 16 developers in one setting, and the 21 factors are hypotheses, not tested causes. The authors concede that experimental artifacts cannot be entirely ruled out, though they find the slowdown robust across analyses. The implication, at the strength the evidence allows: for experienced developers on mature codebases they know well, early-2025 tools slowed completion, and developers' own speedup estimates are weak evidence about that setting. Claims beyond it need their own measurement.