Do AI coding features actually speed up engineer productivity?

The Google study team reports that three AI features already in production in Cider V, Google's internal development environment, significantly shortened the time 96 full-time engineers spent on a complex, enterprise-grade task. The features were AI Code Completion, Smart Paste and Natural Language to Code, and engineers were randomly assigned to use them or not. The best-fit regression (Model 2), controlling for other predictors of time on task, puts the effect at "about 21%." The authors also say their "confidence interval is large," and the excerpt does not give its bounds. The unadjusted comparison is significant (t(83.6) = 2.11, p = .038). In the adjusted model the AI effect is similar in size but misses the p < .05 threshold.

The paper places its figure between two earlier estimates. A randomized trial of GitHub Copilot reported 56% (Peng et al.), but its sample was drawn from freelancers on Upwork, while this study "only sampled full-time Googlers." The authors read the gap mainly as a population difference. They say their figure "aligns with" the 26% throughput gain from Cui et al.'s pooled enterprise analysis. Two subgroup findings carry the paper's larger argument. Developers who spend more hours a day on code were faster with AI, which the authors attribute to current tools requiring "a lot of time verifying and editing code generated by AI." More senior developers also appeared faster. The authors set this against literature suggesting juniors gain more, and offer that AI "is not yet able to close a skill gap when applied to complex tasks." Neither interaction is clean. The code-hours interaction was not significant (β = −0.29), though the model was (p = 0.018), and the excerpt gives no statistic for seniority.

Against the nearest notes, this trial extends the speed side of the picture and leaves the learning side open. Its size is close to the Does Figma Make speed up design task completion? trial, though there product managers gained most, while here seniority and code hours are the moderators. Its only outcome is time on task, so it cannot speak to what Does AI assistance actually harm the way developers learn? measured: conceptual understanding after learning a new library. It also does not test the split in Does ChatGPT help students code better but remember less?, where task scores rose while recall and claimed ownership fell. The conclusion says "questions about the impact of AI on code quality were not explored." The code-hours finding fits the reallocation toward verifying AI output that Does AI really save time, or just change how we spend it? describes.

The excerpt does not establish that the effect carries beyond its setting. The sample is full-time Google engineers with at least a year at Google and proficiency in C++, the tools are Google's own, and the study ran in June and July 2024. The authors themselves decline to assume the result holds at the ecosystem level, across other tool suites, or over time. Because the features come from the company that ran the study, the speedup is the builder's own measurement. With a large interval, the number is better read as a direction than as a rate. The implication is narrower than the headline: AI assistance probably shortened time on this kind of enterprise task for experienced engineers, and any productivity claim built on the 21% needs the caveats the authors attach.