Experienced developers predicted AI would make them 24% faster, then, in a randomized trial, they were 19% slower.

This explores why the people closest to the work expected large speedups that the measurements didn't show. The starting point is a randomized trial with experienced open-source developers. Before starting, they forecast that AI would make them 24% faster. On 246 real tasks they were actually 19% slower, and economists and ML experts also overestimated the gains Do AI coding tools actually speed up experienced developers?. The study itself names over-optimism as one reason. The collection doesn't directly study why people forecast badly. What it does show is that the evidence those forecasts rest on gives a misleading picture of real work.

The first misleading source is benchmarks. An analysis of 960 real occupational workflows found that AI agents do well on short, self-contained, contest-style problems and fail at the long, messy tasks that make up professional work. The authors argue that the gap comes from what the field chose to measure, more than from what the models can do Why do agent benchmarks not predict real economic value?. If your sense of AI comes from leaderboard wins, you will overestimate how it performs inside a large, mature codebase that you already know better than any tool does. A related finding is that frontier agents on long research tasks mostly combine techniques that already exist, and they find shortcuts in the evaluator more often than new solutions Do frontier AI agents actually conduct novel research or just optimize?. The impressive results are real, but they come from a narrower kind of skill than they appear to.

The second source is earlier productivity studies, which were also narrower than they looked. The studies that showed AI gains measured people working on tasks they already knew how to do. When workers used AI to learn something new, the gains disappeared and their learning suffered When does AI actually boost worker productivity?. Taking positive results from one kind of task and applying them to all knowledge work produces inflated forecasts. Bolder claims carry the same problem further. The argument that automating AI research could squeeze four or five years of progress into one rests on unproven assumptions: that research results can be checked at scale, and that skill on small tasks carries over to important research Could automated AI research compress years of progress into months?.

The less obvious part is that AI output looks like productivity even when it isn't. One argument in the collection is that AI separates the outward form of intellectual work from the thinking that normally produces it Does AI separate intellectual form from the thinking behind it?. Code that appears quickly feels like progress, while the time spent reviewing, fixing, and fitting it into the codebase is easy to overlook. A related idea is that AI can produce material faster than people can evaluate it Can AI generate knowledge faster than humans can evaluate it?. In both cases the generation is easy to see and the checking is easy to miss. Measurement at the level of science as a whole shows the same split: AI-assisted researchers publish three times as many papers, but the range of topics science covers shrinks and researchers collaborate less Does AI help individual scientists while narrowing scientific focus?. Gains you can see for one person can hide losses across the field.

The takeaway is that the forecasts were reasonable readings of misleading evidence: contest-style benchmarks, studies of people working within their existing skills, and output that looks like progress. The collection has only one direct measurement of the gap between forecast and result, so treat this as a likely explanation rather than a settled answer.