Did ChatGPT draw the easy questions away from Stack Overflow, leaving harder, unanswered ones behind, or is it more complicated?

This explores whether ChatGPT drew the easy questions away from Stack Overflow, so that what's left (especially what goes unanswered) is harder. The short answer is that the collection doesn't measure question difficulty or unanswered questions directly. What it does have is evidence on what kinds of posts disappeared, and that evidence complicates the intuitive story that only hard questions remain.

The drop itself is well documented. When Stack Overflow is compared with similar Q&A sites that were less exposed to ChatGPT, weekly posts fell about 16% after launch and about 25% within six months Did ChatGPT cause Stack Overflow posting to decline?. The pattern matters here. The decline was steepest for programming languages that ChatGPT had seen the most training data on. That fits the idea of selective displacement: people stopped asking where the chatbot was already good, so questions about less common languages and tools make up a growing share of what remains.

That displacement wasn't limited to easy or low-quality posts, though. Vote scores on Stack Overflow barely changed after ChatGPT's release, which suggests the questions that disappeared included well-received ones and not only duplicates or beginner noise Did ChatGPT displace only low-quality Stack Overflow posts?. One caveat is that votes are a rough stand-in for quality, and no experts checked them. Still, if ChatGPT had pulled away only the trivial questions, you'd expect average quality signals to rise. They didn't, so the leftover questions may not be harder overall. They may just be fewer, and skewed toward less common topics.

There's also a quieter cost on the people asking. In a classroom experiment, students who used ChatGPT scored much higher on coding tasks (89% vs. 69%), but they remembered fewer concepts afterward and claimed ownership of less of their own code Does ChatGPT help students code better but remember less?. If developers solve routine problems privately with a chatbot, they learn less from them, and the public record of solved problems that Stack Overflow built up stops growing. The same thing is happening across the wider web: AI-written articles rose to about half of new content and then plateaued Has AI-generated content stopped growing on the web?.

What's missing is a study that tracks unanswered questions specifically, through answer rates, time to first answer, or how complex questions are. If that's the thread you want to follow, the collection only gets you as far as "the mix shifted, and it isn't simply easy questions leaving." Whether the hard tail is now going unanswered more often is still an open question here.