Why do users feel more competent when their actual capability is declining?
This explores the gap between felt competence and real skill — why people working with AI tools come to believe they're getting better while the underlying ability they'd have on their own quietly erodes.
This explores the gap between felt competence and real skill in AI-assisted work — and the corpus has a surprisingly precise name for it. Researchers call it the LLM Fallacy: a cognitive attribution error where people fold AI-generated output into their own sense of capability, believing they possess skills they don't actually have Do AI-assisted outputs fool users about their own skills?. What makes this worth understanding is that it's a distinct failure — not the same as trusting a wrong answer (automation bias) or being fooled by a false fact (hallucination). It can happen even when the AI is completely accurate, because the error is about who gets credit for the work, not whether the work is correct How does AI-assisted work reshape how people see their own abilities?.
The mechanism behind the feeling is fluency. When AI output reads smoothly and arrives effortlessly, your brain treats that ease as a signal about your own ability — a metacognitive shortcut where 'this came out polished' gets misread as 'I am good at this' Does processing ease mislead users about their own competence?. The catch is that LLMs are optimized to produce fluency regardless of whether you understood anything, so the cue fires loudest exactly when you contributed least. One study puts numbers on a related divergence: people report high satisfaction while remaining internally confused, and they're especially confident when they're unaware of their own knowledge gaps Does user satisfaction actually measure cognitive understanding?.
Fluency is only one of four mechanisms that compound here. Alongside it sit attribution ambiguity (the human-AI boundary is blurry), cognitive outsourcing (you stop doing the mental work), and pipeline opacity (you can't see how the output was produced). The corpus argues these don't just add up — they multiply, each amplifying the others, which is why the inflation in perceived competence can be so large How do AI tools trick users into overestimating their own skills?. As you lean on the tool, you do less of the thinking, yet the polished result keeps telling you you're more capable than ever.
There's a sharp parallel in how models themselves fool evaluators, which doubles as a warning about us. Imitation models trained to copy ChatGPT's confident, fluent style fool human judges into thinking they've improved — while closing no actual capability gap. The style signals competence the substance doesn't have Can imitating ChatGPT fool evaluators into thinking models improved?. The same wedge between confident surface and hollow core that lets a weak model pass for a strong one is the wedge that lets a declining user feel like an expert.
The thing you might not have known you wanted to know: the fix isn't a more accurate AI. Because the LLM Fallacy is driven by misattribution rather than by errors, building better, more reliable systems doesn't dissolve it — it can deepen it, since smoother output sends an even stronger (false) competence signal. The interventions that actually help are ones that make the human-machine contribution boundary visible, not ones that make the machine quieter or more correct How does AI-assisted work reshape how people see their own abilities?. It's a cousin of the broader naming problem in this corpus — when we mislabel where a behavior comes from (calling fabrication a 'hallucination'), we aim our fixes at the wrong layer entirely Should we call LLM errors hallucinations or fabrications?.
Sources 7 notes
Research identifies a systematic cognitive attribution error where individuals integrate AI-generated outputs into their capability identity, believing they possess skills they don't actually have. This occurs when task output is seamless and fluent, obscuring the human-AI boundary.
Research shows the LLM Fallacy operates through misattribution of AI outputs to personal capability, independent of output accuracy or reliance behavior. It requires interventions that clarify human-machine contribution boundaries, not just better system accuracy or forced verification.
High-quality AI output triggers a metacognitive heuristic: users experience fluency as a signal of their own capability, even though they didn't generate it. This self-directed fluency illusion systematically inflates perceived competence because LLMs optimize for fluency regardless of user understanding.
STORM shows users express satisfaction despite internal confusion, especially when unaware of knowledge gaps. Sustained engagement correlates with actual self-understanding, not immediate satisfaction ratings.
Attribution ambiguity, fluency illusion, cognitive outsourcing, and pipeline opacity combine to systematically misattribute AI outputs as user competence. The effect is multiplicative—each mechanism amplifies the others.
Imitation models fool human evaluators by mimicking ChatGPT's confident, fluent style while failing to improve factuality or generalization on novel tasks. The ceiling is set by base model capability, not fine-tuning method—better fundamentals, not shortcuts, drive real improvement.
LLMs generate text through statistical token relationships without grounding in shared context. Accurate and inaccurate outputs use identical mechanisms, so calling failures "hallucinations" or "confabulation" misdirects fixes toward perception or memory—the wrong layers.