The LLM Fallacy: Misattribution in AI-Assisted Cognitive Workflows

The rapid integration of large language models (LLMs) into everyday workflows has transformed how individuals perform cognitive tasks such as writing, programming, analysis, and multilingual communication. While prior research has focused on model reliability, hallucination, and user trust calibration, less attention has been given to how LLM usage reshapes users’ perceptions of their own capabilities. This paper introduces the LLM fallacy, a cognitive attribution error in which individuals misinterpret LLM-assisted outputs as evidence of their own independent competence, producing a systematic divergence between perceived and actual capability. We argue that the opacity, fluency, and low-friction interaction patterns of LLMs obscure the boundary between human and machine contribution, leading users to infer competence from outputs rather than from the processes that generate them. We situate the LLM fallacy within existing literature on automation bias, cognitive offloading, and human–AI collaboration, while distinguishing it as a form of attributional distortion specific to AI-mediated workflows. We propose a conceptual framework of its underlying mechanisms and a typology of manifestations across computational, linguistic, analytical, and creative domains. Finally, we examine implications for education, hiring, and AI literacy, and outline directions for empirical validation. We also provide a transparent account of human–AI collaborative methodology. This work establishes a foundation for understanding how generative AI systems not only augment cognitive performance but also reshape self-perception and perceived expertise.

The rapid integration of large language models (LLMs) into everyday workflows has reshaped how individuals perform cognitive tasks, including writing, programming, analysis, and multilingual communication (Achiam et al., 2023). Beyond incremental productivity gains, this shift reflects a structural change in how cognitive labor is organized, with generative systems functioning as embedded components of knowledge work rather than external tools (Brynjolfsson et al., 2025). LLMs do not merely augment isolated tasks but alter the conditions under which problem solving and content generation occur.

Existing research has primarily focused on system-level concerns such as model reliability, hallucination, and user trust calibration (Ji et al., 2023). Parallel work on alignment and human–AI interaction has emphasized improving system behavior through interpretability, controllability, and responsiveness to user intent (Ouyang et al., 2022). While these directions provide important insights into model performance and interaction design, they remain largely system-centered. Comparatively less attention has been given to how sustained interaction with LLMs reshapes users’ perceptions of their own capabilities, particularly in contexts where outputs are coproduced through iterative human–AI exchange.

This paper introduces the concept of the LLM fallacy, defined as a cognitive attribution error in which individuals misinterpret LLM-assisted outputs as evidence of their own independent competence, producing a systematic divergence between perceived and actual capability. The phenomenon can be situated within a broader class of attributional distortions in which individuals misjudge the sources of their performance or knowledge (Kruger & Dunning, 1999).

However, unlike prior accounts that emphasize internal limitations in self-assessment, the LLM fallacy arises from the integration of external generative systems into cognitive workflows, creating hybrid environments in which authorship and agency are not readily separable.

As LLMs mediate an increasing share of cognitive processes, this misattribution produces a systematic divergence between perceived and actual capability. These divergences extend beyond individual misjudgment to affect institutional systems that rely on observable outputs as proxies for competence (Espeland & Sauder, 2007). When outputs can be produced through human–AI collaboration without corresponding internal expertise, evaluation frameworks risk conflating system-assisted performance with independently grounded skill.

We argue that this phenomenon emerges from the interaction of several properties inherent to LLM systems, including output fluency, interactional immediacy, and the opacity of underlying computational processes (Burrell, 2016). High fluency can function as a metacognitive cue, leading users to infer competence from surface-level coherence rather than from the processes that generate it (Reber et al., 2004). At the same time, the abstraction of intermediate computational steps obscures the division of labor between human and system, limiting users’ ability to accurately attribute contributions.

Taken together, these conditions produce a form of attributional ambiguity that is not incidental but structurally embedded within the interaction. The boundary between human contribution and machine generation is progressively reconstructed through repeated use, leading users to internalize system-assisted outputs as reflections of their own ability. This paper makes the following contributions. First, it formally defines the LLM fallacy as a cognitive attribution error specific to AI-mediated environments. Second, it differentiates this phenomenon from adjacent constructs, including hallucination, automation bias, and cognitive offloading. Third, it proposes a mechanistic account explaining how LLM interaction produces attributional ambiguity. Fourth, it introduces a multi-domain typology of manifestations across computational, linguistic, analytical, creative, epistemic, and professional contexts. Fifth, it analyzes implications for evaluation systems, including hiring, education, and expertise signaling. Sixth, it provides a structured account of human–AI collaborative methodology to ensure transparency in research practice. Finally, it outlines testable hypotheses and directions for empirical validation.

3 Conceptual Definition of the LLM Fallacy

The LLM fallacy is defined as a cognitive attribution error in which individuals misinterpret LLM-assisted outputs as evidence of their own independent competence. This occurs when system contributions are cognitively absorbed into the user’s self-assessment, producing a misalignment between actual and perceived capability (Kruger & Dunning, 1999). More broadly, the phenomenon can be understood as a failure of metacognitive monitoring, in which individuals are unable to accurately assess the sources and limits of their own knowledge (Wilson & Dunn, 2004).

For the LLM fallacy to occur, several conditions must be met. First, the task must involve LLM-mediated output generation, where the system produces content that would otherwise require domain expertise. Second, the interaction must be sufficiently seamless that the distinction between human input and system output is not salient. Third, the output must exhibit a level of fluency or coherence typically associated with skilled human performance (Alter & Oppenheimer, 2009). Under these conditions, users are more likely to rely on surface-level cues as proxies for competence rather than evaluating the underlying generative process (Reber et al., 2004).

It is important to distinguish the LLM fallacy from related phenomena, particularly hallucination. Hallucination refers to cases in which a model produces incorrect or fabricated information, representing a failure at the level of system output (Ji et al., 2023). The LLM fallacy, by contrast, is independent of output correctness and instead concerns how outputs are cognitively interpreted. It persists regardless of whether generated content is accurate or erroneous, as it operates at the level of attribution rather than epistemic validity.

The LLM fallacy is also distinct from automation bias and cognitive offloading. Automation bias involves over-reliance on system outputs, while cognitive offloading involves delegating mental effort to external systems (Risko & Gilbert, 2016). Both focus on task execution and decision-making processes. The LLM fallacy instead concerns how outputs are integrated into the user’s self-perception of competence, extending beyond reliance into the domain of capability attribution.

At its core, the phenomenon reflects an attributional misalignment between human and system contributions. In LLM-mediated workflows, the boundary between user input and system-generated output becomes increasingly opaque, making their respective roles difficult to disentangle (Burrell, 2016). This opacity limits the user’s ability to construct accurate mental models of the generative process, increasing reliance on inferred rather than observed causality (Nisbett & Wilson, 1977). As a result, users may disproportionately attribute outputs to themselves, even when generation is largely system-driven. This misalignment defines the LLM fallacy and provides the foundation for the mechanisms and manifestations examined in subsequent sections.

4 Mechanism of Emergence

The LLM fallacy emerges from the interaction of multiple cognitive and system-level factors that jointly produce attributional ambiguity in LLM-mediated workflows. These factors reinforce one another, creating conditions under which users systematically misattribute system-generated outputs as reflections of their own competence (Sloman, 1996). From a dual process perspective, such misattributions arise when fast, intuitive judgments dominate reflective evaluation, allowing surface-level cues to guide inference.

A primary mechanism is attribution ambiguity between human input and model output. In LLM interactions, users provide prompts that are often partial, underspecified, or iterative, while the system produces structured and coherent outputs. Because results emerge through a continuous interaction loop, the boundary between user contribution and system generation becomes difficult to delineate. This ambiguity increases the likelihood that users incorporate outputs into their sense of authorship, constructing post hoc accounts of their role despite limited introspective access to underlying processes (Nisbett & Wilson, 1977). Research on agency further shows that authorship is often inferred from outcomes rather than directly accessed, leading to systematic illusions of control and contribution (Aarts et al., 2005). In human–AI contexts, this dynamic is amplified: users may not fully experience ownership of generated content at a cognitive level yet still declare authorship at a reflective or social level, revealing a divergence between experienced and attributed authorship (Draxler et al., 2024). Similar dissociations appear in skilled action, where individuals attribute outcomes to themselves despite incomplete awareness of the processes that produced them (Logan & Crump, 2010).

A second mechanism is the fluency illusion produced by high-quality natural language generation. LLM outputs are typically grammatically correct, contextually appropriate, and stylistically consistent, closely resembling skilled human performance. This surface-level fluency functions as a heuristic cue, leading users to infer competence from ease of processing rather than from the generative process (Reber et al., 2004). Fluency also biases judgments of credibility and expertise, increasing the likelihood that outputs are perceived as accurate and skillfully produced even in the absence of deeper evaluation (Metzger & Flanagin, 2013).

Cognitive outsourcing further contributes to the phenomenon. LLMs allow users to externalize complex tasks, including reasoning, composition, and problem solving, with minimal effort. As the system assumes a greater share of cognitive workload, users engage less with the processes required to produce outputs, weakening their ability to assess their own understanding or skill (Kirsh, 2010). Repeated reliance reduces opportunities for self-generated reasoning, reinforcing the gap between perceived and actual competence.

Another critical factor is pipeline opacity, referring to the invisibility of the processes that generate LLM outputs. Unlike traditional tools, where intermediate steps are observable or user-driven, LLMs abstract away retrieval, pattern matching, and synthesis. This opacity prevents users from tracing how outputs are produced and obscures the distinction between system-driven and user-driven contributions (Ananny & Crawford, 2018). In the absence of transparent intermediate steps, users rely on incomplete mental representations of the system, increasing the likelihood of attribution errors.

Taken together, these mechanisms produce perceived competence inflation. Attribution ambiguity obscures authorship, fluency signals capability, cognitive outsourcing reduces reflective engagement, and pipeline opacity removes visibility into the generative process. Their interaction creates a structurally reinforced environment in which users are consistently inclined to overestimate their own independent competence, giving rise to the LLM fallacy. Formally, this relationship can be summarized as follows: capability divergence (ΔC, defined as the gap between perceived and actual capability) emerges from the interaction of system-level properties, namely opacity, fluency, and immediacy, mediated by attribution ambiguity and cognitive outsourcing. As illustrated in Figure 1, these interacting components collectively produce capability divergence through mediated attribution processes.

5 Typology of LLM Fallacy Manifestations

The LLM fallacy manifests across multiple domains in which LLMs are used to perform cognitively demanding tasks. While the underlying attributional mechanism remains consistent, its form and visibility vary by task and output type. This section outlines key domains in which the phenomenon is most salient, illustrating how perceived competence inflation emerges across contexts (Dunning, 2011). Across domains, the common structure is a dissociation between externally supported performance and internally grounded understanding.

In the computational domain, the LLM fallacy appears in coding and systems building, where users generate functional scripts or applications with LLM assistance. Users may produce working outputs without understanding underlying architecture, dependencies, or optimization strategies. Execution is thus misinterpreted as evidence of technical competence (Newell & Simon, 1976), reflecting a distinction between externally scaffolded performance and internally developed understanding.

In the linguistic domain, the phenomenon emerges in multilingual production, where users generate fluent text in languages they do not independently command. Because LLM outputs are often grammatically accurate and contextually appropriate, users may conflate fluency with internalized language ability, overestimating their capacity to comprehend or produce language without assistance (Bender & Koller, 2020). This reflects a gap between surface form and semantic competence.

In the analytical domain, the LLM fallacy manifests in reasoning and problem-solving tasks. LLMs can generate structured explanations and step-by-step analyses, which users may adopt or reproduce. Exposure to such outputs can create the impression of possessing analytical skill, even when the underlying reasoning process is externally generated (Evans, 2008). Users may internalize the structure of reasoning without engaging in the processes required to produce it independently.

In the creative domain, the phenomenon appears in writing, ideation, and content generation. LLMs assist in producing narratives, arguments, and stylistically refined text, which users may incorporate into their work. These outputs may be misattributed as evidence of personal creativity or authorship despite substantial system contribution (Latour, 1999), reflecting a redistribution of creative agency across human and machine.

In the epistemic domain, the LLM fallacy is observed in knowledge acquisition and understanding. LLMs can summarize complex materials and generate accessible explanations, leading users to equate access to information with conceptual mastery. This aligns with the illusion of explanatory depth, in which individuals overestimate their understanding of complex systems (Rozenblit & Keil, 2002).

In the domain of professional signaling, the phenomenon manifests in how individuals represent their capabilities in external contexts such as resumes or interviews. Users may report skills based on their ability to produce outputs with LLM assistance rather than independently acquired expertise, resulting in inflated representations of competence that may not transfer to unaided performance (Lamont, 2012). Together, these domains illustrate the breadth of the LLM fallacy across forms of cognitive work. Despite variation in task type, each domain reflects the same underlying pattern: the conflation of system-assisted output with internally grounded competence.