Hallucinating with AI: AI Psychosis as Distributed Delusions

Abstract: There is much discussion of the false outputs that generative AI systems such as ChatGPT, Claude, Gemini, DeepSeek, and Grok create. In popular terminology, these have been dubbed AI hallucinations. However, deeming these AI outputs “hallucinations” is controversial, with many claiming this is a metaphorical misnomer. Nevertheless, in this paper, I argue that when viewed through the lens of distributed cognition theory, we can better see the dynamic and troubling ways in which inaccurate beliefs, distorted memories and self-narratives, and delusional thinking can emerge through human-AI interactions; examples of which are popularly being referred to as cases of “AI psychosis”. In such cases, I suggest we move away from thinking about how an AI system might hallucinate at us, by generating false outputs, to thinking about how, when we routinely rely on generative AI to help us think, remember, and narrate, we can come to hallucinate with AI. This can happen when AI introduces errors into the distributed cognitive process, but it can also happen when AI sustains, affirms, and elaborates on our own delusional thinking and self-narratives, such as in the case of Jaswant Singh Chail. I also examine how the conversational style of chatbots can lead them to play a dual-function—both as a cognitive artefact and a quasi-Other with whom we co-construct our beliefs, narratives, and our realities. It is this dual function, I suggest, that makes generative AI an unusual, and particularly seductive, case of distributed cognition.

While Clark and Chalmers’ extended mind thesis is likely the most well-known iteration of distributed cognition, there are a cluster of theories that fall within this field. For example, ‘second wave’ extended mind theorists stress that cognitive tools need not have functional parity with the brain but can play a complementary role, extending our cognition by enhancing or enriching it in important ways, such that the cognitive process that spans agent and world emerges through the interaction and would not be the same if you took the tool away (Menary 2007; Sutton 2010). For instance, Evelyn Tribble (2005) describes how actors in the Globe Theatre would use the stage space to help them remember the enormous number of scripts they were expected to have to hand. The use of props, diagrammatic plots, bodily positions, and so on, were used as cognitive tools to help scaffold the actors’ memories—allowing them to enact and swap between a stunningly large number of roles. As John Sutton (2010, 204) stresses, these cognitive tools “are nothing like” the internal mechanisms of organic memory but precisely perform a role that supports the actors’ memories, allowing them to do something that, without these tools, would be nigh on impossible. The information remembered, then, spans the agent and the objects in a way that constitutes a “new systemic whole” (Heersmink 2020, 4).

Still others have stressed, and I am of this camp of thinking, that distributed cognition is not something binary—something that either is or isn’t the case. Rather, cognition always takes places through entanglement of agent and world, and we should instead speak of the degree to which something is distributed (e.g., Sterelny 2010; Sutton 2010; Heersmink 2015, 2017). The degree of integration can vary across a number of dimensions, including: “the kind and intensity of information flow between agent and scaffold, the accessibility of the scaffold, the durability of the coupling between agent and scaffold, the amount of trust a user puts into the information the scaffold provides, the degree of transparency-in-use, the ease with which the information can be interpreted, the amount of personalization, and the amount of cognitive transformation” (Heersmink 2017, 20). The higher the integration across these dimensions, the more robustly distributed the cognitive or affective state across the relevant scaffold. So, Otto’s notebook which he uses all the time, regularly updates and relies upon, carries around with him and consults almost automatically, is highly personalised and allows him to remember what he otherwise could not, is taken to be illustrative of a highly distributed cognitive process, whereas relying upon, say, a metro-map in a city you are just visiting for a few days, might be thought of as a case of a less tightly integrated form of distributed cognition. As such, we should think about distributed cognition as something that happens on a spectrum.5

Generative AI’s ability, then, to stand in as an intersubjective partner is significant. For it opens up the possibility of engaging with technologies that present as being part of a shared world with us and get involved in the kind of cognitive co-construction that we described above that happens between human interlocuters. Now, to some extent, we might think that AI systems are inhabiting a shared reality with us, for example, by drawing on datasets that are grounded in human-produced information, there is a sense in which generative AI does indeed draw from a well of common information and reality. However, as the Chail’s case demonstrates, generative AI often takes our own interpretation of reality as the ground upon which conversation is built. If I log onto Claude and ask about how I might retrieve a huge inheritance that my mother is hiding in a vault in Switzerland, it takes this “difficult family situation” as true and offers me generated solutions on this basis (e.g., talking to my mother, researching the Swiss legal system, instructing a lawyer). Thus, generative AI plays an intersubjective co-constituting role but within the structure of a reality that I have myself already prescribed. It is this that makes generative AI a potentially fertile place in which distributed delusions might take root and bloom.