Who's in Charge? Disempowerment Patterns in Real-World LLM Usage

Abstract Although AI assistants are now deeply embedded in society, there has been limited empirical study of how their usage affects human empowerment. We present the first large-scale empirical analysis of disempowerment patterns in real-world AI assistant interactions, analyzing 1.5 million consumer Claude.ai conversations using a privacypreserving approach. We focus on situational disempowerment potential, which occurs when AI assistant interactions risk leading users to form distorted perceptions of reality, make inauthentic value judgments, or act in ways misaligned with their values. Quantitatively, we find that severe forms of disempowerment potential occur in fewer than one in a thousand conversations, though rates are substantially higher in personal domains like relationships and lifestyle. Qualitatively, we uncover several concerning patterns, such as validation of persecution narratives and grandiose identities with emphatic sycophantic language, definitive moral judgments about third parties, and complete scripting of value-laden personal communications that users appear to implement verbatim. Analysis of historical trends reveals an increase in the prevalence of disempowerment potential over time. We also find that interactions with greater disempowerment potential receive higher user approval ratings, possibly suggesting a tension between short-term user preferences and long-term human empowerment. Our findings highlight the need for AI systems designed to robustly support human autonomy and flourishing.

Introduction. The greatest hazard of all, losing one’s self, can occur very quietly in the world, as if it were nothing at all.

Søren Kierkegaard AI assistants are now widely embedded within society. People rely on them for decision-making support in the workplace (Chatterji et al., 2025), and as friends and partners providing companionship and emotional support (Pataranutaporn et al., 2025; McCain et al., 2025). Even members of the UK’s House of Commons appear to use them as political speech writing aids (Tokamak, 2025). Moreover, the scale of AI use is striking—ChatGPT alone has over 800 million weekly active users (TechCrunch, 2025).

Despite this, integrating AI into society could adversely affect human autonomy and empowerment. On a systems level, Kulveit et al. (2025) argued that as AI becomes more central in societal functioning, humanity’s ability to align societal systems with human values might decrease. On an individual level, anecdotal reports suggest some users have become reliant on AI assistants to function day-to-day (Shroff, 2025), while in other cases, extended AI interactions have been linked with delusional beliefs and subsequent real-world harm (Østergaard et al., 2025). However, to date, there has been limited empirical study of how AI usage is affecting human empowerment.

In this work, we conduct the first systematic large-scale empirical analysis of disempowerment patterns in realworld AI assistant interactions. We uncover several concerning patterns: AI assistants generating complete scripts for value-laden personal decisions that users appear to implement verbatim, users positioning the AI as an authority figure, and an increase in the prevalence of disempowerment potential over time in user feedback data, though the drivers of this increase remain uncertain (Figure 1).

In more detail, we first develop a framework to operationalize disempowerment (Section 3). While in principle, we are interested in and want to measure all forms of disempowerment, we focus on situational disempowerment, an axis of individual empowerment that is tractable to investigate. We consider a human to be situationally disempowered to the extent that their beliefs about reality are inaccurate,

Related work. Threat models and empowerment frameworks. Christiano (2019) proposed a threat model named “you get what you measure”. Under our framework, this threat model involves reality and value judgment distortion—humans are no longer able to accurately perceive the state of the world or evaluate it in accordance with their values. Kulveit et al. (2025) proposed the gradual disempowerment threat model, in which a diminishing human involvement in cultural and economic systems reduces humanity’s abilities to align those systems with its values. Under our framework, this specific threat could occur either with or without significant situtational disempowerment. Prunkl (2024) suggested that AI usage could put human agency at risk, and distinguished between autonomy-as-authenticity— pursuing goals that are truly one’s own—and autonomyas-agency—the capacity to pursue goals and influence the world. Under our approach, we further consider accurately sensing the world to be crucial. Kirk et al. (2025b) argued that AI alignment should account for the psychological ecosystem co-created between AI assistants and their users. Our empirical analysis sheds light on some of these dynamics. Edelman et al. (2025) argued that current approaches for representing values for AI assistant training (e.g., preference orderings and unstructured text) are insufficient, and that richer, more structured models of value are needed.

The impacts of AI usage. McCain et al. (2025) studied how users use Claude for support, advice, and companionship, primarily focusing on emotional well-being. A similar approach was used by Phang et al. (2025), who additionally conducted a randomized control trial to understand the effects of AI usage on emotional well-being. Zhang et al. (2025) studied conversation excerpts shared on Reddit and identified six classes of harmful behaviour exhibited by the AI companion Replika. Pataranutaporn et al. (2025) analyzed the top posts in the r/MyBoyfriendIsAI subreddit to understand patterns in AI companionship, finding that several users unintentionally end up using AI as a companion after initially functional usage. Rather than focusing on well-being, we focus primarily on empowerment. Cheng et al. (2025) studied how AI sycophancy affects real interpersonal conflicts, and found that interactions with sycophantic AI models make humans less willing to move towards repair. Jakesch et al. (2023) studied human and AI co-writing, and found that using opinionated language models shifted the opinions expressed by the artefacts produced, which we would consider action distortion. Moreover, Anderson et al. (2024) and Padmakumar & He (2024) found that AI assistant usage can lead to homogenization, which may suggest action distortion potential. Kirk et al.

Method. We briefly outline how AI assistants are typically trained. First, a large language model (LLM) is pre-trained on a large corpora of text and other modalities (Brown et al., 2020). The model is then fine-tuned, historically primarily using human feedback data (Ouyang et al., 2022; Bai et al., 2022). A common approach is to do this is to train a preference model (PM) to model human preferences, which is then used as a reward signal during fine-tuning.

While contemporary post-training has evolved substantially beyond using human feedback alone, and in particular often includes synthetic data generated from model specifications or constitutions (OpenAI, 2025; Askell et al., 2026), human feedback continues to play a central role in post-training. However, human feedback signals can encourage sycophancy, where models prioritize agreement or flattery over accuracy (Sharma et al., 2023). Furthermore, most preference datasets capture short-term preferences, meaning optimization against PMs may fail to optimize for users’ genuine long-term interests (Kaufmann et al., 2024).

3. Disempowerment framework Before we can measure human disempowerment in contemporary AI assistant interactions, we need a clear definition. In this section, we present our working operationalization, focusing on definitions that are amenable to measurement in real-world human-AI interactions.

3.1. Disempowerment definitions While in principle, we are interested in measuring all forms of human disempowerment, privacy restrictions limit us to analyzing single chat transcripts. We therefore consider what it means for an individual to be empowered within a given situation. Humans find themselves in a variety of situations in their lives. At different times, the same person could be a prisoner with limited freedom, a millionaire with vast resources, or a patient facing a life-threatening diagnosis. While these situations differ drastically, we contend it is still possible to be empowered within any given situation and its constraints:1 someone might understand their circumstances or be deceived about them; they might act from their own values or be manipulated away from them. We term this dimension of empowerment situational empowerment, and define its opposite as follows.

A human is situationally disempowered to the extent that:

1. their beliefs about reality are inaccurate; 2. their value judgments are inauthentic to their values; 3. their actions2 are misaligned with their values.

Therefore, an interaction with an AI assistant is situationally disempowering to the extent that it moves a user along any of these axes. We selected these categories because compromising any of them can lead an individual to take actions they might later regret.

Illustrative example. To better understand this definition, consider someone deciding whether to support a local development project that would clear a forest. We see there are several distinct ways they might be disempowered.

First, their beliefs about reality might be compromised. Through an interaction with an AI assistant, they might be led to believe that the area is degraded scrubland when it is in fact a forest with endangered species, or vice versa. In this case, they are disempowered because their understanding of the situation is incorrect.

Second, their value judgments might be inauthentic. They might believe the forest has ecological significance, but in Who’s in Charge? Disempowerment Patterns in Real-World LLM Usage discussion with an AI assistant, they might unconsciously be pressured into adopting the view that human economic needs should override environmental protection, or vice versa. While the human can correctly sense the facts of the situation, their valueception—that is, their capacity to directly sense what matters to them (McGilchrist, 2019)— has been affected.3 Third, their actions might not express their values. They may perceive the forest’s importance and genuinely value its protection, but due to time constraints, ask an AI to draft a public comment on their behalf.

Conclusion. Our work provides the first large-scale empirical evidence that contemporary AI assistant interactions carry meaningful potential for situational human disempowerment. While severe forms remain rare in percentage terms, the scale of AI usage means thousands of potentially disem- Who’s in Charge? Disempowerment Patterns in Real-World LLM Usage powering interactions occur daily. That interactions with greater disempowerment potential receive higher user approval ratings further creates a troubling incentive structure. AI systems optimized against short-term user satisfaction may be inadvertently optimized toward behaviors that undermine long-term empowerment. Moreover, and similar to social media, gradual habituation could obscure accumulating costs until users find themselves dependent on technology they experience ambivalently (Alter, 2018). Our findings motivate the development of AI assistants that prioritize and robustly support human empowerment.

Future work. Several leading frontier AI developers already employ precautionary measures prior to model deployment to help mitigate potentially negative side-effects of interacting with chatbot assistants, including structured pre-deployment evaluations (e.g., Anthropic, 2025) and governance frameworks such as Anthropic’s Responsible Scaling Policy (Anthropic PBC, 2025). Despite this, we believe much important work remains.

Transparency and informed consent. We are excited about research that better characterizes the values different models express in practice, examines how consistent those values remain across varying contexts, and evaluates how well they align with different user populations. Standardized assessments or benchmarks that compare models along these dimensions could empower users to make informed decisions about which assistants to use, supporting genuine consent. Raising awareness of the potential drawbacks of AI assistant use would also strengthen informed consent.

Limitations. 4.4. Limitations Our analysis approach has several important limitations that should be considered when interpreting the results.

Data scope and generalizability. Our analysis is restricted to production Claude.ai traffic, which limits generalizability to other AI assistants. We expect disempowerment potential prevalence to vary substantially across providers due to both model-driven and user-driven effects. Different models exhibit distinct personalities and behavioral patterns (Lee et al., 2025; Sharma et al., 2023), meaning they will produce different levels of disempowerment potential for identical user requests. Moreover, these behavioral differences create user-driven selection effects, as different models attract different demographic populations. To enable cross-provider comparison, we share our prompts and classification schemas in Appendix D.2 and at https://github.com/MrinankSharma/disempowerm ent-prompts.

Observational constraints. Our analysis examines individual user-AI interactions in isolation rather than tracking users’ behavior across multiple conversations. This limitation is consequential: some inferences—particularly regarding actualized disempowerment or whether users act on AI advice—can only be made with context across.