Expedient Assistance and Consequential Misunderstanding: Envisioning an Operationalized Mutual Theory of Mind

Design fictions allow us to prototype the future. They enable us to interrogate emerging or non-existent technologies and examine their implications. We present three design fictions that probe the potential consequences of operationalizing a mutual theory of mind (MToM) between human users and one (or more) AI agents. We use these fictions to explore many aspects of MToM, including how models of the other party are shaped through interaction, how discrepancies between these models lead to breakdowns, and how models of a human’s knowledge and skills enable AI agents to act in their stead. We examine these aspects through two lenses: a utopian lens in which MToM enhances human-human interactions and leads to synergistic human-AI collaborations, and a dystopian lens in which a faulty or misaligned MToM leads to problematic outcomes.

Design fictions are used within HCI to explore the potential impact of future technologies [4, 13]. This method enables us to explore how new (or speculated) technologies impact work practices [14], raise ethical dilemmas [8], define research agendas [2], and identify the “potential societal value and consequences of new HCI concepts” [11], well in advance of their development and use.

First, the AI agent is equipped with a memory and the capability to learn about the user, both through interaction (“chat space”) and through examination of the user’s work products (“artifact space”). The AI agent is also capable of predicting the user’s behavior – in essence, simulating the user [15] via its theory of mind – and performing actions proactively, on the user’s behalf, based on those predictions [7].

For the user, we observe the ability to interrogate the AI agent’s theory of mind model to understand “what does it know about me?” Building on these core ideas, this story explores the beneficial outcomes that MToM may bring to workers within an organization and how MToM might shape the future of work: helping us identify and focus on the tasks we truly enjoy (Part I), providing a buffer from coworkers to improve their ability to achieve and maintain flow [5] (Part II), proactively filling in knowledge gaps (Part III), improving social connectedness (Part IV), and helping people focus on higher-level work goals (Part V).

• For the user, we observe the ability to interrogate the AI agent’s theory of mind model to understand “what does it know about me?” Building on these core ideas, this story explores the beneficial outcomes that MToM may bring to workers within an organization and how MToM might shape the future of work: helping us identify and focus on the tasks we truly enjoy (Part I), providing a buffer from coworkers to improve their ability to achieve and maintain flow [5] (Part II), proactively filling in knowledge gaps (Part III), improving social connectedness (Part IV), and helping people focus on higher-level work goals (Part V).

• While a theory of mind model includes information about the other party’s knowledge, skills, and capabilities, a mutual theory of mind between a human and an AI agent also includes: (1) a human’s understanding of what the AI agent knows about them; (2) an AI’s representation of the human’s mental model of the AI; and (3) the ability for each party to update their models through interaction with the other party (ALways Happy to Help; Referral Roulette; Aim High, Stuart).

• MToM can enhance productivity by fostering synergistic outcomes of human-AI teams and increasing feelings of self-efficacy, creativity, and social connectedness (ALways Happy to Help).

• An AI’s model of a user’s knowledge and skills can serve as an external memory and proactively fill in knowledge gaps (ALways Happy to Help).

• When AI possesses a predictive model of a user’s behavior, it can take action on the user’s behalf, such as by writing code (AimHigh, Stuart), responding to messages (ALways Happy to Help), or executing business workflows (Referral Roulette).

• Wider adoption of MToM-infused AI agents (e.g.within an organization)may reshape work practices by streamlining communications and delivering the right information to the right people at the right time (ALways Happy to Help).

• User models may be built purely through observations of a user’s behavior, within both “chat space” and “artifact space” (ALways Happy to Help; Referral Roulette; Aim High, Stuart).

• Discrepancies between a human’s mental model of the AI (or AIs) and the AI’s model of the human may lead to conversational breakdowns [1, 3, 10] and have material consequences (Referral Roulette; Aim High, Stuart).

• Users may need signifiers of the presence of an AI’s user model and when it learns new information about the user (ALways Happy to Help; Aim High, Stuart). They may also need the ability to query the contents of the AI’s user model and make corrections (Referral Roulette).

• When an MToM-infused AI agent acts on the user’s behalf, users may need signifiers to know that they are interacting with an AI, not a human (ALways Happy to Help; Aim High, Stuart).

• Problems may arise when a human’s mental model of an AI’s capabilities doesn’t align with the AI’s actual capabilities (Referral Roulette). People may misapply the AI to domains or situations for which it wasn’t designed (Aim High, Stuart).

• Explanations will be crucial for helping people calibrate their trust in MToM-infused AI systems. Users will need to know both what the system did (in their stead) as well as why (ALways Happy to Help; Referral Roulette).