Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents

“Large Language Models (LLMs) are currently capable of generating human-like responses in open-domain tasks [4]. This has led to a new generation of conversational agents, such as chatGPT, which are now being widely used across domains. To ensure that agents generate responses that are contextually relevant and coherent to an ongoing conversation, these agents must maintain a working memory of the conversational history that has occurred up to that point in the conversation. The default strategy is to use as much of the conversational history as will fit within the input size limit of the LLM. Parts of the conversations that go beyond that buffer limit are forgotten, which leads to breakdowns when users assume the model remembers past context. Additionally, as the input buffer size increases, the performance of the LLM degrades as it struggles to retrieve relevant context and can be distracted by irrelevant context [11, 18]. This problem is compounded because users do not know how the LLM is leveraging the memory to generate responses.

Multiple strategies have been introduced to manage agents’ conversational memory. For example, the conversation can be automatically summarized [21] and refined [24] to reduce redundancy while maintaining key information. Some systems selectively store [12, 22] and update [1] key memories. Relevant memories can also be retrieved based on the user input [1, 15, 21]. However, these memory management strategies are hidden behind the interface, resulting in a lack of transparency. Users often do not know what strategy is being used and have limited control over it. This makes it difficult for users to repair conversational breakdowns that happen when there is a misalignment between how the agent manages the memory and how the user perceives the conversation.

We present Memory sandbox, shown in Figure 1, a system that allows users to see and manage the memory of conversational agents to align with user understanding of the conversation. Memory Sandbox transforms conversational memory, previously managed behind the user interface, into interactive memory objects within the interface. Users can manipulate the visibility and content of memory objects, spatially rearrange them, and share them across conversations. We make the following contributions: 1) The conceptualization of memory objects which makes conversational memory transparent and interactive and 2) The Memory Sandbox system that offers novel interaction affordances for users to view and manipulate the conversational memory of an intelligent agent.”