MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation

“We propose MemoChat, a pipeline for refining instructions that enables large language models (LLMs) to effectively employ self-composed memos for maintaining consistent long-range open-domain conversations. We demonstrate a long-range open-domain conversation through iterative “memorization-retrieval-response” cycles. This requires us to carefully design tailored tuning instructions for each distinct stage. The instructions are reconstructed from a collection of public datasets to teach the LLMs to memorize and retrieve past dialogues with structured memos, leading to enhanced consistency when participating in future conversations. We invite experts to manually annotate a test set designed to evaluate the consistency of long-range conversations questions. Experiments on three testing scenarios involving both open-source and API-accessible chatbots at scale verify the efficacy of MemoChat, which outperforms strong baselines.”

Instead of replying on external tools for memory creation and recall (Lee et al., 2023; Zhong et al., 2023; Hu et al., 2023; Kynoch and Latapie, 2023), we propose a streamlined approach where we eliminate the need for these complex associate modules. Instead, we create a simplified pipeline that solely utilizes LLMs to power chatbots. The proposed memo-equipped pipeline, MemoChat, aims for guiding LLMs to use simple on-the-fly memos for maintaining consistent long-range open-domain conversation. We base our approach on the concept of “memorization-retrieval-response” loops within the context of long-range open-domain conversations.