Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration

Conversational systems based on Large Language Models (LLMs), such as ChatGPT, show exceptional proficiency in context understanding and response generation. However, they still possess limitations, such as failing to ask clarifying questions to ambiguous queries or refuse users’ unreasonable requests, both of which are considered as key aspects of a conversational agent’s proactivity. This raises the question of whether LLM-based conversational systems are equipped to handle proactive dialogue problems. In this work, we conduct a comprehensive analysis of LLM-based conversational systems, specifically focusing on three key aspects of proactive dialogues: clarification, target-guided, and non-collaborative dialogues. To trigger the proactivity of LLMs, we propose the Proactive Chain-of-Thought prompting scheme, which augments LLMs with the goal planning capability over descriptive reasoning chains. Empirical findings are discussed to promote future studies on LLMbased proactive dialogue systems.

ChatGPT can still achieve competitive performance under zero-shot setting on different dialogue problems, such as the knowledge-grounded dialogues (Bang et al., 2023), task-oriented dialogues (Zhang et al., 2023), and emotion-aware dialogues (Zhao et al., 2023).

Despite the strength of ChatGPT, there are still several limitations1, such as failing to ask clarification questions to ambiguous user queries or refuse problematic user requests. These kinds of capabilities are typically regarded as the proactivity of the conversational system (Deng et al., 2023b), where the system can create or control the conversation to achieve the conversational goals by taking initiative and anticipating impacts on themselves or the human users. Thus, it raises the question: Are these LLM-based conversational systems equipped to manage proactive dialogue problems?

In this work, we conduct the first comprehensive analysis of LLM-based conversational systems on three common aspects of proactive dialogues, including 1) clarification in information-seeking dialogues (Guo et al., 2021; Deng et al., 2022a) where the system is required to proactively ask clarification questions when encountering ambiguity in user queries; 2) target-guided open-domain dialogues (Tang et al., 2019; Wu et al., 2019) where the system is required to proactively lead the conversation towards the designated target; and 3) non-collaborative task-oriented dialogues (Li et al., 2020; Zhou et al., 2020; Deng et al., 2023a) where the system and the user do not share the same conversational goal while the system aims to strategically reach a consensus with the user.

with standard prompting, LLM-based systems directly provide a randomly-guessed answer to the ambiguous user question (1a), or generate a general bargain response without any negotiation strategy (2a). When providing the system with options to take different dialogue acts (proactive prompting), the generated responses are unaware of the conversational goal, such as generating underspecified clarification questions (1b) and conservative negotiation responses (2b). To this end, Pro- CoT first instructs the system to generate descriptive thoughts about intermediate steps of reasoning and planning for reaching the conversational goal, and then make the decision of the next action to take. Finally, the system generates an appropriate response based on the decided action (1c & 2c).