Diplomat: A Dialogue Dataset for Situated PragMATic Reasoning

“We introduce a new benchmark, Diplomat, aiming at a unified paradigm for pragmatic reasoning and situated conversational understanding. Compared with previous works that treat different figurative expressions (e.g., metaphor, sarcasm) as individual tasks, Diplomat provides a unified understanding towards general pragmatic understanding. Our dataset is created using Amazon Mechanical Turk (AMT), resulting in 4, 177 multi-turn dialogues. In company with the dataset, we propose two tasks: Pragmatic Identification and Reasoning and Conversational Question Answering. Experimental results with state-of-the-art (SOTA) neural architectures demonstrate that: 1) large language models (LLMs) show poor performances in this subjective topic. 2) Context understanding is a crucial factor in building benign human-machine interaction. 3) Current models defect in the application of pragmatic reasoning.”

What are the core components of real-life conversational pragmatic reasoning? Motivated by theories of cognitive linguistics and conversational modeling, we anticipate it to be three-fold:

• Situational Context Reasoning. Understanding pragmatic meaning requires a detailed understanding of conversational contexts. Consider the utterance “You are making the rest of us looking bad”, under different situations of praise and sarcasm, the sentence may convey completely opposite meanings. Furthermore, typical conversational reasoning challenges such as coreference resolution and intention prediction are largely dependent on the success of situated context modeling.

• Open-world Knowledge Acquisition. The open-world knowledge includes commonsense knowledge (e.g., social ethics) that can be learned from different domains of dialogue corpus and domain-specific knowledge (e.g., American histories). Successful pragmatic reasoning requires the acquisition of open-world knowledge and joint reasoning over the conversation.

• Unified figurative language understanding. Figurative language is one of the most frequently used tricks for conveying implicit meanings with subjective emotions. Previous works treat different forms of figurative language understanding as individual tasks, such as metaphors [15], idioms [16, 17], pun [18], etc. Pragmatic reasoning provides a feasible unified perspective that considers all these tasks as recovering their literal meanings.

we introduce Diplomat, a real-life conversational dataset that focuses on pragmatic reasoning. Diplomat stems from an interview dataset [19], and experiences three steps of curation: automatic selection, fine-grained manual annotation and human refinement (Sec. 3). Our dataset owns 4, 177 dialogues and covers a vocabulary of 48, 900 words. More than that, human-annotated answers reach the amount of 6, 494 and hold a vocabulary size of 20, 000. Along with the dataset, we propose two tasks, Pragmatic Identification and Reasoning (PIR) and Conversational Question Answering (CQA), to benchmark machines’ pragmatic reasoning capabilities (Sec. 4).