Lexical Entrainment for Conversational Systems

lexical entrainment (LE), a phenomenon in which speakers in human-human conversations tend to naturally and subconsciously align their lexical choices with those of their interlocutors, leading to more successful and engaging conversations.

This highlights the importance of LE in establishing a shared terminology for maximum clarity and reducing ambiguity in conversations. However, we demonstrate in this work that current response generation models do not adequately address this crucial humanlike phenomenon.

When people engage in a conversation, they naturally adapt their way of speaking to align with their conversational partner. For instance, they tend to refer to something based on how their conversational partners refer to it, using the same terms when discussing the same object repeatedly and negotiating a common description, particularly for items that may be unfamiliar to them (Brennan, 1996a). This linguistic phenomenon is known as lexical entrainment (LE) (Garrod and Anderson, 1987; Brennan, 1996b).

LE plays a key role in the success and naturalness of interactions in conversations.

LE is associated with a broad range of positive social behaviours and outcomes

In summary, the main contributions of this paper are as follows:

• We formalize a precise and practical definition of LE (§2.1), as well as a new LE measure to evaluate the natural degree of LE in human-to-human conversations (§2.2);

• We highlight the importance of LE and provide an analysis of state-of-the-art response generation models, pointing out issues caused by the neglect of LE (§3);

• We propose a LE dataset, MULTIWOZ-ENTR, specifically designed for studying LE, and provide detailed annotations (§4) and statistical analysis (appendix B);

• We present a methodology for integrating LE into conversational systems through two novel sub-modules (tasks). Specifically, we provide two baseline models for the LE extraction submodule (§5), providing valuable insights into the incorporation of LE into conversational systems. This approach lays the groundwork for the development of a LE generator submodule, which is left for future research (§6).