Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions

Nass and Moon (2000) conclude that people have similar expectations from talking to bots and humans. This similarity is a possible explanation for why sometimes user requests might be ambiguous and incomplete, as shown in Fig. 1 (b) and (c). This ambiguity is especially challenging to handle in a dialogue setting, where a system is limited by returning only one answer in response to each request, unlike in web search setup where diversification of results is possible and acceptable (Vallet and Castells, 2012). Previous research has shown that users are much more forgiving about system mistakes if they can act on them with minimal efforts spent (Kocielnik et al., 2019; Kiseleva et al., 2016b). Therefore it is more appropriate to ask a clarifying question in user request ambiguity rather than generating incorrect answers.