A Non-Factoid Question-Answering Taxonomy

INSTRUCTION

REASON

EVIDENCE-BASED

COMPARISON

EXPERIENCE

DEBATE

INSTRUCTION

You want to understand the procedure/method of doing/achieving something.

Instructions/guidelines provided in a step-by-step manner.

REASON

You want to find out reasons of/for something.

A list of reasons with evidence.

EVIDENCE-BASED

You want to learn about the features/description/definition of a concept/idea/object/event.

Wikipedia-like passage describing/defining an event/object or its properties based only on facts.

COMPARISON

You want to compare/contrast two or more things, understand their differences/similarities.

A list of key differences and/or similarities of something compared to another thing

EXPERIENCE

You want to get advice or recommendations on a particular topic.

Advantages, disadvantages, and main features of an entity (product, event, person, etc) summarised from personal experiences.

DEBATE

You want to debate on a hypothetical question (is someone right or wrong, is some event perceived positively or negatively?).

Arguments on a debatable topic consisting of different opinions on something supported or weakened by pros and cons of the topic in the question.

To do so, we must first define possible question categories. Unfortunately, there is no unified and well-evaluated taxonomy for NFQs, unlike factoid QA where a few taxonomies of question categories and forms of target answers exist [22, 28, 41]. While some related works (described in Section 2) involve taxonomies of NFQ categories, the information on particular details of those taxonomies is rather scattered. In our preliminary user study, described in Section 3.2, we tried to adopt an existing theoretical taxonomy [42] for complex questions, but the agreement on question categories between study coordinators was extremely poor, and did not improve even after a few rounds of discussion. For example, there is only a nuanced difference between the Causal Antecedent and Causal Consequent categories of that taxonomy. Thus, there was a need to gather information on existing NFQ taxonomies from all available sources and to create a taxonomy that is built with a transparent methodology and is thoroughly evaluated. In this paper, we aim to accelerate the research of non-factoid QA by studying which categories of NFQs exist, what their distribution is in existing datasets, and what potential forms of answers they require. Our contributions can be summarized as follows:

• We propose a new taxonomy of NFQ categories and their respective target answer structures. We revised the initial taxonomy version via a controlled editorial user study. The study also revealed which categories are the most difficult to answer from a human perspective, and how system- and human-generated answers for different categories compare. We extensively evaluated the taxonomy via crowdsourcing studies, including a comparison of how people group questions naturally, when no taxonomy is provided. (Section 3)