Answering Questions by Meta-Reasoning over Multiple Chains of Thought

Modern systems for multi-hop question answering (QA) typically break questions into a sequence of reasoning steps, termed chain-of thought (CoT), before arriving at a final answer. Often, multiple chains are sampled and aggregated through a voting mechanism over the final answers, but the intermediate steps themselves are discarded. While such approaches improve performance, they do not consider the relations between intermediate steps across chains and do not provide a unified explanation for the predicted answer. We introduce Multi-Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought, rather than aggregate their answers. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in generating an explanation and predicting the answer.

focusing exclusively on the final output discards relevant information that is present in the intermediate reasoning steps. Consider answering the question “Did Brad Peyton need to know about seismology?” (Fig. 1). Reasoning chain #1 leads to an incorrect answer (“No”), but its steps provide useful information. For example, the intermediate question, and following answer, on “What is seismology?” constitute an important fact that is absent from the other two chains. Last, using SC jointly with chain-of-thought prompting reduces interpretability, as there is no single reasoning chain that can be considered as an explanation