Break the Chain: Large Language Models Can be Shortcut Reasoners

Despite its benefits, CoT is also critiqued for its substantial token usage, as it explores numerous reasoning pathways before arriving at a conclusive answer. This characteristic is particularly prominent in variants such as Tree-of-Thought (ToT) (Yao et al., 2023), which scrutinize every possible reasoning chain. Traditionally, CoT has been predominantly applied to mathematical reasoning, with scant application to commonsense, or complex logical reasoning tasks. This limited focus may hinder a comprehensive understanding of CoT’s potential to emulate intricate human-like reasoning processes.

Human reasoning uses heuristics to find local rational maximum (Karlan, 2021; Neth and Gigerenzer, 2015; Lancia et al., 2023), which often relies on cognitive shortcuts (Fernbach and Rehder, 2013; Ferrario, 2004), a characteristic that can be mirrored and exploited in LMs.

As depicted in Figure 1, when prompted with shortcut reasoning, the ChatGPT model swiftly arrives at answers with minimal token consumption. The ability of LLMs to employ shortcut reasoning not only mirrors human cognitive strategies but also has the potential to streamline problem-solving processes, thereby reshaping computational efficiency and model performance.

The ShortcutQA dataset is designed to evaluate Language Models’ (LMs) ability to employ heuristic shortcuts in reasoning, addressing a gap in existing resources that primarily focus on sequential reasoning approaches. Comprising 449 diverse reasoning problems, ShortcutQA spans logical puzzles to real-world problem-solving scenarios. Each problem is presented with a shortcut-based solution alongside a detailed step-by-step solution, categorized into three reasoning types.