StructGPT: A General Framework for Large Language Model to Reason over Structured Data

In this paper, we aim to improve the reasoning ability of large language models (LLMs) over structured data in a unified way. Inspired by the studies on tool augmentation for LLMs, we develop an Iterative Reading-then-Reasoning (IRR) framework to solve question answering tasks based on structured data, called StructGPT. In this framework, we construct the specialized interfaces to collect relevant evidence from structured data (i.e., reading), and let LLMs concentrate on the reasoning task based on the collected information (i.e., reasoning). Specially, we propose an invoking linearization-generation procedure to support LLMs in reasoning on the structured data with the help of the interfaces. By iterating this procedure with provided interfaces, our approach can gradually approach the target answers to a given query. Experiments conducted on three types of structured data show that StructGPT greatly improves the performance of LLMs, under the few-shot and zero-shot settings.

we propose an Iterative Reading-then-Reasoning (IRR) framework for LLMs to utilize the interfaces to solve the tasks based on structured data, namely StructGPT. This framework considers two major functions to fulfill different tasks, namely collecting relevant evidence (reading) and inferring the answer or planning subsequent steps (reasoning).

LLMs for Structured Data. Benefitting from the strong few-shot and zero-shot capability, recent studies have leveraged LLMs to perform reasoning over structured data (Chen et al., 2023; Li et al., 2023a; Cheng et al., 2022; Rajkumar et al., 2022). Existing work can be roughly divided into two types. The first type of method linearizes the structured data into a sentence (e.g., table rows), and feeds it into the LLMs to generate the answer according to in-context exemplars (Cheng et al., 2022; Chen, 2023). For complex questions or structured data, they first decompose it into multiple simple and short ones and then perform linearization and generation (Ye et al., 2023). Another type of method leverages LLMs to evaluate the plausibility of the solution plan based on the knowledge base (Gu et al., 2023), or first generate a solution draft with in-context exemplars and then revise the draft grounding on the knowledge base (Li et al., 2023c). However, most of them only focus on a specific type of structured data, and are lack of generality across various data and tasks. In StructGPT, we provide a unified paradigm that is general to various structured data and downstream tasks.

In this work, we mainly focus on three types of structured data, namely knowledge graphs (KG), data tables (Table), and databases (DB), since they play an important role as the knowledge source in helping solve complex reasoning tasks, described as follows.

• Knowledge Graph. A knowledge graph (KG) consists of a number of triples to store the factual knowledge, denoted as G = {⟨e, r, e′⟩|e, e′ ∈ E, r ∈ R}, where E and R denote the set of entities and relations, respectively. A triple ⟨e, r, e′⟩ represents the fact that there is a relation r between the head entity e and the tail entity e′.

• Data Table. A data table T (table in short) contains multiple columns {ci}Ci =1 and rows {lj}Rj =1, where each row lj denotes a data record formatted by the attributes indexed by columns {ci}Ci =1, and vi,j denotes the content in the cell corresponding to the position at column i and row j.

• Database. A database (DB) typically consists of N data tables, denoted as D = {T1, T2, ..., TN}. Besides the column names, the foreign keys across all tables are also available to link the data from two tables, denoted as {(c(k) i , c(h) j )}, where c(k) i and c(h) j denote the i-th and j-th columns in the k-th and h-th tables, respectively.