SParC: Cross-Domain Semantic Parsing in Context

Paper · Source
LLM ArchitectureDomain Specialization in LLMs

We present SParC, a dataset for cross-domain Semantic Parsing in Context. It consists of 4,298 coherent question sequences (12k+ individual questions annotated with SQL queries), obtained from controlled user interactions with 200 complex databases over 138 domains. We provide an in-depth analysis of SParC and show that it introduces new challenges compared to existing datasets. SParC (1) demonstrates complex contextual dependencies, (2) has greater semantic diversity, and (3) requires generalization to new domains due to its cross-domain nature and the unseen databases at test time. We experiment with two state-of-the-art text-to-SQL models adapted to the context-dependent, crossdomain setup. The best model obtains an exact set match accuracy of 20.2% over all questions and less than 10% over all interaction sequences, indicating that the crossdomain setting and the contextual phenomena of the dataset present significant challenges for future research. The dataset, baselines, and leaderboard are released at https:// yale-lily.github.io/sparc.

Introduction. Querying a relational database is often challenging and a natural language interface has long been regarded by many as the most powerful database interface (Popescu et al., 2003; Bertomeu et al., 2006; Li and Jagadish, 2014). The problem of mapping a natural language utterance into executable SQL queries (text-to-SQL) has attracted increasing attention from the semantic parsing community by virtue of a continuous effort of dataset creation (Zelle and Mooney, 1996; Iyyer et al., 2017; Zhong et al., 2017; Finegan-Dollak et al., 2018; Yu et al., 2018a) and the modeling innovation that follows it (Xu et al., 2017; Wang et al., 2018; Yu et al., 2018b; Shi et al., 2018). While most of these work focus on precisely mapping stand-alone utterances to SQL queries, generating SQL queries in a context-dependent scenario (Miller et al., 1996; Zettlemoyer and Collins, 2009; Suhr et al., 2018) has been studied less often.

Discussion / Conclusion. In this paper, we introduced SParC, a largescale dataset of context-dependent questions over a number of databases in different domains annotated with the corresponding SQL representation. The dataset features wide semantic coverage and a diverse set of contextual dependencies between questions. It also introduces unique challenge in mapping context-dependent questions to SQL queries in unseen domains. We experimented with two competitive context-dependent semantic parsing approaches on SParC. The model accuracy is far from satisfactory and stratifying the performance by question position shows that both models degenerate in later turns of interaction, suggesting the importance of better context modeling. The dataset, baseline implementations and leaderboard are publicly available at https:// yale-lily.github.io/sparc.

Synthesis notes from this paper's topics 8

This paper has no direct semantic matches yet — here are notes from the topic(s) it's filed under.

Lines of inquiry this paper opens 24

Research framings built by reading the notes related to this paper — the questions it feeds into.

How do interpretive frames override surface features in text comprehension? How does scaling reasoning capabilities affect models' appropriate abstention behavior? Do language models encode knowledge that influences generation, or primarily imitate surface patterns? Can latent reasoning match or exceed explicit reasoning performance? What prediction granularity best trains models to generate reliable reasoning? How do neural networks learn compositional structure from training? Can inference-time computation adaptively substitute for static model capacity? Why do training associations persist despite contradictory contextual information? How do sequence length and task type interact with sparsity tolerance? What limits language model accuracy in evaluating ideas? How does model capacity affect learning performance on diverse downstream tasks? What prevents language models from performing systematic logical reasoning? Do accumulated memories help or hurt continual learning in models? Why do language models struggle to implement user intent accurately from prompts? Can language models reason beyond surface pattern matching? What prevents LLMs from applying their reasoning knowledge to improve outputs? Can AI systems evade safety evaluations through reasoning manipulation?