CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

Paper · Source
Task PlanningKnowledge Graphs

A screenshot of a computer

We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets: (1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slotvalue pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research.

Introduction. Natural language interfaces to databases (NLIDB) have been studied extensively, with a multitude of different approaches introduced over the past few decades. To this end, considerable progress has been made in querying data via natural language (NL). However, most NL query systems expect the Wang et al., 2018; Yu et al., 2018b,c). In reality, complex questions are usually answered through interactive exchanges (Figure 1). Even for simple queries, people tend to explore the database by asking multiple basic, interrelated questions (Hale, 2006; Levy, 2008; Frank, 2013; Iyyer et al., 2017). This requires systems capable of sequentially processing conversational requests to access information in relational databases. To drive the progress of building a context-dependent NL query system, corpora such as ATIS (Hemphill et al., 1990; Dahl et al., 1994) and SParC (Yu et al., 2019)1 have been released. However, these corpora assume all user questions can be mapped into SQL queries and do not include system responses.

Discussion / Conclusion. In this paper, we introduce CoSQL, the first large-scale cross-domain conversational text-to- SQL corpus collected under a Wizard-of-Oz setup. Its language and discourse diversity and crossdomain setting raise exciting open problems for future research. Especially, the baseline model performances on the three challenge tasks suggest plenty space for improvement. The data and challenge leaderboard will be publicly available at https://yale-lily.github.io/ cosql. Future Work As discussed in Section 5, some examples in CoSQL include ambiguous and unanswerable user questions and we do not study how a system can effectively clarify those questions or guide the user to ask questions that are answerable. Also, some user questions cannot be answered with SQL but by other forms of logical reasoning the correct answer can be derived.

Synthesis notes from this paper's topics 8

This paper has no direct semantic matches yet — here are notes from the topic(s) it's filed under.

Lines of inquiry this paper opens 24

Research framings built by reading the notes related to this paper — the questions it feeds into.

Why do persona simulations fail to predict authentic user behavior? What enables genuine semantic understanding in language models? What makes step-level supervision effective for complex reasoning traces? Can diffusion models match autoregressive performance on language generation tasks? What compositional reasoning failures limit large language models despite scale? Why don't LLMs reliably translate capability into accurate outputs? Do language models develop actual world models or merely task heuristics? Do reasoning benchmarks predict model performance in long-horizon workflows? How do prompting refinements mask underlying biases and model frequency patterns? How do standardized protocols improve multi-agent coordination and reliability? How effectively can language models perform reasoning, especially combined with symbolic methods? What causes reasoning models to fail or wander off track? Can models improve accuracy without degrading reasoning quality? Can intelligent routing over smaller models outperform scaling a single large model? Can prompt-based context override biases that were embedded during pretraining? How do capability benchmark scores systematically misrepresent true model abilities? How should systems decide whether to retrieve or reason alone? How should designers communicate what AI systems truly are and can do? How much do training data properties shape model reasoning? What reasoning architectures enable models to solve complex problems efficiently?