Paper deep dive
Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go?
Daniel Gomm, Maarten de Rijke, Madelon Hulsebos
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/13/2026, 5:37:35 AM
Summary
This paper introduces Open Tabular Insight Extraction (OpenTI), a holistic framework for democratizing access to knowledge in large corpora of tables. It consolidates fragmented research from database, NLP, IR, and HCI communities by formalizing OpenTI from first principles, focusing on the user's insight need, the procedure for deriving analytical knowledge, and the sufficiency of the result. The authors conduct a systematic review of systems and benchmarks, finding that current efforts are fragmented, often lack end-to-end coverage, and use benchmarks that are unfit for open settings due to presupposed knowledge and mismatched validation. The paper concludes with a research agenda for trustworthy, cooperative, and interactive OpenTI systems.
Entities (9)
Relation Signals (8)
Daniel Gomm → affiliatedwith → Centrum Wiskunde & Informatica
confidence 95% · Daniel Gomm... Centrum Wiskunde & Informatica
Maarten de Rijke → affiliatedwith → University of Amsterdam
confidence 95% · Maarten de Rijk... University of Amsterdam
Open Tabular Insight Extraction → consolidates → Data Analysis Agents
confidence 90% · To bring these communities onto common ground, we establish a holistic framework... which we refer to as Open Tabular Insight Extraction (OpenTI)... consolidate frameworks and terminology across information retrieval, natural language processing...
Open Tabular Insight Extraction → consolidates → Table Question Answering
confidence 90% · To bring these communities onto common ground, we establish a holistic framework... which we refer to as Open Tabular Insight Extraction (OpenTI)... consolidate frameworks and terminology across information retrieval, natural language processing...
Open Tabular Insight Extraction → consolidates → Text-to-SQL
confidence 90% · To bring these communities onto common ground, we establish a holistic framework... which we refer to as Open Tabular Insight Extraction (OpenTI)... consolidate frameworks and terminology across information retrieval, natural language processing...
Current Benchmarks → isunfitfor → Open Settings
confidence 90% · benchmarks are largely unfit for evaluations in an open setting as inputs presuppose knowledge of tables, and validation mechanisms do not match the setup.
Current Systems → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Democratizing access to the knowledge held in large corpora of tables such as data lakes is emerging as a central research challenge. Research in this space is advancing and broadening in scope, increasingly supplying the components to satisfy a person's insight need end-to-end. Yet these efforts remain fragmented across communities that frame the problem under their own conventions, such as table question answering, text-to-SQL, and data analysis agents, with works six times as likely to cite within the same task label as across labels. To bring these communities onto common ground, we establish a holistic framework for this pursuit, which we refer to as Open Tabular Insight Extraction (OpenTI). We formalize OpenTI from first principles around the analytical knowledge a person needs, the procedure for deriving it from a corpus of tables, and how well a result serves the person who sought it. In doing so we consolidate frameworks and terminology across information retrieval, natural language processing, machine learning, databases, and human-computer interaction, and apply this grounding in a systematic review and analysis of systems and benchmarks that work towards OpenTI. We find that current systems do not cover the end-to-end scope of OpenTI, mainly focusing on the analysis itself, and that benchmarks are largely unfit for evaluations in an open setting as inputs presuppose knowledge of tables, and validation mechanisms do not match the setup. Finally, we distill a research agenda towards OpenTI systems, evaluation, and interaction paradigms that surface the insights users need. An interactive companion to our paper is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.07629v1
- Canonical: https://arxiv.org/abs/2609.07629v1
Trouble viewing inline? Open PDF directly →
Full Text
388,012 characters extracted from source content.
Expand or collapse full text
Open Tabular Insight Extraction: Where Do We Stand, and Where Should We Go? Daniel Gommdaniel.gomm@cwi.nl Centrum Wiskunde & Informatica University of Amsterdam Maarten de Rijkem.derijke@uva.nl University of Amsterdam Madelon Hulsebosmadelon.hulsebos@cwi.nl Centrum Wiskunde & Informatica Abstract Democratizing access to the knowledge held in large corpora of tables such as data lakes is emerging as a central research challenge. Research in this space is advancing and broadening in scope, increasingly supplying the components to satisfy a person’s insight need end-to-end. Yet these efforts remain fragmented across communities that frame the problem under their own conventions, such as table question answering, text-to-SQL, and data analysis agents, with works six times as likely to cite within the same task label as across labels. To bring these communities onto common ground, we establish a holistic framework for this pursuit, which we refer to as Open Tabular Insight Extraction (OpenTI). We formalize OpenTI from first principles around the analytical knowledge a person needs, the procedure for deriving it from a corpus of tables, and how well a result serves the person who sought it. In doing so we consolidate frameworks and terminology across information retrieval, natural language processing, machine learning, databases, and human-computer interaction, and apply this grounding in a systematic review and analysis of systems and benchmarks that work towards OpenTI. We find that current systems do not cover the end-to-end scope of OpenTI, mainly focusing on the analysis itself, and that benchmarks are largely unfit for evaluations in an open setting as inputs presuppose knowledge of tables, and validation mechanisms do not match the setup. Finally, we distill a research agenda towards OpenTI systems, evaluation, and interaction paradigms that surface the insights users need. An interactive companion to our paper is available at https://open-tabular-insight-extraction.github.io. 1 arXiv:2609.07629v1 [cs.IR] 7 Sep 2026 Contents 1 Introduction3 2 Open Tabular Insight Extraction5 2.1 Conceptual Foundation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Sufficiency Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.3 Taxonomy of Insight Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3 Anatomy of OpenTI Systems15 3.1 OpenTI Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.2 Orchestration Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.3 Functional Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4 Interaction and Interpretability in OpenTI31 4.1 Interaction Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.2 Interaction Means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3 Interpretability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 5 Evaluations in OpenTI38 5.1 Corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5.2 Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.3 Validation Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6 A Research Agenda for OpenTI50 6.1 Addressing Data and Methodological Needs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 6.2 Making Systems Trustworthy and Cooperative . . . . . . . . . . . . . . . . . . . . . . . . . . 51 6.3 Rethinking How Systems Are Built and Evaluated . . . . . . . . . . . . . . . . . . . . . . . . 53 7 Conclusion55 A Overview of Notation85 B Comparison with Related Work86 C Literature Review Protocol87 D Input Classification Methodology89 E Citation Analysis91 2 1 Introduction Tabular data, found in relational databases, spreadsheets, data lakes, on the web, and as standalone files, is among the most pervasive forms of data in our digital society (Borisov et al., 2024; Shwartz-Ziv & Armon, 2022; Cafarella et al., 2008). Organizations across the public and private sectors accumulate it at scale and depend on it to make decisions (Jagadish et al., 2014). Tables owe this prevalence to their compact and semantically rich structure that encodes entities, their attributes, and the relationships among them in a form that is both machine-readable and amenable to analysis (Codd, 1970). Yet the knowledge that makes this data valuable does not just reside in the cells themselves. A table of measurements does not state how a quantity develops over time, where it concentrates, or how two variables relate. Instead, such knowledge is implicit, surfacing only while relevant data is located, integrated, and methodically subjected to computations (Jagadish et al., 2014; Herzig et al., 2021; Ji et al., 2024). What people require to act is therefore not the tabular data itself but the knowledge derived from it. Deriving this knowledge is demanding (De Bie et al., 2022), and not only because of the analysis it requires. Organizations often hold vast collections of tables without a precise account of what they contain or where any particular information resides, which makes locating the relevant data itself a substantial task (Bogatu et al., 2020; Nargesian et al., 2019). Both locating the data and analyzing it require skills, effort, and familiarity with the data (Jagadish et al., 2014) that those who have a need for that knowledge may lack. This is characteristic of the open setting in organizations, where individuals do not know the location, contents, structure, and coverage of the required data in advance and where time, resource, and skill constraints are practical hurdles to extracting knowledge. For instance, consider a public-health officer who wants to know how nitrogen dioxide (NO 2 ) pollution varies across a city’s boroughs to inform a future traffic policy. Measurements are collected by the city’s sensor network, yet the officer does not know where in the municipality’s data lake they are stored, how they are organized, and how complete they are. The measurements are tagged with coordinates, so further data on administrative boundaries needs to be located and combined to analyze the pollution per borough. Analytical knowledge remains effectively locked behind the discovery and analysis needed to surface it. Democratizing access to it could do for the knowledge held in tables what web search does for information held in documents, letting anyone with a need obtain insights without a background in data analysis or programming. Realizing this potential calls for treating the problem end-to-end, spanning the full path from the need a person has to presenting the knowledge in a form that delivers insights. We refer to the pursuit of making the knowledge held in tabular data accessible to those who require it as Open Tabular Insight Extraction (OpenTI). We understand end-to-end OpenTI as a destination that research is advancing toward and whose foundations this work develops. Open Tabular Insight Extraction (OpenTI) is the task of satisfying a sufficiently directed insight need by deriving and presenting the analytical knowledge that most fully serves it, over a corpus of tabular data in an open setting where corpus contents are potentially diverse and not known when the need is expressed. The term Open Tabular Insight Extraction captures the characteristics of the task. It is open in that the need is expressed over a corpus whose contents, schema, and coverage are unknown in advance, requiring exploration of the available data. It concerns tabular data, which enables analysis and shapes how data is discovered and combined. It targets insights because the underlying need is satisfied through analytical knowledge that, interpreted within a person’s contextual understanding, yields an insight. And it is extraction because this knowledge is derived from the discovered data through analytical operations. Several research communities work toward easing access to the knowledge held in tabular data, each from a different starting point. Database (DB) research has long pursued dataset search to surface relevant tables within large collections (Chapman et al., 2020; Nargesian et al., 2019), while semantic parsing of natural language into logical forms such as SQL (Liu et al., 2025a) sits at the intersection of DB and natural language processing (NLP). NLP has further focused on answering questions over tables and reasoning about their 3 20212022202320242025 Year 0 8 16 24 32 40 48 56 64 Number of relevant papers Total papers DB HCI IR ML NLP Tabular 0k 5k 10k 15k 20k 25k Total published papers (a) Total number of papers, and papers we select as relevant, across research communities, illustrat- ing increased attention in the topic of OpenTI. 202120222023202420252026 Year Lookup Aggregation Characterization Association Grouping Statistical inference Prediction Interventional Counterfactual 603939122317 405853595840 221052 6988 1 3 4314 116 111 Descriptive Inferential Causal 0 10 20 30 40 50 60 % of queries (b) Prevalence of different types of analytical tasks in benchmarks per year, as the share of inputs averaged with equal weight per benchmark, showing a trend towards more diverse and complex tasks. Figure 1: Growth and diversification of research relevant to OpenTI. Benchmark statistics cover 2026 only partially. structure (Zhou et al., 2026; Lu et al., 2025a). Information retrieval (IR) has advanced the retrieval of tables relevant to a query (Wang et al., 2021; Chen et al., 2024a; 2025d), building on the open-domain tradition of retrieving evidence before answering (Chen et al., 2021; Herzig et al., 2021). Recent research on agents assembles foundation models into systems that plan and execute analytical work over data (Chen et al., 2025a; Tang et al., 2025). Human-computer interaction (HCI) has studied visual presentations of analytical results and the interfaces for specifying and refining the analysis (Srinivasan & Stasko, 2017; Setlur et al., 2016). Each direction advances a complementary facet of OpenTI, yet none spans the whole. This activity is broad and accelerating faster than surrounding fields. We identify 160 relevant works from 18 venues spanning ML, IR, HCI, DB, and NLP in a systematic literature review, as detailed in Appendix C. Between 2021 and 2025 the number of works relevant to OpenTI rose roughly eightfold, far outpacing the overall growth at the same venues (Figure 1a). Simultaneously, relevant works have increasingly appeared at cross-cutting machine learning venues and specialized workshops, indicating that interest is not limited to any single originating community. OpenTI is thus emerging as an area in its own right. That this is happening now reflects that the capabilities needed to holistically address the problem are maturing. Advances in retrieval and learned representations of tables enable uncovering relevant data from large corpora (Wang et al., 2021; Chen et al., 2024a; Herzig et al., 2020), while Large Language Models and the agentic methods built upon them provide flexible means to address the full path from the expression of a need to presenting an insight (Chen et al., 2025a; Tang et al., 2025). Capabilities traditionally pursued in isolation can thus be assembled into systems that attempt insight extraction end-to-end. Simultaneously, the range of data analysis workloads that systems are expected to cover is broadening. Figure 1b exemplifies this, showing a clear trend of the scope of benchmarks broadening over time, and only recently expanding beyond lookup and aggregation workloads. The problem is thus outgrowing the narrow framings under which its facets have been studied. The components and the demand exist, yet the work remains scattered across communities without a shared account of the problem they collectively address. While the systems proposed across these communities have grown more capable, they remain largely opti- mized for a narrow target, the information directly queried for, with success measured by how accurately an exact answer is returned. This only partly captures the requirements on surfacing knowledge from tabular data. Information extracted from tables is seldom wanted for its own sake but for the insight it enables for the one who sought it. What is requested and the insight actually sought need not align. The public-health officer from above may ask for a simple average ofNO 2 per borough, yet that figure reflects where sensors sit rather than where people are exposed, misleading the very policy decision the officer needs to inform. Existing efforts to systematize the problem approach parts of it but fall short of providing a holistic account that spans the full end-to-end scope. Systematic efforts have organized works on open-domain question an- 4 swering (Zhang et al., 2023b), retrieval augmented generation (Zhao et al., 2024a), and LLM-agents (Kapoor et al., 2024; Sumers et al., 2024), establishing a structured perspective on systems that perform complex tasks over unseen information, yet none address tabular data. Those that do remain largely confined to surveying works in a narrow descriptive manner, organizing works within the conventions of specific research communities under labels such as text-to-SQL, table question answering, or data science agents (Fang et al., 2024; Lu et al., 2025a; Liu et al., 2025a; Zhou et al., 2026; Tang et al., 2025; Tian et al., 2026; Chen et al., 2025a), which we compare against in detail in Appendix B. We argue that these labels reflect facets of one problem, that treating them separately obscures what has been established and what remains open, and that a unified account of that problem is achievable from first principles. This separation is reflected in how works engage with one another, where works cite others that use the same label at six times the rate they cite across labels (Section 2.3). We therefore develop OpenTI as a field of its own, providing the shared vocabulary and problem definition needed to transfer knowledge between communities and to compare sys- tems and benchmarks beyond the conventions of the community each originates from. Overall, we make the following contributions: 1. Conceptual foundation. We establish principled foundations for OpenTI by formalizing it as a unified problem setting that addresses insight needs with analytical knowledge derived through realizations over corpora of tabular data (Section 2). This formulation allows deriving the functional capabilities systems must exercise (Section 3), and it separates establishing that a realization is valid from measuring how well it serves the need, revealing that current benchmarks approximate the former while leaving the latter largely unmeasured (Section 5). We use this framework to position systems and benchmarks from different research communities within a single scheme (Tables 1 and 2). 2. Review and analysis. We systematically review how works from different research areas work towards OpenTI. We apply the conceptual framework to position systems by the functional capa- bilities they exercise and the way that they orchestrate the interplay between them (Section 3) and analyze benchmarks against desired criteria for robust evaluations (Section 5), showing that current systems and benchmarks only partially cover the scope and requirements of OpenTI. 3. Synthesis. We consolidate fragmented work on the human-side dimensions of OpenTI under a framing of cooperative interaction. We systematize interaction modes and means and examine interpretability, personalization, and contextualization as means of aligning system realizations with users’ latent insight needs (Section 4). 4. Open challenges and research agenda. We derive a research agenda for OpenTI in which directions are anchored in the gaps and limitations surfaced in the review, structured along key questions towards reliable and accessible insight extraction (Section 6). We find a consistent pattern across existing research. Effort concentrates on performing limited types of analysis over data that is already at hand, while complementary aspects within the end-to-end scope remain thin. 41 of the 58 systems we review lack any capability to retrieve tables, while outputs are predominantly text and rarely contextualized for specific insights. Similarly, we find current benchmarks to be largely unfit for evaluations under an open setting. We analyze inputs from benchmarks and find that they routinely reference columns, files, and values that a user in an open setting could not know, and that they can be interpreted in multiple ways in the majority of instances of all but two benchmarks while largely assessing the validity of outputs or the analysis against a single reference. 2 Open Tabular Insight Extraction Since research towards OpenTI originates in different communities that address overlapping problems with divergent terminology and assumptions, this section develops a formal framework for Open Tabular Insight Extraction (OpenTI). We lay out this shared understanding by establishing the core concepts and formal- izing OpenTI (Section 2.1). We then examine the conditions under which insight needs can be satisfied (Section 2.2). Finally, we cover the spectrum of insight needs that OpenTI addresses (Section 2.3). 5 Insight Need Data need Output need Methodological need Expression of insight need Representation of insight need Contextual understanding Insight Corpus of tables Conclusion Realization Analytical knowledge User Presentation Execution user-system interaction system derives execution yields influences influences basis for contributes to contributes to linked by data binding Figure 2: Relationships and dependencies among core concepts of OpenTI. A user holds an insight need i, consisting of a data need i d , methodological need i m , and output need i o , and expresses it as ˆ i. A system maps ˆ i to a realization r = (O,E,β), a graph of operations O connected by dataflow edges E with a binding β that links operations to tables from the corpus T. Executing r yields analytical knowledge as output out(r), which is the basis for an insight. 2.1 Conceptual Foundation We develop the conceptual foundation of OpenTI from first principles. We first ground the general notion of an insight as the form of knowledge to be delivered (Section 2.1.1). We then operationalize this into insight needs, characterizing what users seek in tabular data and along which dimensions it varies (Section 2.1.2). Building on this, we formulate the problem of OpenTI (Section 2.1.3), show how the analytical knowledge that serves an insight need derives from data through realizations, executable compositions of operations bound to data (Section 2.1.4), and finally formulate OpenTI as an optimization problem (Section 2.1.5). Figure 2 provides an overview of the relationships between the concepts introduced throughout this section. 2.1.1 Insights Insights have been defined and discussed in various ways across visualization research (North, 2006; Chang et al., 2009; Saraiya et al., 2005), cognitive science (Sternberg & Davidson, 1995; Mai et al., 2004), and data analytics (Mikalef et al., 2018; Tang et al., 2017; Ding et al., 2019), with ongoing debate about their defining characteristics, including whether insights arise unexpectedly by definition or not (North, 2006; Chang et al., 2009; Battle & Ottley, 2024). Adapting the recent synthesis of Battle & Ottley (2024), which consolidates these perspectives, we understand an insight as a collection of knowledge that links analytical knowledge derived from data with a person’s pre-existing contextual understanding. Analytical knowledge is the direct output of computational operations on data, whereas contextual understanding denotes a person’s pre- existing knowledge, including domain expertise, organizational background, prior experiences, and previously acquired information (Pirolli & Card, 2005). As such, insights exhibit the following characteristics: • Grounded in data. An insight is grounded in analytical knowledge extracted from data through analytical operations, such as transformations, statistical modeling, or other computations (Battle & Ottley, 2024). • Contextual. An insight is not the analytical knowledge itself but its interpretation within a person’s contextual understanding (Battle & Ottley, 2024; Karer et al., 2021). The role of contextual un- 6 derstanding ranges from minimal, when the analytical knowledge is largely self-explanatory (e.g., a retrieved fact), to substantial, where domain expertise fundamentally shapes interpretation (e.g., in understanding ROC curves or residual plots). • Independent of presentation. Since insights emerge from interpretation, they are not bound to a specific form of presenting analytical knowledge (Battle & Ottley, 2024). Instead, different forms of presenting analytical knowledge may invoke the same insights. • Compositional. Insights contribute to a person’s contextual understanding, and subsequent insights can build on this enriched context (Battle & Ottley, 2024). This hierarchical structure means that complex insights may build on the context of prior insights (North, 2006), that is, there exist insights that compose a set of sub-insights, which may themselves be composed of sub-insights. 2.1.2 Insight Needs An insight need arises when a person 1 requires analytical knowledge that is derived from data. We ground this concept in the established notion of information needs from information retrieval, where an information need represents the gap in a person’s knowledge that motivates interaction with an information system (Taylor, 1962; Belkin, 1980). An insight need specializes this to the analytical domain by targeting a knowledge gap concerning analytical conclusions residing in data, where satisfying this need requires computationally deriving knowledge rather than retrieving pre-existing information alone. Definition 1. An insight need i is a person’s requirement for analytical knowledge which, when interpreted within their contextual understanding, constitutes an insight. Insight needs are analytical and insight-centered. They seek analytically derived knowledge (i.e., facts, patterns, relationships, predictions, or other analytical conclusions), rather than to produce an artifact. For instance, a need for a trained machine learning model is not an insight need, though artifacts may be instrumental in delivering insights, such as when a model is applied to make a desired prediction. In addition to these characteristics, insight needs vary along the following dimensions: • Directedness. Insight needs range from highly directed, targeting specific phenomena, entities, or relationships (e.g., “What is the correlation between drug dosage and patient recovery time?”), to loosely directed, where the analytical perspective remains open (e.g., “What patterns exist in hospital readmission data?”) (Marchionini, 2006). The degree of directedness determines whether it is possible to evaluate whether candidate analytical knowledge satisfies the insight need. • Latency. The actual insight need i exists in a person’s mind and may not be entirely specified in their expression of it (Zhu et al., 2025a). A person can express an approximation ˆ i through utterances and contextual signals. This gap between latent need and expressed need mirrors the classical model of information need formation by Taylor (1962), which traces how needs progress from an unexpressed visceral sense through conscious and formalized stages to a compromised expression adapted to system capabilities. For insight needs, the complexity of the need is compounded by the analytical requirements of selecting which operations should be performed on which data. Latency is rooted in the person’s contextual understanding, including tacit knowledge the person cannot fully articulate (Polanyi, 1967), assumptions the person considers obvious, and preferences the person has not consciously examined (Belkin, 1980; Taylor, 1962). • Compositionality. Following the compositional nature of insights, insight needs can also be composed of atomic needs. They may decompose into sub-needs that jointly describe the insight need, which may themselves decompose recursively, allowing arbitrarily complex compositions of insight needs. 1 We refer to a person when characterizing insight needs conceptually and to a user when referring to the entity interacting with a system. We note that this entity need not be human and may instead be an automated agent acting on an insight need. 7 2.1.3 Open Tabular Insight Extraction Insight needs may target any form of data and any analytical domain. Open Tabular Insight Extraction scopes this to satisfying insight needs by deriving analytical knowledge from a corpus of tables, where users may have an incomplete knowledge of its contents, schemas, and coverage. Definition 2. Open Tabular Insight Extraction (OpenTI) is the task of satisfying a sufficiently directed insight need by deriving and presenting the analytical knowledge that most fully serves it, over a corpus of tabular data in an open setting where corpus contents are potentially diverse and not known when the need is expressed. Recalling the directedness dimension introduced above, OpenTI addresses insight needs that are sufficiently directed so that it is determinable whether candidate analytical knowledge satisfies the need. Since the need itself is latent, no party can check this condition directly. We thus operationalize this sufficient specification on the observable expression of the insight need ˆ i in Section 2.2.1. Tabular Data. The task operates on a collection of tabular data. A table t consists of a header, an ordered set of column names that defines the table’s schema, a data matrix holding the core data values, where a row typically corresponds to an entity and a cell holds the value of the attribute named by its column, and meta- data that may provide contextual information such as titles, captions, column descriptions, or surrounding text (Liu et al., 2024a). A corpus T =t 1 ,t 2 ,...,t n ⊂T comprises n such tables and can correspond to any large-scale collection of tables like a data lake, warehouse, or repository. In practice, tabular data largely exists in relational databases (Codd, 1970), which add interconnections between tables through explicit schema relationships such as primary and foreign keys that define how entities relate across tables and how distributed information can be combined. In contrast, data lakes (Nargesian et al., 2019) and collections of web tables (Cafarella et al., 2008; Hulsebos et al., 2023) aggregate tables from heterogeneous sources without such explicit relationships, exhibiting variation in schema quality, naming conventions, and completeness. The existence or absence of these relationships places different requirements on systems (Section 2.1.4). Open Setting. OpenTI operates under an open setting in which users express insight needs over a cor- pus whose contents, schemas, and coverage are unknown to them. This contrasts with closed settings (Yu et al., 2018; Li et al., 2023), where users formulate requests with implicit knowledge of the available data. Users therefore cannot be expected to express insight needs in terms that align with the data’s vocabulary, structure, or scope, requiring systems to identify relevant data from T. As under the open-world assump- tion (Reiter, 1981), users do not know whether relevant data is captured in the corpus, meaning the system has to establish whether T covers a given insight need. As in open-domain (tabular) question answering and retrieval (Voorhees, 2001; Kong et al., 2023; Chen et al., 2021; Herzig et al., 2021), the relevant data must be retrieved from T, though without being contingent on topical breadth, meaning that a large single-domain data lake is also considered open under OpenTI. Decomposition of Insight Needs. We decompose insight needs into data, methodological, and output factors following task typologies in visualization, which distinguish the dimensions of “what”, “how”, and “why” (Brehmer & Munzner, 2013), and mirroring the implicit structure of database queries, which separately specify data selection, analytical operations, and output formatting: 1. Data need (i d ): The specification of what data entities, temporal scope, and domain boundaries are relevant for the analysis. This may be explicitly stated (e.g., “revenue of Fortune 500 companies in 2023”) or implicitly indicated through contextual cues and conventions (e.g., “GDP of Liechtenstein last year”, if the current year is known by the system). 2. Methodological need (i m ): The specification of what analytical operations and procedures should be applied to extract the desired analytical knowledge. This ranges from explicit methodologi- cal directives (e.g., “calculate the Pearson correlation”) to high-level analytical goals that delegate methodological choices to the system (e.g., “analyze the relationship between. . . ”). 3. Output need (i o ): The specification of how results should be presented. This may remain com- pletely unspecified by a user that expects the system to select an appropriate presentation. 8 2.1.4 Realizations We formalize the relationship between data and the derived analytical knowledge as realizations: Definition 3. A realization r = (O,E,β) is a complete, executable specification of computational logic with a binding of specific tables to that logic, which, when executed, yields analytical knowledge as output out(r). We adopt the term from systems theory, where a realization denotes a concrete implementation of an input- output behavior (Kailath, 1980; Kalman, 1963). In OpenTI, a realization captures a derivation path from data in the corpus to analytical knowledge end-to-end. Whether the analytical knowledge a realization derives satisfies a given need is a property of the realization relative to that need. We represent realizations as a Directed Acyclic Graph (DAG) of operations connected by edges denoting data flow, similar to representations of computations established in scientific workflow specifications (Ludäscher et al., 2006; Deelman et al., 2009) and database query execution plans (Graefe, 1994). We adapt this representation to insight extraction, where computational operations are anchored to data discovered from a corpus rather than pre-specified inputs. Unlike a database query, a realization is often a multistep workflow involving data discovery, integration, transformation, analysis, and presentation. Formally, a realization r ∈R is a tuple r = (O,E,β) describing a converging directed acyclic graph of operations: • O =o 1 ,o 2 ,...,o m is the set of fully parameterized operations applied in the derivation. • E ⊂ O×O is a set of directed edges connecting the output of operations to the inputs of subsequent operations, defining the data flow. • β ⊂T ×O is the data binding that connects specific tables T r ⊆T to the source nodes of the graph. An operation o represents a discrete functional unit, which can range from granular primitives (e.g., filter, join, . . . ) to high-level analytical functions (e.g., fit a regression model, solve an optimization problem, . . . ). Each operation in O is fully parameterized, meaning all relevant choices, such as aggregation functions, model specification, or visualization parameters, are determined. The edges E capture the compositional structure of the derivation, specifying how intermediate results flow between operations. The data binding β anchors the computation to concrete tables, specifying which tables serve as inputs to which operations. The DAG converges into a single sink operation, which produces the final output resulting from the realization. Figure 3 exemplifies an abstracted realization that derives analytical knowledge to address an insight need. Importantly, a realization is a declarative account of how analytical knowledge is derived from data. It does not capture the process by which a system arrived at this specification, which may include exploratory search, retrieval failures, user interactions, and iterative refinement. This parallels the established differentiation between prospective provenance (the specification of a computational task) and retrospective provenance (the record of what was actually executed) in scientific workflow research (Davidson & Freire, 2008; Freire et al., 2008). Section 3 examines the process by which systems derive realizations. Multiple Realizations and Utility. A single insight need may admit many realizations. When users express a methodological goal like “analyze the relationship between X and Y,” different operational compo- sitions (e.g., Pearson correlation, Spearman correlation, regression analysis) may each constitute a valid way to address this intent. Similarly, different combinations of tables from T could support distinct realizations. This one-to-many relationship reflects a division of labor in cooperative interaction between the user and the system (Gomm et al., 2025; Horvitz, 1999), where users grant systems agency over the aspects of their needs they leave open, requiring the system to select among the resulting alternatives. These alternatives, however, are rarely equal in value. A realization drawing on more recent or more complete data, employing more robust analytical methods, or producing clearer output may substantially better serve the user’s goals. We model this qualitative dimension through a utility function u i . Definition 4. For a given insight need i, the realization utility function u i :R→ R maps possible real- izations to a real-valued score, where a higher value indicates a realization that more effectively, accurately, or suitably satisfies the user’s insight need. 9 Filter Join Join Transform Plot Integrate Corpus Realization Output Figure 3: Example of an abstracted realization r = (O,E,β). Nodes are fully parameterized operations O, edges E denote dataflow, and the binding β anchors source operations to corpus tables. The graph converges to a single sink operation producing the output. The utility function aggregates the quality with which a realization addresses data, methodological, and output needs, capturing not just the value of the final outputs but the full realization performed (Ghallab et al., 2004). For the data component, utility reflects the relevance of the data (Saracevic, 2007; Cooper, 1971), reflecting factors such as recency, completeness, and reliability (Wang & Strong, 1996). For the methodological structure, utility may capture the appropriateness and rigor of analytical procedures given the analytical intent. For the output presentation, utility may reflect clarity, informativeness, as well as the effectiveness and appropriateness of visualizations (Lei et al., 2025b). While users do not explicitly specify what contributes to their utility functions (Borlund, 2003), these functions embody the implicit quality criteria that distinguish superior realizations from adequate ones. Utility only captures how well a realization satisfies an insight need, regardless of how it was derived. Computational effort, latency, and the interaction burden placed on the user are properties of the process by which a system constructs a realization, which influence the usefulness of a system but do not influence the utility of a realization. 2.1.5 Problem Formulation A realization is valid for an insight need i if the tables bound through β provide the information the data need i d mandates, its operations and their composition correctly address the methodological need i m , and its output delivers the kind of result the output need i o calls for. Validity is a property of the realization relative to the insight need, judged on the content of the bound tables and the analytical logic applied to them, independent of whether the bound tables actually exist in the corpus. We denote the set of valid realizations for an insight need i as R valid (i) ⊂ R. Validity and utility are complementary. Validity is a binary criterion that separates realizations addressing the insight need from those that do not, whereas the utility function u i grades how well realizations serve it. A realization can thus be valid yet exhibit low utility, for instance when its output delivers the required analytical knowledge but presents it poorly. A realization is feasible given a corpus T if the tables it requires are present in the corpus, that is, all tables bound via β exist in T, T r ⊆ T. Feasibility is a property of the data binding relative to the corpus, indepen- dent of whether the realization addresses any particular insight need or whether a given system possesses the capabilities to combine the bound tables. We denote the set of feasible realizations as R feasible (T )⊂R. The set of candidate realizations R cand (i,T ) = R valid (i)∩R feasible (T ) captures realizations that both address the insight need and are grounded in available data. The realization that optimally satisfies the insight need is then the candidate realization r ∗ that maximizes utility: r ∗ = ( arg max r∈R cand (i,T) u i (r), if R cand (i,T )̸= ∅ ⊥,if R cand (i,T ) = ∅, (1) where ⊥ represents a distinct failure state indicating that the insight need cannot be resolved given the corpus. This objective is defined relative to the insight need i and thus characterizes an idealized target. A system never has direct access to i, and consequently not to R valid (i) or u i , observing only the expressed need 10 ˆ i. To construct an approximation of r ∗ , a system must construct its own representation of the need, denoted as ˆ i s , from the expression ˆ i and available contextual information, bringing ˆ i s into closer correspondence with i (see Section 4). The realization space R is combinatorially vast, as the number of possible operation compositions, parameterizations, and data bindings grows with the scale of the corpus and the expressiveness of the available operations. Neither an analytical solution nor enumeration is practical. Section 3 thus explores how systems can navigate this space effectively to construct utility-maximizing realizations. 2.2 Sufficiency Criteria Not all insight needs can be satisfied in OpenTI. As implied in the formalization, a system can only produce an output that satisfies the insight need if the candidate set R cand (i,T ) = R valid (i)∩ R feasible (T ) is non- empty. This requires that the expression of the insight need ˆ i is sufficiently specified for a system to delineate R valid (i) (Section 2.2.1), and that the corpus T contains sufficient data so that at least one valid realization is feasible, i.e., R valid (i)∩ R feasible (T ) is non-empty (Section 2.2.2). 2.2.1 Sufficient Specification Sufficient specification describes the expressed need ˆ i relative to the underlying need i. ˆ i needs to allow delineating R valid (i), separating realizations that satisfy the insight need from those that do not. This requires that the data, methodological, and output components i d , i m , i o are specified or can be inferred accordingly. For instance, “What is the average temperature in Amsterdam in January?” is sufficiently specified since the implicit methodological details can be resolved through conventions and reasonable defaults. In contrast, “How does construction waste compare?” is insufficient since no convention fixes what to compare against, by what metric, or in what region, leaving valid and invalid realizations indistinguishable. We discuss how systems close such gaps autonomously and in collaboration with the user in Section 4. 2.2.2 Data Sufficiency Even a fully specified insight need is impossible to satisfy if the corpus lacks the required data, which is particularly relevant when users are unaware of the contents and structure of T. Satisfiability requires that the corpus T supports at least one valid realization. We formalize this as sufficient sets of tables: Definition 5. A set of tables T s ⊆ T is sufficient for an insight need i if there exists at least one valid realization r = (O,E,β)∈ R valid (i) where the set of tables T r bound via β satisfies T r ⊆ T s . When no sufficient set exists within the corpus, that is ∄T s ⊆ T sufficient for i, the candidate set is empty and the insight need cannot be resolved, an instance of insufficient data coverage for i in T. Within a sufficient set, tables align both with the insight need and with each other. Following Kothyari et al. (2023) and Chen et al. (2024a), we distinguish two forms of alignment: 1. Semantic alignment (need-table): The tables in T s collectively cover all semantic concepts specified in the data need i d . 2. Structural alignment (table-table): Tables holding complementary information are combinable through the operations available to the system (e.g., joinable via shared keys, unionable,. . . ). The two forms are independent. A set may cover all relevant semantic concepts yet be structurally incom- patible, and tables may be perfectly joinable yet lack the required semantic coverage, both rendering the set insufficient. The insight need defines how semantic components must relate, while a system’s operational capabilities determine how information can be combined across tables. Figure 4 exemplifies these conditions. Where the required information does not reside in a single table, it must be integrated across multiple tables. As shown, a sufficient set must cover population counts, GDP values, country identifiers, and the year 2021 (semantic alignment), and these must be integrable through the means a system possesses, here through shared keys like CountryCode (structural alignment). The conditions for structural alignment depend on the capabilities of the system processing the data. For a system capable 11 Example Query: Plot the relationship between the population andGDP per capita of eachEU country in2021 . Not semantically & structurally aligned Not structurally aligned Semantically & structurally aligned CountryPopulation Germany83 200 000 France67 400 000 · population_by_country_2021 Region EmploymentRate DE1175.5% DE1266.2% · employment_2021 GDP information is missing and tables cannot be combined. CountryPopulation Germany83 200 000 France67 400 000 · population_by_country_2021 PaysPIB Allemagne€46 000 France€39 000 · pib_par_habitant _2021 All attributes present but can- not be combined because of missmatch in country keys. CountryCodePopulation DE83 200 000 FR67 400 000 · country_population_2021 CountryCodeGDP DE€46 000 FR€39 000 · gdp_per_capita _2021 All attributes present and joinable on CountryCode. Figure 4: Example of semantic and structural alignment. A candidate set is semantically aligned when its tables jointly cover the required concepts and structurally aligned when they are combinable, here via a join on CountryCode. Both must hold for sufficiency. of aligning French and English contents, for example, the middle column would be structurally aligned. How systems surface such sets and navigate the search over candidate sets that both conditions induce is examined under the data retrieval capability in Section 3.3.2. 2.3 Taxonomy of Insight Types Deriving the analytical knowledge that satisfies an insight need requires substantially different assumptions and capabilities depending on which kind of knowledge is sought. We organize this space by insight type, the kind of analytical knowledge a realization derives, rather than by insight need. An insight need i is latent and only partially captured by its expression ˆ i (Section 2.1.2), whereas the insight type is observable on the realization and most directly determines the capabilities a system requires. The operations O and their composition E differ categorically across types, such as between reading a stored value and estimating a causal effect, while data and output complexity vary within a type without imposing qualitatively different requirements. Figure 5 organizes insight types into three tiers ordered by what must be assumed, beyond locating the relevant data, for the derived knowledge to be justified, following the established distinction between descriptive, inferential, and causal analysis (Tukey, 1977; Pearl, 2009) applied to OpenTI. Descriptive insight types concern the observed data itself and require no assumptions about anything beyond it. They form the most varied and, in current research, the most common tier (see Figure 1b), ranging from exact computed facts to summary characterizations of the data. Retrieving a cell value or selecting entities that meet a criterion (Lookup), summarizing values over records such as counts, sums, extrema, and rankings (Aggregation), describing the distribution of a variable (Characterization), measuring how variables co-vary (Association), and surfacing latent groups among records (Grouping) all describe what is present in the data and commit to nothing further. Many such needs, like computing a key performance indicator or ranking entities by an aggregate, have a single correct answer fixed by the data, yet deriving them can demand considerable retrieval, integration, and computation across a corpus. Others, like characterizing a distribution or quantifying an association, produce summary statistics, but these still describe the actual 12 Insight Types Descriptive Knowledge about the observed data, requiring no assumptions beyond it. Lookup Reading specific values or records present in the corpus, covering both attribute values for identified entities and sets of entities matching stated criteria. “What was the population of Germany in 2022?” “Which countries joined the European Union after 2000?” Aggregation Computing summary values over a set of records, including counts, sums, extrema, and rankings. “Which Mercosur member country has the highest public debt as a share of GDP?” Characterization Describing the statistical distribution of a variable, including its central tendency, spread, and shape. “How are household incomes distributed in New Zealand?” Association Measuring how two or more variables co-vary in the observed data, ranging from correlation to general dependence. “What is the correlation between public education spending and literacy rates across countries?” Grouping Discovering segments of entities or records that share latent structure across attributes, without a predefined target variable. “What clusters of countries exist based on their energy-mix profiles?” Inferential Knowledge about the data-generating process behind the observed data, requiring assumptions that link the two. Statistical Inference Determining whether an observed pattern reflects a stable feature of the underlying process rather than noise, with parameter estimates and quantified uncertainty. “Is there evidence of a positive association between public education spending and literacy rates across countries?” Prediction Estimating unknown or future values for unobserved instances by generalizing observed patterns, spanning regression, classification, and forecasting. “What is the expected population of Algeria in 2040?” Causal Knowledge about the effects of interventions or counterfactual states, requiring assumptions on the underlying causal structure. Interventional Determining the effect of an action or policy at the population level. “How would commuting mode choices change if fuel prices rose by 25%?” Counterfactual Determining the outcome for a specific observed instance under a condition contrary to the known facts. “Had the price not increased, would this customer have churned?” Figure 5: Taxonomy of insight types in OpenTI, organized by the assumptions required for the derived analytical knowledge. Each tier encompasses representative insight types with examples. records rather than generalizing beyond them. Inferential insight types concern the data-generating process behind the observed data, making claims that are meant to hold for instances the corpus does not contain and therefore relying on assumptions that link the data to that process. Determining whether an observed pattern reflects a stable feature of the process rather than noise (Statistical Inference) and estimating values for instances the corpus does not contain, including future states (Prediction), both reach beyond the data at hand, differing in whether the goal is to characterize the process or to forecast values (Shmueli, 2010; Breiman, 2001). Causal insight types make the strongest claims, concerning not what the process looks like but how it would respond to intervention. Estimating the effect of an action or policy at the population level (Interventional) and determining the outcome of a specific case under a condition contrary to fact (Counterfactual) cannot be settled from observed patterns alone and require assumptions about the causes behind the data, such as a structural causal model (Pearl, 2009). 13 These tiers stand in an asymmetric relationship to each other. Knowledge from a higher tier entails regular- ities on a lower tier, but not vice versa. A model of the process entails the patterns in any data it generates, and a causal account of the process entails both those patterns and the effects of intervening on it. On the other hand, no observed association, however strong, implies how the process would respond to intervention, and no body of observations uniquely determines the process that produced them. For example, a structural causal model fixes the distribution of the data it generates and the effect of intervening on any variable, whereas that same distribution is consistent with many distinct causal models that disagree about those effects. The familiar maxim that correlation does not imply causation is an example of this relationship. The tiers in the taxonomy shown in Figure 5 span the full spectrum of possible insight types based on analytical knowledge extracted from tabular data. An analytical claim either concerns the observed data or generalizes beyond it to the process that produced it. A claim about that process either concerns the process as observed or under intervention (Pearl, 2009). Any analytical outcome thus falls into exactly one tier, and no further level of assumptions exists between them. The insight types listed within each tier are representative but not necessarily exhaustive, capturing the categories most relevant in OpenTI. We ground the descriptive types in established accounts of low-level analytical primitives (Amar et al., 2005), the inferential split in the distinction between explanatory and predictive modeling (Shmueli, 2010; Breiman, 2001), and the causal tier in the causal hierarchy (Pearl, 2009). The insight types in the taxonomy are atomic. Since insights are inherently compositional (Section 2.1.1), an insight may compose an arbitrary number of atomic insights of the same or different types, and many common analytical tasks represent such compositions. For instance, comparing two means composes two aggregations when it asks which is larger, and becomes a statistical inference when it asks whether the difference reflects the underlying process rather than noise. Similarly, analyzing a trend decomposes into a descriptive slope over observed values or an inferential forecast. At the far end of this spectrum, insight needs may demand elaborate compositions that layer substantial methodology onto atomic types, as in scientific workflows that apply optimization, computational modeling, or simulation, all of which inherit the tier of the knowledge they operate over rather than forming a tier of their own. The latent nature of insight needs means that the type implied by an expression need not match the type that satisfies the underlying need. The same comparison, expressed as “Does X or Y have the larger mean Z?”, resolves on its surface to a composition of aggregations, yet the user may actually seek to know whether the two means differ meaningfully, which is a question of statistical inference. We develop this methodological latency in Sections 4 and 6. text-to-SQLtable QA data analysis agent Task Label of Cited Work text-to-SQL table QA data analysis agent Task Label of Citing Work 10.9%0.9%0.6% 3.4%10.4%0.3% 3.0%1.3%6.9% (a) Share of chronologically feasible citations between works by their task label. 01234-5 Shared Functional Capabilities 0% 5% 10% 15% 20% Citation Rate Same-label pairs Different-label pairs (b) Citation rate among reviewed systems by the number of capabilities two systems have in common. Figure 6: Citation rates between reviewed works against how they label the task they target. Appendix E details the analysis. This taxonomy situates OpenTI within a landscape of related but independently studied tasks. A substantial body of work covers deriving knowledge from tables, yet this work originates from different research com- munities that frame the challenge in diverging terminology and without an overarching direction. Tabular question answering (Zhang et al., 2024b; Zhao et al., 2024c; Zhu et al., 2024), table-based reasoning (Kong et al., 2023; Wang et al., 2024b; Lu et al., 2025b), text-to-SQL (Zhang et al., 2025d; Kothyari et al., 2023; 14 Wang et al., 2025a), data science and analysis agents (Huang et al., 2024; Hu et al., 2024; Guo et al., 2024b), and various further characterizations (Wang & Li, 2025; Chen et al., 2024b; Li et al., 2025b) all address the problem space of OpenTI, facing similar challenges and often arriving at substantially similar system designs. Looking at how works across these task terminologies cite each other shows this fragmentation. Figure 6a shows that across the reviewed works, works cite 10.6% of works that are chronologically possible within the same label against just 1.8% of the works of a different label. These labels primarily reflect conventions, such as the assumed input format or data setting, rather than the covered analytical scope. A tabular question answering system may address only Lookup, or it may extend across the descriptive tier and into inferential or causal types, which demand categorically different capabilities, and the label alone does not reveal which. Conversely, systems operating under different labels may address the same insight types with similar capabilities. As shown in Figure 6b, the more capabilities two systems share, the more likely they are to cite one another when they share the same label, but they are barely more likely to do so if they do not. The taxonomy of insight types makes the analytical scope explicit. Together with the preceding formalization, which centers the insight need as the object a system seeks to satisfy, it provides a unified vocabulary for describing what a given work actually concerns. The subsequent sections employ this framework to examine how systems construct realizations (Section 3), the possibilities of user-system interaction (Section 4), how evaluation methodology can assess system capabilities (Section 5), and what research is missing on the path toward capable OpenTI systems (Section 6). 3 Anatomy of OpenTI Systems Section 2 formalized the aim of OpenTI systems to produce realizations that capture how analytical knowl- edge derives from data as an optimization problem over a combinatorially vast space of operations. In this section, we examine how OpenTI systems derive this analytical knowledge. We frame OpenTI systems as agents that navigate the realization space under partial observability and uncertainty (Section 3.1). We then examine how agents orchestrate the analytical process (Section 3.2), and then decompose the functional capabilities that agents must exercise to construct realizations (Section 3.3). For details on the literature review protocol and the inclusion criteria for systems in Table 1, consult Appendix C. 3.1 OpenTI Systems Constructing a realization requires a sequence of mutually dependent decisions, like interpreting the insight need, acquiring relevant data from the corpus, selecting and composing analytical operations, and assessing whether the emerging result satisfies the user’s need. These decisions are mutually dependent since data availability constrains what methods can be applied, the choice of method determines what data is needed, and both depend on the interpretation of the insight need, which itself may need to adapt as data and analytical constraints surface. This makes constructing a realization an instance of a coupled decision problem as studied in automated planning (Ghallab et al., 2004), where consequences of early actions are uncertain and affect which later actions are appropriate. A system must moreover make these decisions with incomplete knowledge of the user’s latent insight need i and the corpus’s contents and structure. OpenTI is therefore a problem of sequential decision-making under mutual dependencies and partial observability (Ghallab et al., 2004; Russell, 2022; Sutton & Barto, 2018; Kaelbling et al., 1998). Accordingly, we understand OpenTI systems as agents that interact with their environment to construct a realization that maximizes the utility. Definition 6. An OpenTI system is an agent that constructs a realization r from a corpus of tables T, which maximizes the utility for an expressed insight need ˆ i by observing the state of its environment, taking actions that progressively construct the realization, and directing this process according to a policy. We borrow this terminology from the study of sequential decision-making under partial observability (Kael- bling et al., 1998; Sutton & Barto, 2018) to characterize the functional role of OpenTI systems without prescribing whether these functions are realized through language model reasoning, learned neural mod- els, classical algorithms, or other mechanisms. While current implementations largely employ LLMs, the design space extends beyond. In OpenTI, the environment, observations, actions, and policy have specific instantiations that shape the setting and the space of possible systems. 15 Environment. The agent’s environment consists of everything outside the agent that it interacts with (Sut- ton & Barto, 2018; Russell, 2022). Following Sutton & Barto (2018), we understand everything the agent cannot change arbitrarily as part of the environment. Thus, the corpus T and insight need i are necessary components of the environment in OpenTI. Agents can explore and use T, but cannot alter it. Similarly, they can observe the expressed insight need ˆ i but not influence the underlying need i. The environment may extend beyond these necessary components to data relationships, contextual resources like documentation or ontologies that reduce uncertainty, interaction histories, or user profiles (see Sections 4 and 6). Observations. While deriving a realization, the agent continually makes observations that inform its subsequent actions (Sutton & Barto, 2018). In OpenTI, these observations comprise the expressed insight need and the intermediate analytical state. In interactive settings users may specify the expression of their insight need throughout the analytical process, for instance through clarification, feedback, or analytical guidance (see Section 4.1). The observed expression ˆ i thus accumulates during the process as the user contributes to it, whereas in non-interactive settings it remains fixed to what the user provided initially. The intermediate analytical state captures the partial realization the agent has constructed so far, the outcomes of executed operations, and the intermediate results it has produced. Since realizations are converging DAGs, it can span multiple independent analytical strands that have not yet converged. The environment is only partially observable since the agent cannot access the latent insight need i, the corpus T has to be explored, and the agent can only estimate the utility against the expression of the insight need it has observed so far. Actions and functional capabilities. At any time, an agent takes an action within its action space (Sutton & Barto, 2018). Actions are the agent’s means of influencing and exploring the environment (Russell, 2022). In OpenTI, actions span the functional capabilities required to derive a realization. The concrete actions available to an agent are determined by its specific implementation and may, for instance, include retrieving tables, and composing and executing analytical operations on the tables. The outcomes of actions in OpenTI are uncertain. For example, retrieval may surface irrelevant data and analytical operations may produce errors. Section 3.3 examines these functional capabilities. The agent’s policy governs how it selects actions based on its observations (Sutton & Barto, 2018; Russell, 2022). In OpenTI, the policy determines how the agent navigates the realization space, how it allocates effort across functional capabilities, and how it selects specific actions within each capability. The policy affects all choices the agent makes, from high-level decisions such as which capability to invoke next to fine-grained ones such as what analytical operations to apply to a table. The policy itself is often bound up in model parameters, prompt design, or hardcoded logic, making it only indirectly observable. OpenTI agents can instead be characterized by higher-level orchestration strategies, which Section 3.2 examines and structures. 3.2 Orchestration Strategies The orchestration strategy describes the structural pattern of how an OpenTI agent composes its func- tional capabilities. Thereby, it provides the high-level scaffolding for the policy applied by the agent. This subsection develops the space of orchestration strategies and positions existing systems within it. The underlying question of when and how the structure of a process is determined has been studied across several traditions. In planning under uncertainty, approaches range from offline derivation of complete policies before actions are taken, through online methods that interleave deliberation with execution and replan as observations are received, to reactive strategies that select actions based solely on the current belief state (Ghallab et al., 2004; Kaelbling et al., 1998). Research into intelligent agent architectures (Wooldridge & Jennings, 1995) distinguishes deliberate agents, which maintain explicit models of the world and plan before acting, from reactive agents, which dynamically decide on actions based on immediate observations, and hybrid architectures that layer both. Database systems differentiate between static optimization, where the execution plan is fixed before any data is processed, and adaptive query processing, which tunes an initial execution plan based on data-dependent feedback (Deshpande et al., 2007). Across these traditions, the fundamental design choice concerns the degree to which the structure of the computational process is fixed before and during the execution of operations. Put differently, it concerns the influence execution- time information, like the actual instance and observations made during processing, has on not-yet-executed operations. Following these traditions, we categorize orchestration strategies by when the topology of the 16 Fixed orchestration Topology fixed at design time CQR-SQL (2022), Cao et al. (2023), OpenTab (2023), Tool- Writer (2023), MITQA (2023), CRUSH4SQL (2023), ITR (2023a), DataQue (2023), S3HQA (2023), DIN-SQL (2023), Multi- TabQA (2023), Dater (2023), ReFSQL (2023a), TabSQLify (2024), ROUTE (2025), PoTable (2024), SynTQA (2024a), TabDSR (2025), CHASE-SQL (2025), ACR (2025b), ALTER (2025c), TART (2025b), SGAM (2025c), TAMO (2025d), FISQL (2025), STaR-SQL (2025), Pi-SQL (2025), H-STAR (2025), SAFE-SQL (2025), TiIn- sight (2025a), TableLLM (2025b), QCMA-SQL (2025), DA- gent (2025), Table-Critic (2025), GTR (2026), UCS-SQL (2025c), RTS++ (2026), LitE-SQL (2026), DS-GURU (2026) Plan-based orchestration Topology fixed at invocation time Construct instance- specific plan Weaver (2025), AixelAsk (2025a) Reactive orchestration Topology determined at execution time Evaluate analytical state against insight need and select next capabilities DAAgent (2024), DA-Agent (2024), Chain-of-Table (2024b), Re- AcTable (2024b), TALON (2025), DataMind (2026), MACT (2025), DACO (2024b) Hybrid orchestration Topology fixed at design, invocation, and/or execution time Example: Fixed macro & Reactive micro Step 1Step 2Step 3Step 4 TableRAG (2024b), TableGPT2 (2024), TaPERA (2024c), Au- toTQA (2024), AOP (2025) Functional capability ConditionFixed transitionConditional transition Legend: Figure 7: Overview of orchestration strategies alongside works that employ the respective strategies. The schematic in each row depicts the corresponding control flow. composition of functional capabilities is determined. In fixed orchestration, the topology is fixed at design time based on assumptions about the analytical process. Plan-based orchestration fixes the topology at invocation time, generating an insight-need-specific topology based on anticipation of required actions for the instance. In reactive orchestration the topology is not predetermined or planned but dynamically composed at execution time based on observations of the analytical environment. These orchestration strategies can also be combined in a hybrid orchestration strategy, for instance by employing different strategies at different levels of granularity, like fixing a macro-level structure while allowing incremental decisions within each phase. 2 Figure 7 organizes the systems surfaced in our literature review along this categorization of orchestration strategies, showing 54 of the 58 systems that orchestrate multiple functional capabilities, with the remaining four only commanding a single capability, requiring no orchestration. We classify most studied systems under the fixed orchestration category, a smaller number as reactively orchestrated or following a hybrid orchestration strategy, and only two as purely plan-based. 2 The main source of ambiguity in classifying the orchestration of systems originates in chain-of-thought prompting. For replicability, we understand chain-of-thought prompting for code generation as a single functional capability of analytical composition and execution, while it could also be interpreted as inherently reactive, where the chain-of-thought interleaves analytical composition with the interpretation of self-induced observations. 17 3.2.1 Fixed Orchestration Fixed orchestration follows a topology of capability invocations that is determined at design time. The parameterization, for instance which retrieval query to employ or what code to generate, is determined at execution time. The simplest form of fixed orchestration is following a single sequence of capabilities that is applied identically to every instance. Beyond this, fixed orchestration may also employ conditionals and branching with multiple predetermined paths. This strategy resolves mutual dependencies between functional capabilities by assumption as the designer commits to a fixed ordering based on expectations about how insight needs are typically structured and processed, assuming that this ordering is adequate for the range of instances the system will encounter. A fixed topology provides a well-defined context for every invocation of a functional capability, since the in- puts to each capability are determined by its predecessors and its outputs feed its successors. This bounded structure enables independent optimization of individual capabilities and makes a system’s behavior pre- dictable and easier to trace, which supports interpretability and simplifies fine-grained evaluation of individ- ual capabilities. It also lets the designer of a method encode domain knowledge about effective analytical workflows directly into the topology, drawing on established practices such as the data analysis pipelines common in data science (Martínez-Plumed et al., 2021), for instance by hard-wiring an analyst-inspired sequence of stages (Mao et al., 2024; Lai et al., 2026). These benefits come at the cost of adaptability. A fixed topology cannot adjust to insight needs that deviate from the assumed processing pattern, to unexpected data characteristics discovered during execution, or to errors that fall outside predefined recovery paths. Since the agent operates in an uncertain and only partially observable environment (Section 3.1), the assumptions underlying the fixed structure may be violated for any given instance. A strategy that performs all data retrieval before analytical composition, for example, cannot retrieve tables that become relevant only after initial analysis. The more diverse the insight needs and the more heterogeneous the corpus, the more likely these assumptions are to break. Many works mirror established retriever-reader (Chen et al., 2017) and Retrieval Augmented Generation (RAG) (Lewis et al., 2020) architectures, separating an upfront retrieval stage from a downstream data processing step (Lin et al., 2023b; Zhang et al., 2025d; Zou et al., 2026; Wang et al., 2025a), often extended with interpretation of the inputs as a first step (Kothyari et al., 2023; Chen et al., 2024a; Kong et al., 2023) and output synthesis as a last step (Kong et al., 2023; Nahid & Rafiei, 2024; Menon et al., 2025; Abhyankar et al., 2025). While most methods traverse these fixed stages one-by-one, other systems parallelize the analytical process by decomposing it and following a fixed structure for each sub-analysis (Zhang et al., 2025c; Xu et al., 2025), resulting in a more complex, fixed topology of functional capabilities. 3.2.2 Plan-Based Orchestration In plan-based orchestration, the topology of capabilities is determined at the invocation of the agent, which generates an instance-specific topology before starting the execution. In contrast to fixed orchestration, the agent constructs a plan that specifies which capabilities to invoke, in what order, and with what dependencies based on the expressed insight need ˆ i. Execution follows this plan, resembling a fixed topology, which may include specified branches and error recovery mechanisms. This strategy resolves mutual dependencies between functional capabilities by anticipation, as the agent seeks to factor in the likely requirements of the insight need and the expected behavior of its capabilities to produce a topology that addresses the anticipated dependencies upfront. Since each plan is constructed for a specific insight need, it provides an end-to-end perspective that can account for instance-specific requirements for the analytical process before any operation is executed. This perspective lets the agent assess a plan’s coherence, identify potential bottlenecks, and optimize the structure, for instance by expressing independent steps as a dependency graph that can be executed in parallel (Zhang et al., 2025a) or by generating and evaluating alternative candidate plans against estimated quality crite- ria (Wang & Li, 2025). As an explicit artifact, the plan can also be verified before execution (Khoja et al., 2025), surfaced to users for intervention, and used to support interpretability. 18 The effectiveness of this strategy relies on the quality of the plan, which is generated without execution feedback. The plan rests on anticipations that may not hold, as the required data may have an unexpected structure, an analytical method may prove inappropriate once applied to the actual data, or intermediate results may reveal that the insight need calls for a different approach than planned. Recovery from such deviations and from errors is possible only to the extent that it was anticipated and encoded in the plan. The two systems with a plan-based orchestration differ in how expressive their plans are. Weaver (Khoja et al., 2025) generates a linear plan of interleaved SQL and language model steps that a second model verifies and refines before the plan is executed step by step. AixelAsk (Zhang et al., 2025a) instead constructs a directed acyclic graph of retrieval and reasoning nodes with explicit dependencies, which exposes independent branches for parallel execution. Beyond these, AOP (Wang & Li, 2025) illustrates the optimization potential of an explicit plan, compiling operators into a graph that enables parallelization and prefetching. 3.2.3 Reactive Orchestration Agents with a reactive orchestration strategy select each subsequent action based on their current obser- vations, which can include the partial realization constructed so far, the outcomes of previous actions, and the expressed insight need at that time. The process structure emerges from the accumulation of these step-by-step decisions. This strategy resolves mutual dependencies by reaction as they manifest. When the analysis reveals a data gap, the agent can invoke data retrieval, and when retrieval returns unexpected data, it can adjust its analytical approach without being locked into a plan that assumed otherwise. Reactive orchestration leverages observations of the uncertain environment that OpenTI agents operate in. By conditioning each decision on the observation history, the agent responds to the actual state of the analysis rather than to assumptions or anticipations that may no longer hold, which gives it inherent capacity for recovery and adaptation, as seen in agents that condition continued execution on observed errors to debug and retry their own operations (Hu et al., 2024; Qiao et al., 2026; Zhang et al., 2024b). This adaptivity foregoes an inherent end-to-end perspective on the process. Purely step-by-step decision-making risks pursuing locally reasonable actions that are globally suboptimal, for example selecting data that is adequate in isolation but ill-suited to operations performed later in the process, mirroring the principle from planning under uncertainty that optimal behavior requires accounting for long-range consequences (Kaelbling et al., 1998). How well a reactive agent avoids such trajectories depends on how much anticipation its underlying mechanism brings to each decision, which some systems reinforce by adding an explicit planning or critic component to the reactive loop (Zhou et al., 2025; Jin et al., 2025). In the reviewed works, reactive orchestration rests exclusively on a language model backbone, with most systems following the paradigm of interleaved reasoning and acting introduced by ReAct (Yao et al., 2023; Zhang et al., 2024b; Huang et al., 2024; Hu et al., 2024; Jin et al., 2025; Qiao et al., 2026). A variation is to distribute the process across multiple language model agents, where a planning agent reactively invokes further agents that embody functional capabilities (Zhou et al., 2025). 3.2.4 Hybrid Orchestration Hybrid orchestration combines other orchestration strategies, typically by applying different strategies at different levels of granularity or different phases of the analytical process. The general aim is to use the strengths of the combined strategies so that they counteract their respective shortcomings. Many such combinations are conceivable, and the reviewed systems instantiate three forms that differ in which strategy governs the macro structure and which governs the steps nested within it. One hybridization is to embed a reactive loop within an otherwise fixed multi-step pipeline, combining an informed macro structure with local adaptivity inside each step. In TableRAG (Chen et al., 2024b), a fixed sequence of query expansion and retrieval provides the inputs to a reactive program-aided solver that iterates over generated code and its observed outputs. TableGPT2 (Su et al., 2024) likewise wraps a reactive code and tool-calling agent inside a fixed pipeline of input preparation, agent execution, and answer generation. The intuition behind this combination is to bound the overall process with a reliable skeleton while letting the least predictable stage, the analytical composition, adapt to intermediate results. 19 Similarly, embedding reactive behavior in an instance-specific plan allows treating the plan as the primary strategy and reactive orchestration as a fallback, so the system retains the global coherence and efficiency of an optimized plan while avoiding the rigidity that would otherwise make a plan fail once observations deviate from what was anticipated in the plan. AOP (Wang & Li, 2025) compiles operators into an optimized graph that supports parallelization and prefetching, then executes it as planned and halts to re-plan the remaining pipeline when an operator fails or returns unsatisfactory results. Nesting a plan inside a reactive loop that governs the macro structure inverts this combination. Instead of committing to a plan, such systems treat plans as provisional and reactively decide when to extend or revise them. TaPERA (Zhao et al., 2024c) iteratively builds and refines a plan based on sub-questions, deriving sub-answers and deciding whether to finalize or refine the plan from them, while AutoTQA (Zhu et al., 2024) uses a critic that judges intermediate results and directs the planner to revise it whenever it detects gaps. 3.3 Functional Capabilities The functional capabilities of OpenTI systems describe the means with which they need to be equipped to derive a realization. We derive the necessary functional capabilities from the problem formulation (Sec- tion 2.1.5), which requires constructing a realization r = (O,E,β) ∈ R valid (i)∩ R feasible (T ) for an insight need i over a corpus of tables T. Determining which realizations are valid, that is, defining R valid (i), requires mapping the expressed insight need ˆ i to a system-side representation ˆ i s that constrains which realizations satisfy i, which in turn requires specifying and disambiguating its data, methodological, and output aspects i d ,i m ,i o . We summarize this capability as interpretation (Section 3.3.1). Identifying feasible realizations R feasible (T ) and constructing data bindings β requires exploring T to discover and bind relevant tables, a capability we refer to as data retrieval (Section 3.3.2). Beyond this, OpenTI agents compose the structure of operations (O,E) by selecting, parameterizing, and combining the analytical operations that derive knowl- edge from the data, which we capture as analytical composition and execution (Section 3.3.3). Finally, the agent produces outputs that satisfy the insight need. While formally part of (O,E), producing this output addresses qualitatively distinct concerns like selecting output modalities, contextualization, and the form of presenting the analytical knowledge, which correspond to the output need i o rather than the analytical derivation itself. We thus distinguish output synthesis (Section 3.3.4) as a separate functional capability. In combination, these functional capabilities cover all necessary aspects of constructing a realization and are thus jointly sufficient for constructing some r ∈ R cand . However, OpenTI seeks a realization that maximizes the utility r ∗ = arg max r∈R cand u i (r) in a combinatorially vast space. The necessary functional capabilities introduced above all constitute object-level computation (Davis & Buchanan, 1977; Davis, 1980), directly producing components that constitute r. They do not explicitly cover navigating the realization space towards high-utility realizations, which requires meta-level capabilities (Davis & Buchanan, 1977; Davis, 1980; Russell & Wefald, 1991). Meta-level capabilities concern reasoning about the process of constructing r itself, aiming to work towards the optimization objective but leaving no direct trace in r. The process of constructing a realization can diverge substantially from the clean DAG of the resulting realization. The execution trace of an agent may include cycles (e.g., retrying after errors), branches (e.g., exploring alternative analytical strategies), dead ends (e.g., retrieval returning insufficient data), and back- tracking (e.g., revising a strategy based on intermediate results). Meta-level capabilities let the agent navigate this process by assessing whether intermediate results are on course towards the insight need, validating the quality of partial realizations, detecting and recovering from errors, and deciding when to backtrack or pur- sue alternatives. These capabilities shape which realization gets constructed without themselves appearing as nodes in r, which parallels the distinction between reasoning and action in LLM agent frameworks (Yao et al., 2023). While the orchestration strategy (Section 3.2) provides the structural pattern within which the agent operates, meta-level capabilities are the functional means the agent exercises within that structure to navigate toward high-utility realizations. We examine these under process governance (Section 3.3.5). Table 1 provides an overview of systems with a broad coverage of functional capabilities (selection criteria in Appendix C.3). Our discussion of functional capabilities is based on the full set of 58 systems, extended with capability-specific literature. Few systems cover all functional capabilities. Most works lack retrieval capabilities entirely (41 of 58), operating on pre-specified data rather than discovering relevant tables from 20 a corpus. Interpretation often remains implicit (18 of 58), relying on inherent interpretation capabilities of language models rather than explicit mechanisms. Output modalities are largely limited to text or code execution results (46 of 58), with few systems capable of producing tables (15 of 58) or visualizations (6 of 58), despite the diversity of analytical results that may benefit from such presentation. 3.3.1 Interpretation Interpretation allows agents to determine which realizations satisfy the insight need, that is, which realiza- tions are part of R valid (i) by mapping the expressed insight need ˆ i to the system’s representation ˆ i s . The latency of insight needs introduces a gap between ˆ i and i. Users leave aspects implicit because they employ conventions, delegate choices to the system, or lack awareness of what must be specified (Section 4). In- terpretation bridges this gap by specifying implicit aspects and resolving ambiguities. All systems perform some interpretation before acting, yet many of the reviewed systems lack explicit mechanisms and instead do so implicitly in a neural model, mostly through the language understanding of an underlying language model. Interpretation may also recur during insight extraction, triggered when data retrieval, analytical execution, or further user inputs reveal misalignment. Input Decomposition and Augmentation. Decomposing and augmenting inputs seeks to transform and specify ˆ i s to better serve downstream capabilities. Decomposition breaks the expressed need into parts. Augmentation enriches or rewrites it to increase its specification or alignment with the data environment. The data need i d is often augmented by mapping inputs to data-level representations, for instance by gen- erating hypothetical schemas of the required data (Kothyari et al., 2023; Zhang et al., 2025d) or extracting concept-attribute pairs (Chen et al., 2024a;b; Wu et al., 2025c) that serve retrieval and analytical execution, and, after retrieval, by schema linking that aligns concepts mentioned in the expressed need with concrete schema elements (Pourreza & Rafiei, 2023; Qin et al., 2025; Shao et al., 2025), a technique originating in text-to-SQL systems (Liu et al., 2025a). In contrast, the methodological need i m is augmented by trans- forming inputs into an actionable plan for the analytical process (Zhao et al., 2024c; Lai et al., 2026; Lu et al., 2025b; Zhu et al., 2024; Zhou et al., 2025; Zhang et al., 2025a), bridging the inputs with retrieval and analytical execution. Beyond these input-level transformations, Zhang et al. (2025c) apply step-back augmentation (Zheng et al., 2024b), abstracting the expressed need to a higher level of generality before deriving specific sub-needs to surface implicit aspects that a direct transformation would miss. Reflecting the compositional nature of insight needs, decomposition is commonly employed to extract logi- cally dependent (Zhao et al., 2024c; Lai et al., 2026; Zhu et al., 2024; Pourreza & Rafiei, 2023; Abhyankar et al., 2025; Jiang et al., 2025) and independent (Xu et al., 2025; Zhang et al., 2025c) sub-needs from the expressed insight need, making the downstream analysis less complex to compose and execute. Decomposi- tion and augmentation are the most widespread explicit interpretation mechanisms, listed as “Decompose” and “Augment” in the Interpretation column of Table 1. Contextualization. Contextualization aims to inform the interpretation of the insight need with knowl- edge that is not directly expressed in the input. This contextual knowledge can originate from different sources. Interaction-histories can serve as shared context, resolving references to previous results and inter- preting follow-up needs in the light of prior results (Li et al., 2025b; Kochedykov et al., 2023; Xiao et al., 2022). Similarly, demonstration-retrieval can contextualize the analytical process in prior instances of pro- cessed insight needs (Lee et al., 2025; Zhang et al., 2023a; Pourreza et al., 2025; Chen et al., 2025f; Gu et al., 2025) or historical planning paths (Xu et al., 2025). Additionally, domain-knowledge is used to resolve terminology (Zhu et al., 2025a) and conventions specific to the analytical domain (Su et al., 2024), such as standard operating procedures from business operations (Zhu et al., 2024), directly addressing cases where users omit detail because a default interpretation exists in the domain. Systems integrate such external expert knowledge through different mechanisms, retrieving it from an index via retrieval augmented gener- ation (Zhao et al., 2024b), integrating a dedicated data-expert model that supplies knowledge left implicit in the schema and the expressed need (Hong et al., 2024), or materializing it as supplementary data that extends the retrieved tables with the missing information (Liu et al., 2025b). 21 Table 1: Functional capabilities of the reviewed systems. Columns correspond to the five functional capabilities of Section 3.3, with Analysis abbreviating Analytical Composition and Execution and Output abbreviating Output Synthesis. Under Retrieval , 1 (single table) indicate that no retrieval is performed, indicates that a limited set of relevant and irrelevant tables is provided (e.g., a small database), requiring selection of necessary tables, and indicates an open setting, requiring retrieval. Under Analysis , 3 denotes general-purpose code, SQL domain-specific code (e.g., SQL), å tool calling, and Æ direct model inference. Under Output , denotes text, O tables, ¡ visualizations, Ú contextualization, and ¶ modality selection. Under Process Governance , Error Handling abbreviates Error Detection and Recovery and Validation abbreviates Result Validation (Section 3.3.5).System/Name Interpretation Retrieval Analysis Output Process Governance Dater 2023 Decompose, Augment, Scope 1 SQL Æ Ensembling ITR 2023a Scope 1 SQL Æ Validation, Ensembling ROUTE 2025 Augment SQL Error Handling, Validation ReAcTable 2024b Decompose 1 3 SQL O Error Handling, Ensembling TableRAG 2024b Augment, Scope 1 3 Error Handling, Ensembling FISQL 2025 Augment, Clarify SQL O Ú H-STAR 2025 Decompose, Scope 1 SQL Æ Validation, Ensembling SAFE-SQL 2025 Augment, Scope, Context SQL Validation TabDSR 2025 Decompose 1 3 Error Handling, Validation TALON 2025 Scope 1 3 å Error Handling, Adaptive Steering ALTER 2025c Decompose, Augment, Scope 1 SQL Æ Ensembling TART 2025b Augment, Scope 1 3 Ú Error Handling TabSQLify 2024 Scope 1 SQL Æ Error Handling, Validation DAgent 2025 Decompose, Context Embedding SQL Æ Ú Validation DS-GURU 2026 Augment LLM-select 3 Error Handling, Validation MACT 2025 Decompose, Augment 1 3 å Error Handling, Ensembling Pi-SQL 2025 Decompose, Scope 3 SQL Error Handling, Validation, Ensembling QCMA-SQL 2025 Decompose, Augment, Context SQL Error Handling Weaver 2025 Decompose, Augment, Scope 1 SQL Æ Error Handling, Adaptive Steering RTS++ 2026 Scope, Clarify SQL Validation, Ensembling AutoTQA 2024 Decompose, Augment, Context LLM-select 3 SQL O Ú Error Handling, Validation, Adaptive Steering TaPERA 2024c Augment 1 3 Ú Error Handling, Adaptive Steering ACR 2025b Augment, Clarify, Context 1 3 O ¡ Ú Error Handling SGAM 2025c Decompose, Augment Embedding SQL Æ DataQue 2023 Augment, Clarify, Context Classifier SQL å O ¡ Ú ¶ Error Handling, Validation OpenTab 2023 Scope Lexical SQL Æ Error Handling, Validation TableGPT2 2024 Scope, Clarify, Context Embedding 3 SQL å O ¡ Ú ¶ Error Handling, Adaptive Steering AOP 2025 Decompose, Augment Embedding SQL å Æ Ú Error Handling, Validation, Adaptive Steering, Ensembling TiInsight 2025a Decompose, Augment, Scope, Context Embedding SQL O ¡ ¶ Error Handling, Validation 22 Scoping. Part of interpreting the data need i d is interpreting it against specific tables. Scoping aims to interpret a table and scope its contents to those that are actually relevant to i d by selecting only the relevant rows, columns, or cells. Approaches vary in granularity, from selecting relevant rows and columns based on semantic similarity to the expressed insight need (Zhang et al., 2025c; Kong et al., 2023; Patnaik et al., 2024), to retrieving individual schema elements and cell values (Chen et al., 2024b). Following a different approach, Gemmell & Dalton (2023) generate code on demand to filter rows programmatically before a downstream model computes the answer. The efficacy of an explicit scoping step is disputed, with Mao et al. (2024) finding a reduction in accuracy. Engaging Users. Beyond automatically resolving ambiguities, systems may involve users by soliciting clarification or presenting alternative interpretations. This is particularly relevant when the expressed need has multiple sensible interpretations that cannot be resolved through conventions or context alone (Saparina & Lapata, 2025a; Gomm et al., 2025). To address this, systems may engage users by asking targeted follow- up questions (Su et al., 2024; Li et al., 2025b; Menon et al., 2025; Chen et al., 2026; Kochedykov et al., 2023), or otherwise soliciting their input to specify the expressed insight need. Choosing when to autonomously handle specification and when to solicit clarification from the user introduces a trade-off between burdening the user with interactions and reducing the risk of misalignments. We assess this trade-off and the design of interactions in Section 4 in more detail. Interpretation Throughout Processing. The interpretation mechanisms in the reviewed works are pre- dominantly applied before the analytical process begins, operating on the expressed insight need in isolation from data and analytical outcomes that emerge during execution. Yet the environment is only partially observable, so information relevant to interpretation may surface only during processing, for example when a seemingly unambiguous entity matches multiple retrieved tables with different semantics. Nonetheless, explicit re-interpretation during processing, that is, revising ˆ i s in light of new observations, remains limited, except for iterative retrieval approaches (Zhang et al., 2025d; Boutaleb et al., 2025) that progressively re- fine the data need i d based on intermediate retrieval results, and iterative revisions of analytical plans that re-interpret the insight need under intermediate results (Zhao et al., 2024c). The degree to which re-interpretation is possible depends on the orchestration strategy. Fixed orchestration confines interpretation to designated positions in the topology, preventing revisions based on downstream ob- servations. Plan-based orchestration fixes interpretation at planning time, making re-interpretation possible only if explicitly anticipated. Reactive orchestration lets the agent condition each decision on prior observa- tions, though whether this constitutes re-interpretation depends on whether the agent’s representation ˆ i s is itself revised rather than merely adjusting execution within an unchanged interpretation. 3.3.2 Data Retrieval Data retrieval capabilities allow agents to explore the corpus and surface sufficient data to address an insight need, enabling the agent to determine which realizations the corpus supports, R feasible (T ), and to retrieve tables for data bindings β of candidate realizations. This requires mapping the data need i d to a set of tables for the data bindings β, and may additionally be constrained by the methodological need i m , for instance when an analytical method requires a minimum sample size to yield statistically valid results. When an insight need can be satisfied from a single table, retrieval concerns identifying the most relevant table. However, insight needs may require integrating information from multiple tables, requiring retrieval to surface a sufficient set (Definition 5) that satisfies both semantic alignment between the insight need and the retrieved tables and structural alignment among the retrieved tables themselves. The Retrieval column in Table 1 shows the data setting each system operates in, separating systems that operate over a pre-selected set of tables from those that select among a bounded set of tables and those that retrieve from an open corpus. Out of the systems in Table 1, only four target an open setting. We examine how existing work represents tables and inputs, the granularities at which retrieval is performed, how sets of multiple tables are composed, and how retrieval integrates within the broader analytical process. 23 Table and Input Representations. Index-based retrieval maintains an explicit collection of table repre- sentations and scores table relevance against an input at retrieval time. Bridging the semantic gap between input and table representations requires appropriate representations of both that make them comparable. Tables have predominantly been represented through linearization into text, which enables applying retrieval methods developed for text documents. The design space for linearization is large (Zhang et al., 2025e), ranging from schema-only representations using column headers (Kong et al., 2023) to full-content serial- izations including row-level data (Pan et al., 2022), representations enriched with surrounding contextual text (Chen et al., 2021; Huang et al., 2022), and LLM-generated natural language descriptions of table con- tents (Balaka et al., 2025; Zhang et al., 2025e). These linearization choices have been shown to significantly influence retrieval performance (Gomm & Hulsebos, 2025). However, linearizing tables into text has been criticized for ignoring semantic dependencies and structural information (Trabelsi et al., 2022; Wang et al., 2021), limiting the amount of tabular content that can be represented (Gomm & Hulsebos, 2025), and being unable to handle different cell datatypes individually (Wang et al., 2021). This has motivated table-specific encoding approaches that modify transformer attention mechanisms to account for tabular structure (Herzig et al., 2020; Trabelsi et al., 2022), apply graph-based encoders to capture structural properties (Wang et al., 2021), or operate on images of tables using multimodal language models (Xu et al., 2026). Despite these efforts, linearization to text with general-purpose encoders remains the dominant approach in research. Orthogonally, the semantic gap can be narrowed from the input side by transforming the expressed insight need into a form that better matches the table representations. One approach that has been explored is to transform inputs into text that is structurally similar to the linearizations of the indexed tables, for instance by generating a hypothetical table schema from the input and retrieving against the actual schemas (Kothyari et al., 2023; Zhang et al., 2025d), as well as query decomposition and expansion techniques discussed further in the context of semantic and structural alignment. More broadly, retrieval is often deeply intertwined with interpretation capabilities (Section 3.3.1) that expand, decompose, or rewrite inputs to improve alignment. Given these representations, the reviewed works apply established retrieval paradigms including lexical meth- ods (Kong et al., 2023; Ji et al., 2024), bi-encoder dense retrieval (Karpukhin et al., 2020; Pan et al., 2022; Zhang et al., 2025d; Kothyari et al., 2023; Chen et al., 2024a; Xu et al., 2025; Zhu et al., 2025a; Wang et al., 2025c), late interaction mechanisms (Khattab & Zaharia, 2020; Lin et al., 2023b), cross-encoder re- ranking (Luo et al., 2026), and hybrid combinations of lexical and dense signals (Balaka et al., 2025; Li et al., 2025a). For systems that perform retrieval, the Retrieval column in Table 1 annotates their retrieval mechanism. Retrieval mechanisms are applied largely independently of the representations of tables and inputs. Rather than which retrieval paradigm to apply, research has mostly focused on what representations effectively bridge the semantic gap and how to incorporate sufficiency criteria into retrieval. Retrieval Granularity. Besides representation and retrieval mechanisms, approaches vary in the gran- ularity at which tables are indexed. Most index at the table level, producing one representation per ta- ble (Kong et al., 2023; Herzig et al., 2021; Balaka et al., 2025), which limits the detail that can be captured but preserves cross-column semantics that convey what a table is about as a whole. In contrast, column- level indexing (Kothyari et al., 2023; Chen et al., 2024a) enables finer-grained matching between individual attributes and components of the data need, at the cost of a substantially larger index and the potential loss of inter-column context. This finer granularity is particularly relevant for composing sufficient sets of multiple tables, where alignment must be assessed per individual attribute rather than per table. Row-level indexing (Kumar et al., 2023; Ji et al., 2024) has also been explored, yielding fine-grained representations that capture specific records in the tables, facilitating retrieval in settings where relevance is concentrated in spe- cific records. Since these granularities capture complementary signals, approaches may also combine them, for instance, leveraging coarse-grained representations for initial filtering and fine-grained representations for precise matching (Chen et al., 2024a; Wang & Li, 2025; Kong et al., 2023; Zhu et al., 2025a). Semantic and Structural Alignment. When the data required to satisfy an insight need spans mul- tiple tables, retrieving individually relevant tables does not guarantee a sufficient set (Definition 5). The retrieved set must be semantically aligned with the data need, collectively covering all required concepts, and structurally aligned among its tables, enabling their integration through operations available to the 24 system (Section 2.2.2). Composing such sets can be understood as a search over candidate table combina- tions (Boutaleb et al., 2025), where candidate sets are assessed along both alignment dimensions. Approaches differ in how they track semantic coverage, assess structural compatibility, and navigate the search space. Semantic alignment is primarily addressed by decomposing the data need into sub-components and matching which of them are covered by retrieved tables. Kothyari et al. (2023) generate a hypothetical minimal set of table schemas from the insight need, while Chen et al. (2024a) decompose it into concept-attribute pairs, each targeting a specific data requirement. Both decompose in a single step without considering which tables are actually available, then track coverage during a subsequent selection phase. These contrast with iterative retrieval, where retrieval steps are interleaved with an assessment of the alignment of the tables retrieved so far (Boutaleb et al., 2025). Zhang et al. (2025d) retrieve one table per iteration and rewrite the input to remove aspects already covered, continuing until all concepts in the data need are covered. The iterative structure allows adapting to the actual structure of the data and implicitly determining the size of the sufficient set, at the cost of computational overhead that scales with the number of iterations. Chen et al. (2025d) instead align the input with the organization of the corpus by exploring relationships among its tables, retrieving a sufficient set in a single pass rather than decomposing the need without regard to available data or querying iteratively. Structural alignment requires assessing whether retrieved tables can be integrated, for instance through join or union operations. Approaches estimate structural compatibility from signals intrinsic to table pairs, derived from an offline schema graph (Kothyari et al., 2023), from online value overlap such as Jaccard similarity over columns (Chen et al., 2024a), or from a search for join paths that connect the matched attributes (Wang et al., 2025c). These signals can then be combined with semantic coverage into a joint selection objective (Chen et al., 2024a; Kothyari et al., 2023), or folded directly into the per-step selection of iterative retrieval, avoiding a separate optimization phase (Boutaleb et al., 2025). These column relationships are also applied to expand retrieved sets of tables with joinable ones to include tables that have a low semantic alignment (Agarwal et al., 2025). Across these approaches, compatibility is judged on properties intrinsic to table pairs, such as key relationships or value overlap. Yet structural compatibility is ultimately determined by the agent’s capabilities for analytical composition and execution, since capable agents may construct integration paths through transformations such as schema matching or feature engineering. The assessment of structural compatibility should thus depend on the capabilities of the agent. Zou et al. (2026) take a different approach. Instead of a regular index, they construct a hypergraph of tables based on semantic, structural, and heuristic similarity features. They then apply clustering and PageRank to surface groups of related tables. Unlike the approaches above, which compose sets that align with the specific data need, this approach captures general table relatedness within the corpus. It can surface structurally similar tables but does not explicitly optimize for semantic coverage and structural compatibility. Notably, none of the discussed approaches actively handles cases in which no sufficient set of tables exists within the corpus for the given data need. When R cand = ∅, most retrieval methods would still return a set of tables, leaving the insufficiency undetected. Generative Retrieval. Generative retrieval presents an alternative to maintaining an explicit index by encoding the corpus into the parameters of a sequence-to-sequence model that directly generates identifiers of relevant tables given an input (Tay et al., 2022), unifying indexing and search into a single differentiable process. Guo et al. (2025) apply this to individual tables, assigning each a semantic identifier derived from hierarchical clustering of metadata and cell values, while Wang et al. (2025a) extend it to multi-table settings by serializing a schema graph into the generation target and constraining decoding to valid connected table sets. Both train on synthetic queries generated over the target schemas. Since generative retrieval learns the index directly it can capture semantic and structural properties jointly, which, depending on the training, enables deeper input-table interaction than bi-encoders that encode queries and tables independently (Guo et al., 2025). Its effectiveness is limited by the quality of the synthetic training data, which may propagate hallucinations or distributional biases into retrieval errors (Wang et al., 2025a). Additionally, since the index is learned, corpus changes require new or additional training, with parameter- isolation strategies for new tables explored but not validated at scale (Guo et al., 2025). 25 Dynamic Retrieval. Most OpenTI systems treat retrieval as a separate phase preceding the analytical process, constructing the complete data bindings β upfront. This assumes the data need i d can be fully determined from the initial input. Yet intermediate results may reveal requirements for additional data, such as linking tables for data integration, reference data for normalization, or variables whose relevance becomes apparent only after initial analysis. Furthermore, treating retrieval as a decoupled step makes recovery from retrieval misses difficult. Works on text documents have thus explored conditioning retrieval on the processing state (Jiang et al., 2023; Asai et al., 2024). In the tabular context, iterative retrieval methods (Zhang et al., 2025d; Boutaleb et al., 2025) interleave retrieval and interpretation capabilities, but still operate within a dedicated retrieval phase. Wang & Li (2025) take an initial step towards integrating retrieval with other capabilities by treating it as an operator that is invocable during execution, though without systematic methods for leveraging the intermediate analytical state to inform retrieval decisions. 3.3.3 Analytical Composition and Execution Analytical composition and execution capabilities allow OpenTI agents to construct, parameterize, and execute the operational structure (O,E) of a realization. They thus enable OpenTI agents to derive the analytical knowledge to satisfy the user’s insight need. We describe them based on the mechanisms that are employed to specify and execute operations, the analytical composition, that is, the manner in which operations are composed, and the role of data preparation in this process. Mechanisms. The mechanisms through which an agent executes operations determine what operations it can express and, therefore, which insight needs the agent can address. OpenTI systems may employ code generation in domain-specific and general-purpose languages, tool calling over defined functionalities, and direct inference techniques as mechanisms for executing operations. The Analysis column of Table 1 shows which of these mechanisms each system employs, showing that most systems rely on domain-specific and general-purpose code, whereas fewer systems make use of direct inference and tool calling. Domain-Specific Language (DSL) generation, mostly studied in the form of SQL generation, is the most extensively researched mechanism for deriving knowledge from tabular data, building on a substantial body of research into text-to-SQL generation (Liu et al., 2025a). Several systems apply SQL generation as their sole analytical mechanism (Pourreza & Rafiei, 2023; Lee et al., 2025; Menon et al., 2025; Xu et al., 2025; Chen et al., 2026), while others combine it with other mechanisms (Zhang et al., 2025c; Xu et al., 2025; Kong et al., 2023; Wang & Li, 2025; Zhang et al., 2024b; Huang et al., 2024; Zhu et al., 2024; Su et al., 2024). DSLs like SQL are designed for a particular application domain. In SQL, this specialization provides formal semantics grounded in relational algebra (Codd, 1970), enabling a well-defined execution behavior and a constrained output space. However, the specialization of DSLs bounds what operations can be expressed. SQL has limited capacity to express statistical modeling, machine learning, optimization, or visualization operations (Zhao et al., 2024c; Kong et al., 2023), confining systems that rely on it as their sole execution mechanism largely to descriptive insight needs, with inferential and causal needs falling outside its expressive scope. Systems that employ SQL generation alongside other mechanisms typically delegate operations outside the scope of SQL to complementary mechanisms, such as direct inference for result synthesis (Zhang et al., 2025c; Xu et al., 2025; Kong et al., 2023) or general-purpose code for data transformations (Zhang et al., 2024b; Zhu et al., 2024; Huang et al., 2024; Su et al., 2024). General-Purpose Language (GPL) generation, mostly studied in the form of Python generation, generates code in a general-purpose language to interact with data (Gao et al., 2023). While GPLs can express arbi- trary computations, making them theoretically capable of addressing the full spectrum of insight needs, they provide less domain-specific scaffolding than DSLs, resulting in a less constrained output space and a greater range of potential errors. In OpenTI, the extensive ecosystem of Python libraries for data manipulation, statistical analysis, machine learning, and visualization (Zhang et al., 2024b; Chen et al., 2025f) supplies this scaffolding within a GPL, letting systems leverage these abstractions while retaining the expressiveness to address arbitrary operations. Many of the reviewed works therefore employ Python generation as a pri- mary (Zhao et al., 2024c; Mao et al., 2024; Hu et al., 2024; Chen et al., 2024b; 2025f; Lai et al., 2026; Li et al., 2025b) or complementary mechanism (Zhang et al., 2024b; Zhu et al., 2024; Huang et al., 2024). Despite the theoretical breadth, these systems mostly demonstrate their Python code generation on descriptive needs 26 that largely overlap with tasks also addressable through SQL, with only a few extending to inferential needs such as statistical inference and prediction (Hu et al., 2024; Huang et al., 2024; Qiao et al., 2026; Li et al., 2025b), leaving the practical effectiveness for more complex analytical intents largely undemonstrated. More- over, the unconstrained output space has motivated the application of safety measures like containerization for executing operations (Hu et al., 2024; Huang et al., 2024; Su et al., 2024). Tool calling employs functions with specified semantics that the agent selects and parameterizes at execution time. Unlike code generation, where the operation is specified in full by the generated code, tool calling constrains the agent to invoking specific functions from a limited set. Abstracting operations into predefined callable functions allows tighter design-time control over the operations applied to the data, at the cost of limiting what an agent can express, making the expressiveness of tool-calling systems mostly dependent on the scope of the tool set. Methods vary in the number and complexity of tools they have access to, from small sets of simplistic table operations like select rows and group by (Wang et al., 2024b; Jin et al., 2025), to larger sets of more expressive operations (Wang & Li, 2025; Kochedykov et al., 2023). Automatically generating missing tools for specific inputs (Gemmell & Dalton, 2023; Lu et al., 2025b) has been proposed to rectify the inflexibility of predefined tools. Direct inference employs models directly to perform operations over tabular data and intermediate results. Unlike code generation and tool calling, where operations are explicitly specified and executed by an external interpreter, direct inference performs computations within a model itself, making the operation implicit in its learned behavior. The expressiveness of this mechanism depends on the capabilities of the employed models. In principle, a sufficiently capable model could map directly from raw data to analytical knowledge in a single step, subsuming all intermediate operations. In practice, systems apply direct inference at varying levels of granularity, ranging from atomic operations such as cell selection or fact verification (Wang et al., 2024b; Zou et al., 2026; Lin et al., 2023b) to synthesizing free-form responses from intermediate results produced by other mechanisms (Kong et al., 2023; Zhang et al., 2025c; Xu et al., 2025). In direct inference, operations are not externalizable, meaning no executable specification can be inspected, verified, or reused independently of the model, limiting the interpretability of how analytical knowledge is derived. Moreover, the reliability of computational operations such as arithmetic, aggregation, or statistical computation relies on the model’s precision rather than formal execution semantics. Studies into the application of language models for numerical computations show degradations (Wu et al., 2025b; Wolff & Hulsebos, 2025; Zhou et al., 2024), particularly as input size increases (Zou et al., 2026). Most reviewed works thus combine direct inference with other mechanisms, relegating direct inference to higher-level functions such as aggregating results across sub-needs (Zhang et al., 2025c; Wang & Li, 2025), synthesizing free-form responses from execution outputs (Kong et al., 2023; Xu et al., 2025), or performing simple extraction tasks that do not require precise computation (Zhang et al., 2024a; Zou et al., 2026; Lin et al., 2023b). Analytical Composition. Analytical composition concerns how the analytical logic (O,E) is assembled, that is, the manner in which operations O are selected, connected and parameterized. Analytical compo- sition is coupled with the orchestration strategy, which determines when and how often the agent invokes analytical composition, constraining the possible composition patterns. In plan-based orchestration, the operational structure may be largely determined during plan generation, integrating composition directly into orchestration (Wang & Li, 2025). In reactive orchestration, composition decisions are distributed across successive actions during execution, making the capability more clearly distinguishable. Approaches mostly differ in how much of the analysis they specify at a time. In one-shot composition, agents specify the analytical logic in a single step, for instance a full SQL query (Kothyari et al., 2023; Zhang et al., 2025d; Wang et al., 2025a; Chen et al., 2024a) or a complete Python script (Chen et al., 2025f; Lai et al., 2026), committing to the operational structure without observing intermediate states. In incremental composition, the operational structure is built over successive actions, each producing an intermediate result the agent can observe before determining the next operation (Zhang et al., 2024b; Mao et al., 2024; Wang et al., 2024b; Zhou et al., 2025; Huang et al., 2024; Chen et al., 2024b; Zhu et al., 2024; Hu et al., 2024), allowing it to detect unexpected results, adjust the analytical approach, or refine the scope of operations based on the data. Works report improved accuracy over one-shot generation (Zhang et al., 2024b; Wang et al., 2024b), contingent on the agent’s ability to effectively use intermediate state information. When 27 insight needs are decomposed into sub-needs, each sub-need can be treated individually and integrated with the others downstream (Zhao et al., 2024c; Zhang et al., 2025c; Xu et al., 2025; Wang & Li, 2025), which also lets independent sub-needs be processed in parallel (Wang & Li, 2025; Zhang et al., 2025c). Execution mechanisms and composition patterns are largely coupled, though not rigidly. The declarative nature of DSLs favors one-shot composition, as a single query typically encodes the full analytical logic, yet DSL generation is also used incrementally (Zhang et al., 2024b; Mao et al., 2024; Chen et al., 2024b). Tool calling is inherently incremental, since each call is a separate action with observable intermediate state (Wang et al., 2024b; Zhou et al., 2025), whereas general-purpose code generation spans both one-shot (Chen et al., 2025f; Lai et al., 2026) and incremental settings and allows arbitrarily sized actions. Data Preparation. Data preparation aims to select, clean, and transform retrieved data into a form that enables the analytical operations that generate the analytical knowledge. It forms a substantial component of manual analytical workflows (Martínez-Plumed et al., 2021), addressing the selection of relevant data, data quality issues such as inconsistent formatting, missing values, and structural irregularities (Rahm & Do, 2000), as well as structural mismatches between data and the requirements of intended operations (Wickham, 2014). Insufficient data preparation has been identified as a significant contributor to failures of systems operating in open environments with messy, real-world data (Lai et al., 2026). Nonetheless, it has seen limited coverage in the reviewed works. Several systems integrate specific data cleaning stages (Mao et al., 2024; Zhang et al., 2025c; Su et al., 2024), sanitizing data before applying analytical operations. Beyond these, other approaches may apply ad-hoc cleaning, such as type coercions and string normalization (Chen et al., 2024b; 2025f; Zhang et al., 2024b). Departing from these programmatic approaches, Lu et al. (2025b) explore fine-tuning a language model to perform data cleaning as a sequence-to-sequence task. 3.3.4 Output Synthesis Output synthesis capabilities enable OpenTI agents to produce the final user-facing output from the ana- lytical knowledge derived in the realization. Formally, this corresponds to executing the sink node of the realization DAG to address the output need i o . While this is part of the operation structure (O,E), output synthesis addresses how analytical knowledge is communicated to the user rather than how it is computed. This distinction matters because the utility of a realization depends not only on the soundness of its analytical derivation but also on how effectively its results are presented (Munzner, 2014; Moritz et al., 2019). OpenTI agents perform output synthesis to different degrees. At the minimal end, a system returns solely the raw computational result (e.g., a retrieved or calculated value, a result table). At the other end, systems select appropriate modalities, integrate results from multiple analytical branches into a coherent response, and add interpretative framing that contextualizes the analytical knowledge. In practice, the output need i o is often underspecified or entirely absent from the input, as reflected in current benchmarks where the expected modality is typically specified in the evaluation protocol rather than in the input (Section 5.2). This means that systems need to compose a fitting presentation (Gomm et al., 2025; Mackinlay, 1986). The degree to which a system performs output synthesis thus determines to what extent it can make the analytical knowledge accessible to the user. Output Modalities. The output modality determines the form the analytical knowledge is presented in. Outputs may consist of natural language text, structured data such as tables, visualizations, and multimodal combinations thereof. As shown in the Output column of Table 1, the majority of the reviewed systems only output text. Within these, however, the outputs range considerably, from single discrete values concerning descriptive insight needs (Zhou et al., 2025; Mao et al., 2024; Kong et al., 2023), through short natural language responses that verbalize computational results (Zou et al., 2026; Wang et al., 2024b), to paragraph- length answers that narrate analytical findings (Zhao et al., 2024c) and long-form analytical reports that synthesize complex results (Xu et al., 2025). Systems that return raw execution results, whether that is SQL result tables (Zhang et al., 2025d; Kothyari et al., 2023; Chen et al., 2024a; Wang et al., 2025a) or code output values (Chen et al., 2025f; Lai et al., 2026), surface the computational result without verbalization or interpretation, requiring that users understand the analytical context to interpret the raw results directly. 28 Beyond text and tables, systems have explored visual outputs, either as a separate modality, selected based on the input and standing on its own (Huang et al., 2024; Li et al., 2025b; Zhu et al., 2025a; Kochedykov et al., 2023), or contextualized in a multimodal response (Su et al., 2024). Dedicated capabilities for generating visualizations from data and natural language descriptions exist (Yang et al., 2024) but have not been widely integrated into the reviewed OpenTI systems. Generally, different types of analytical knowledge benefit from different modalities. Retrieved facts and summary statistics, for example, are well served in text form, whereas trends and distributions may benefit from visualizations. Result Integration. If insight needs decompose into sub-needs, particularly independent sub-needs that are processed in parallel (Section 3.3.1), the output must integrate the results of these sub-needs into a coherent output that addresses the original need. Systems that handle result integration specifically employ LLMs to compose a unified output from the results of sub-needs (Zhang et al., 2025c; Xu et al., 2025; Wang & Li, 2025; Zhao et al., 2024c). Modality Selection. To best address an output need i o and maximize the utility, a OpenTI system must select an appropriate modality, or combination of modalities, for outputting the analytical knowledge. The appropriate modality depends on the nature of the analytical result, the insight need, and the user’s ability to interpret different forms of presentation (Munzner, 2014). However, in most reviewed systems, the selection is fixed to an evaluation-task-specific modality. In systems that support multiple modalities (Huang et al., 2024; Li et al., 2025b), the modality for a given instance is typically specified in the task instruction rather than determined by the system. A few systems do select the modality themselves. TableGPT2 (Su et al., 2024) dynamically selects it based on learned behavior in training, while TiInsight (Zhu et al., 2025a) and DataQue (Kochedykov et al., 2023) apply explicit rules and heuristics that map the result’s data characteris- tics to a chart type or fall back to a table. Adjacent works on visualizations have addressed modality selection through methods that map data characteristics and analytical tasks to effective visualizations (Mackinlay, 1986; Moritz et al., 2019; Mackinlay et al., 2007). However, among the reviewed systems, similar efforts have thus far not been undertaken. Contextualization. Recalling the definition of an insight as a collection of knowledge that links analytical knowledge with a person’s contextual understanding (Section 2.1.1), the output of an OpenTI system may integrate the presented analytical knowledge with contextual information to improve the insight the user gets. This may involve relating findings back to the insight need, noting limitations of the underlying data, qualifying the scope of the results, or explaining what the analytical knowledge means in context. Yet contextualization is largely absent from output synthesis in current systems. A few systems target it explicitly. For instance, DAgent (Xu et al., 2025) generates reports that include interpretations alongside analytical results, DACO (Wu et al., 2024b) is trained to translate quantitative findings into strategic, domain-specific recommendations, and AOP (Wang & Li, 2025) attaches generated reasoning to its results through a dedicated explanation operator. In most other systems, contextualization is incidental rather than a designed capability, as in TableGPT2 (Su et al., 2024), which occasionally notes data availability limitations. The absence of contextualization of outputs is also reflected in evaluation practice, which mainly assesses the correctness of final values and treats additional verbalization as noise rather than value (Section 5.3). 3.3.5 Process Governance Process governance capabilities operate at the meta-level (Davis & Buchanan, 1977; Davis, 1980), shaping which realization r gets constructed without directly contributing executable operations to it. This includes error detection and recovery, validating intermediate and final results, reflecting on and revising the analytical strategy, and determining when the process is complete. As shown in Table 1, many systems lack explicit process governance or only handle errors. Other systems employ a range of process governance techniques extending to result validation, adaptive steering, and ensembling. Error Detection and Recovery. Error detection and error recovery deal with failure states in the analytical process. This requires detecting failure states and exerting appropriate corrections, which may range from rigid fallbacks that follow predetermined alternatives to adaptive corrections informed by the 29 observations of the error. In the reviewed works, error detection is universally implemented as programmatic checks on code execution, using runtime exceptions from Python and SQL execution as a signal for failure states. In contrast, error recovery approaches follow different patterns. The most rigid form consists of applying predetermined fallbacks that are fixed at design time, such as cascading to executing simpler code (Kong et al., 2023), switching to an alternative tool (Zhou et al., 2025), or falling back to a different mode of generating the output (Lu et al., 2025b). More adaptive recovery feeds the error message back to a generating LLM to re-generate the failing operation. This pattern is rooted in self-debugging (Chen et al., 2023) and widely adopted in the reviewed works (Chen et al., 2025f; Lai et al., 2026; Mao et al., 2024; Huang et al., 2024; Hu et al., 2024; Li et al., 2025b; Su et al., 2024; Zhao et al., 2024c; Chen et al., 2024b; Jin et al., 2025; Jiang et al., 2025; Qin et al., 2025; Pourreza et al., 2025), typically bound by a fixed limit for retries (Chen et al., 2025f; Mao et al., 2024), or a global step budget in ReAct-based systems (Huang et al., 2024; Hu et al., 2024). In contrast, Zhang et al. (2024b) explore error recovery by retrying failed SQL queries arising in a ReAct step against intermediate tables from earlier steps. Result Validation. Result validation assesses whether intermediate or final results meet quality criteria beyond successful code execution, since code can run error-free yet yield results that are empty, irrelevant, or misaligned with the insight need. Validation differs in depth. At a surface level, systems verify that execution produces non-empty or structurally compliant outputs and trigger error recovery otherwise (Chen et al., 2025f; Lai et al., 2026). At a deeper level, systems assess whether a result is relevant to the insight need, either before execution by scoring or rewriting generated queries against the analytical requirements (Xu et al., 2025; Chen et al., 2026; Chi et al., 2025) or after execution by filtering out results obtained from irrelevant tables (Kong et al., 2023). At the process level, validation can assess whether accumulated results collectively address the insight need, as in AOP’s (Wang & Li, 2025) operators that search external knowledge for supporting or contradicting evidence and AutoTQA’s (Zhu et al., 2024) critic agent that compares results against the original need and triggers replanning where sub-needs remain unaddressed. Adaptive Steering. Adaptive steering covers capabilities to diagnose whether the analytical process is on track to yield the desired result and to adjust course where it is not. This mirrors the monitoring and replanning cycle in automated planning under uncertainty (Ghallab et al., 2004), where agents detect when observations invalidate the assumptions underlying their current strategy and adapt accordingly. Adapta- tions can range from assessments and alterations within individual steps, to backtracking, the exploration of different analytical paths, and, most comprehensively, to strategic revisions of the full analytical plan. Lan- guage model-based systems can perform the diagnostic component through self-reflection, reasoning about the analytical process as a whole as part of the generative process. In ReAct-based systems (Huang et al., 2024; Hu et al., 2024; Chen et al., 2024b; Su et al., 2024; Wang et al., 2024b), the interleaved reasoning steps provide an implicit form of reflection, where the agent reasons about its observations before selecting the next action. AOP (Wang & Li, 2025) handles self-reflection in a dedicated operator that generates justifications for intermediate results, making the agent’s reasoning explicit and available for subsequent in- tegration across analytical paths. For correcting the analytical course, some systems implement re-planning, adjusting an overarching analytical plan to the results of self-reflection. TaPERA (Zhao et al., 2024c) and AutoTQA (Zhu et al., 2024) perform a post-hoc analysis of the analytical process, informing re-planning of a further iteration of analysis. In contrast, AOP (Wang & Li, 2025) adapts its generated plan forward by evaluating intermediate results after each execution layer and pruning branches where operations have failed, re-invoking pipeline generation conditioned on the results gathered so far. Table-Critic (Yu et al., 2025) structures correction as a multi-agent loop in which a judge identifies errors in the reasoning chain, a critic suggests fixes, and a refiner revises the affected steps. Ensembling. Ensembling mitigates the variance inherent in stochastic processes, such as language model generation, by sampling multiple analytical trajectories and aggregating their results. A common approach is majority voting over independent runs of the full analytical process (Chen et al., 2024b; Zhang et al., 2024b; Sun et al., 2023), selecting the most frequent answer. Alternatively, a trained selector can be applied to select among candidates (Zhang et al., 2024a; Pourreza et al., 2025). Ensembling is also applied to the analytical operations themselves, as CHASE-SQL (Pourreza et al., 2025) selects between candidate SQL queries and AOP (Wang & Li, 2025) generates multiple candidate pipelines with justifications and has a 30 language model combine them by assessing the soundness of each analytical process. Instead of compar- ing outputs or complete realizations, Zhang et al. (2024b) explore branching at each analytical step and grouping intermediate steps that produce computationally equivalent results, finally selecting by aggregated log-probability of the respective output generation, grounding the selection in computational outcome rather than surface-level agreement. On a finer granularity, MACT (Zhou et al., 2025) samples multiple candidate actions per step and selects by majority vote. Governance Mechanisms. The mechanisms discussed above vary in how far meta-level governance ca- pabilities are separated from object-level analytical capabilities. Separating the component that assesses results from the one that produces them is a well-established principle, originating in actor-critic architec- tures in reinforcement learning (Konda & Tsitsiklis, 1999). For LLM-based systems, which dominate current OpenTI implementations, this separation is particularly relevant as self-correction by language models has been shown to be unreliable without external verification (Huang et al., 2023), though structured verification procedures can partially mitigate this (Wu et al., 2024c). The majority of the reviewed works use the same language model that performs the analytical work for process governance, differing only in prompting (Chen et al., 2025f; Lai et al., 2026; Mao et al., 2024; Zhao et al., 2024c; Huang et al., 2024; Hu et al., 2024). Some systems introduce a separation by instructing the same underlying language model for different roles, as in AutoTQA’s (Zhu et al., 2024) critic agent and AOP’s (Wang & Li, 2025) validation operators. In contrast, other systems use distinct models for a stronger separation, such as a pretrained cross-encoder for relevance assessment (Kong et al., 2023), a fine-tuned evaluator for query quality (Xu et al., 2025), or multiple agents in different roles that mutually validate one another (Zhou et al., 2025). Complementing these, programmatic checks for code execution errors and results provide deterministic governance signals (Chen et al., 2023). 4 Interaction and Interpretability in OpenTI The preceding section examined how systems orchestrate functional capabilities to derive insight realizations from tabular corpora. Yet the quality of any realization ultimately also depends on how accurately the system interprets what the user, be that a person or an agent, seeks in the first place. This section shifts the focus from the mechanics of insight extraction to the interface between users and systems, examining how insight needs are communicated, interpreted, and refined through interaction. Compared to other sections in this paper, the discussion here is more conceptual and draws extensively on adjacent work in human-computer interaction and information retrieval beyond the systems and benchmarks reviewed in Sections 3 and 5. Requiring users to fully specify all choices involved in selecting data (i d ), defining the analytical methodology (i m ), and presenting results (i o ) is both impractical and often impossible. Complex insight needs entail numerous interdependent choices that users cannot reasonably enumerate upfront, and doing so would place a prohibitive burden on interactions. More fundamentally, insight needs are inherently latent, as established in Section 2.1.2. The expressed need ˆ i is a compromised formulation of the latent need i, shaped by what users can articulate and believe the system can accommodate (Taylor, 1962; Belkin, 1980). The system consequently never has direct access to i, and can only construct an approximation ˆ i s from the expressed input and available contextual information. The fundamental challenge of designing user-system interactions is thus aligning the system’s representation of the insight need ˆ i s with the user’s latent insight need i. How well this gap can be closed depends on the mechanisms for interpreting and specifying inputs available to systems and their users. Instead of demanding exhaustive specification of the insight need i = (i d ,i m ,i o ), OpenTI systems should treat users as cooperative communicators and interpret their inputs accordingly. Grice (1975) posits that participants in cooperative communication provide sufficient but not excessive information (maxim of quantity) while remaining truthful (maxim of quality). In the context of insight extraction, this implies that users are intentional about which aspects of their insight needs they specify explicitly and which they leave implicit. When a user asks “What is the development of global renewable energy installations since 2010?”, they specify the core semantic concepts but leave the exact data to retrieve, the methodological implementation, and the output format unspecified. This underspecification is not nec- essarily a failure of articulation but can be interpreted as a delegation of authority (Clark & Brennan, 1991; Gomm et al., 2025), where the user relies on the system to infer reasonable interpretations for unspecified components, and the system accepts this delegation as its contribution to the grounding process. 31 Extending the argument of Gomm et al. (2025), whether and how a system can resolve a given instance of underspecification depends on the character of that instance. We propose to distinguish three ways of resolving underspecification, illustrated along the running example of a user asking “What is the average income in Paris?”: • Conventional resolution. Where strong conventions or domain defaults exist, the system can re- solve unspecified aspects autonomously by assuming the most reasonable interpretation. The entity “Paris” is conventionally interpreted as the capital of France rather than any of the towns bearing the same name in the US or elsewhere, and the system can proceed on this basis, potentially disclosing the interpretation to the user. • Selective resolution. Where the expressed need admits multiple valid realizations but the user has expressed no preference among them, the system may exercise agency in selecting one. In the same example, the boundaries of Paris are not uniquely determined by the input, as the city of Paris proper, the greater Paris metropolitan area, and the Île-de-France region are all defensible interpretations. The system may select one based on what data is available or what is most commonly intended, but such choices must be disclosed, since users cannot otherwise judge whether the resulting realization reflects their intent and may mistake a contingent analytical decision for an objective finding. • User engagement. Where neither convention nor reasonable selection is applicable, the system must engage the user directly. This can take one of two forms. First, straightforward underspecification, where the user has not provided enough information to differentiate valid from invalid realizations. A user asking “What is the average income?” without any geographic or demographic boundary illustrates this, as there exists no convention to select a default scope, no principled set of alternatives can be derived from the input alone, and the system must elicit the missing specification before proceeding. Second, unresolvable underspecification, where the user cannot specify because they do not know the methodological possibilities of the system or the data it has access to. In this case, no clarifying question can be meaningfully answered until the system first makes something about its data and capabilities visible to the user. This distinction parallels the obligations of user revealment and system revealment formulated by Radlinski & Craswell (2017), applied to OpenTI systems. User revealment requires the system to actively support users in articulating and refining their insight need over the course of interaction, through mechanisms that elicit, disambiguate, and extend the expressed need. System revealment requires the system to continuously make its data coverage, analytical capabilities, and the choices it has made sufficiently transparent that users can form accurate expectations, update their inputs, and assess whether the system’s contributions meet the grounding criterion. System revealment is not a one-time disclosure at the start of an interaction but an ongoing obligation that runs throughout the analytical process, as the system retrieves data, selects methods, and constructs realizations whose basis the user cannot otherwise inspect. The extent to which conventional and selective resolution can align ˆ i s with i depends on the contextual knowledge the system can draw on when interpreting ˆ i. Systems that do not adapt to specific users are limited to population-level conventions. Yet users bring distinct expertise, roles, and analytical perspectives (Zhang et al., 2020a), and they create a shared context with the system through continued interactions, where prior interactions serve as context for subsequent ones (Yin et al., 2023), so the same expressed need may warrant different interpretations and realizations depending on who expresses it and in what context. Personalization aims to address this by conditioning interpretation on contextual information such as user profiles and interaction histories, a mechanism with a long tradition in information retrieval (Teevan et al., 2005) that has more recently been applied to conditioning language model outputs on retrieved user context (Salemi et al., 2024b;a). The mechanisms for contextualization reviewed in Section 3.3.1 partially address this conditioning, yet they ground interpretation exclusively in the domain, the corpus, and prior interactions within a session, whereas persistent adaptation to the individual user remains unexplored in the reviewed systems. Successful contextualization enables autonomous resolution of inputs that would otherwise require user engagement, reducing the interaction burden. At the same time, adaptation that is neither visible nor controllable risks resolutions that are misaligned in ways users cannot detect (Jameson & Schwarzkopf, 2002), extending the disclosure obligations of system revealment to personalized interpretations. 32 This section explores the mechanisms through which user-system interactions support this dynamic. We characterize user-system interactions in OpenTI along the interaction mode (Section 4.1), which governs when and by whom input is provided during the insight extraction process, and the interaction means (Section 4.2), which describe the mechanisms through which input is conveyed. Together, interaction modes and means shape how insight needs are communicated, refined, and aligned between users and systems to enable appropriate realizations. Beyond these mechanisms, Section 4.3 examines interpretability, required for users to verify the system’s analytical derivations. 4.1 Interaction Modes Collaborative Result Proactive Guidance & Feedback User Input User Response Data Processing Data Processing Request Input while system requires more input System-Initiated Clarification Result User Input User Response Data Processing Data Processing Request Input while system requires more input Step-Wise User Input for Step Result Intermediate Sub-Result Data Processing for all analytical steps User Input Result Data Processing iterate until user is satisfied Output-Driven User Input Result Request Clarification User Response Data Processing while system requires clarification Iterative Elicitation Single-Input User Input Result Data Processing Interaction Modes Upfront Interaction User-Initiated- System-InitiatedSystem-Initiated Interaction Throughout Mixed-Initiative 55 systems 36 benchmarks 1 system 1 benchmark 4 systems 0 benchmarks 4 systems 8 benchmarks 4 systems 4 benchmarks 0 systems 0 benchmarks Figure 8: Interaction modes organized by interaction timing, whether input occurs only upfront or continues during execution, and interaction initiative. Counts state how many of the 58 reviewed systems and 42 benchmarks support the respective mode. Counts are not mutually exclusive; 7 systems and 7 benchmarks support more than one mode. Interaction modes define the control flow between user and system throughout the insight extraction process. As illustrated in Figure 8, we propose distinguishing interaction modes by interaction timing, differentiating whether user input occurs solely before data processing begins or continues throughout, and by the interaction initiative, differentiating whether the system or the user takes the lead in steering the insight extraction and alignment with the user’s intent. The initiative dimension draws on the spectrum of mixed-initiative interaction established by Allen (1999), ranging from purely user-led interactions through increasing degrees of system involvement to fully negotiated interactions in which neither party has a fixed role and both take active roles in steering the interaction. The timing dimension distinguishes modes in which user-system interaction is confined to an upfront stage before the system performs any data processing, from those in which interactions unfold throughout the execution, which is particularly relevant in OpenTI settings where users cannot anticipate the data and analytical choices the system will confront during processing (Horvitz, 1999; Radlinski & Craswell, 2017). Most systems (Section 3) and benchmarks (Section 5) assume input occurs solely before processing. The simplest case is single-input interaction, shown as the left-most mode in Figure 8, where a single user 33 input triggers the full extraction process without intermediate exchange, corresponding to purely user- initiative interaction. This mode minimizes user involvement and relies entirely on conventional and selective resolution to derive ˆ i s , but provides no mechanism to resolve remaining underspecifications or adapt to data particularities that users are not aware of, resulting in the propagation of misalignments into the final output. This places a high burden on the initial input to provide sufficient specification (Yu et al., 2019), which may be particularly challenging the more complex the required analysis (Grammel et al., 2010). Iterative elicitation (second from the left in Figure 8) extends single-input by engaging users in pre-processing dialogue to refine ˆ i s before data binding or analysis begins, mirroring commercial 3 and academic (Huang et al., 2025) “deep research” systems that ask clarifying questions to stabilize intent. This can surface and correct misconceptions before committing computational resources, but adds interaction overhead before users see any results and cannot address ambiguities arising from data selection or corpus- specific particularities encountered during execution. While only taking input upfront minimizes user interactions, it does not enable users to react to and steer the insight extraction process, requiring them to be highly aware of and capable of expressing their insight need with sufficient specification. In contrast, integrating user input throughout processing enables reactive steering based on intermediate results, at the cost of increasing the interaction complexity. This provides users with the opportunity to disambiguate, verify intermediate results (Yu et al., 2019), contribute domain knowledge, specify latent preferences, or intervene on critical decisions (Shao et al., 2026). The simplest such extension is output-driven interaction (third from the left in Figure 8), where users observe system outputs and provide refinements, triggering iterative cycles until the realization r satisfies their need (Xie et al., 2024; Menon et al., 2025). This allows users to recognize retroactively when their initial specification led to misaligned realizations, correcting poor selective specification by the system as it becomes apparent upon seeing concrete results. Yet, achieving a satisfactory realization may require many iterations if the initial ˆ i diverges significantly from i, and requires full processing to complete before feedback can be incorporated. Step-wise interaction (Li et al., 2025b; Yin et al., 2023; Dong et al., 2025; Yu et al., 2019; Xiao et al., 2022), shown as fourth from the left in Figure 8, grants finer control by having users provide instructions step-by-step, directly specifying individual analytical steps rather than the complete, potentially high-level, insight need upfront. This is still a user-initiative interaction, but applied at the granularity of individual operations rather than the full extraction task. It suits more exploratory needs where i emerges for users through data engagement (Marchionini, 2006), but shifts the orchestration burden to the user, requiring greater analytical expertise and end-to-end planning. In system-initiated clarification (second from the right in Figure 8), initiative shifts to the system, which solicits user input when encountering low-confidence decisions or missing information (Wu et al., 2024a) required to resolve ˆ i, that is, when selective resolution has low confidence or when information is genuinely missing. Rather than proceeding with uncertain interpretations, the system explicitly engages the user to resolve ambiguities, operationalizing the cooperative principle by exercising agency where it can while deferring when alignment is uncertain (Horvitz, 1999). This, however, requires a system that reliably esti- mates its uncertainty and may interrupt user workflows at unpredictable points. Collaborative interaction (right-most in Figure 8) generalizes this to a bidirectional exchange throughout processing (Chen et al., 2025e; Freund et al., 2025). The system requests clarification while users can also intervene proactively, observing operations and providing guidance without explicit solicitation (Allen, 1999). This flexibility lets collaborative interaction address all three resolution challenges bidirectionally, but introduces implemen- tation complexity and may overwhelm users unfamiliar with the analytical process. System-initiated and collaborative modes are only emerging for OpenTI, building on established mixed-initiative principles from HCI research (Horvitz, 1999; Allen, 1999). These modes are not discrete alternatives but form a continuum, and aspects of different modes may be combined, for instance by applying step-wise control with iterative refinement at each step. The appropriate mode depends on user expertise and the complexity and requirements of the analytical task (Jameson & Schwarzkopf, 2002). Empirical evidence from information retrieval suggests that the degree to which an in- sight need is directed elicits measurably different interaction behaviors (Athukorala et al., 2016). Needs that are less directed at the start, which iteratively become more concrete through engagement with data (Mar- 3 E.g., Google Gemini Deep Research, Mistral Le Chat. 34 chionini, 2006; White & Roth, 2009), require sustained interaction throughout processing and tend to be poorly served by upfront specification modes, whereas needs that are more concrete and directed from the start place lower demands on continued interaction. Overall, interaction modes should accurately approxi- mate the insight need while minimizing the burden placed on the user (Wu et al., 2024a). The interaction modes described above assume the system commits to a single realization r at each step, with iteration correcting misalignments between r and the user’s latent i. An alternative, compatible with several of the modes above, is for the system to surface multiple candidate realizations r 1 ,...,r k simul- taneously (Bhaskar et al., 2023), making selective resolution explicit by presenting users with the space of valid alternatives rather than having the system choose implicitly. Saparina & Lapata (2025a) gener- ate multiple natural language interpretations of ambiguous queries before parsing, allowing users to select among semantically distinct readings. Similarly, multiverse analysis interfaces (Sarma et al., 2024) present multiple valid analytical paths arising from defensible methodological choices, enabling users to evaluate how results vary across reasonable alternatives. These approaches are particularly relevant in upfront speci- fication modes where post-processing iteration is limited, making selective resolution explicit and preserving user agency over choices that would otherwise be delegated to the system. However, presenting too many alternatives risks overwhelming users, while too few may miss valid interpretations (Saparina & Lapata, 2025a). Balancing comprehensiveness with cognitive manageability remains an open design challenge. 4.2 Interaction Means Orthogonal to the interaction mode, the means of interaction describe the mechanisms by which users provide input. Human-computer interaction research has established a progression of interaction styles, from command languages and form-filling through menu selection to direct manipulation (Shneiderman, 1981), each trading expressiveness against the guidance it provides to users. This is especially relevant in OpenTI where users express analytical intent over data whose vocabulary and structure they may not know. Natural language text constitutes the dominant interaction means in current systems and benchmarks, re- flecting the broader emergence of conversational interfaces following the proliferation of chatbots. Text aligns naturally with open-ended insight needs, accommodating varied user goals without requiring familiarity with system-specific syntax. However, natural language is also well-documented as an ambiguous medium for data interaction (Affolter et al., 2019), where unconstrained text provides no guidance about what information the system requires (Chopra et al., 2023), leaving users unaware of what they must specify to avoid unresolvable underspecification. Complementary interaction means address these limitations by scaffolding specification to make the space of choices explicit, and by enabling direct manipulation at appropriate abstraction levels to correct misaligned resolutions as they emerge. Specification Interfaces. Specification interfaces impose a structure on the input channel so that users are guided towards providing the relevant information the system requires, adding a layer of guidance beyond free-text interaction. They address users’ knowledge gap about what information the system expects by directing input into categories that signal what must be specified, something a blank text box does not do and that has been observed to slow users down and lead to underspecified requests (Chopra et al., 2023). They can also surface valid options among which users can select rather than delegating decisions to systems. Specification interfaces can be organized around the components of the insight need i d ,i m ,i o . To target the data need i d , specification interfaces may prompt for temporal scope, entity types, or domain boundaries, using the selected values to constrain retrieval over the corpus. For the methodological need i m , specification interfaces can enumerate the options for analytical operations or modeling choices, freeing users from having to articulate the corresponding terminology. For the output need i o , interfaces may expose modality choices such as tables, charts, or textual summaries, making the form of presentation a user-controlled rather than system-assumed dimension. Guo et al. (2024a) provide an empirical demonstration of this decomposition, with a prototype that separates fields for dataset descriptions, domain knowledge injection, and selection among system-generated execution plans, finding that users benefit from this explicit separation. A specification interface can provide static or dynamic scaffolding. Static interfaces fix the structure of the input channel at design time, which makes the system’s expectations predictable but assumes that the same 35 components are relevant across all insight needs. In contrast, dynamic interfaces synthesize the scaffolding from analytical context, presenting controls only at points where they serve a concrete underspecification. In natural language interaction with data this has been realized through follow-up questions (Chen et al., 2025e), ambiguity widgets (Gao et al., 2015; Setlur et al., 2016) that bind controls to specific ambiguous phrases (e.g., a distance slider for “near Paris”), and through type-aware widgets that adapt to the attribute being referenced (e.g., calendars for temporal attributes or maps for geospatial ones) (Setlur et al., 2020). The continuous adaptation of the specification interface throughout the analytical process aligns with the partial observability in OpenTI where the relevant gaps may not be known before retrieval and analytical composition begin. Infrastructure for such dynamic specification is re-emerging in LLM-based agents through toolkits such as MCP UI 4 , though design principles for when and how to invoke structured prompts within OpenTI systems remain underdeveloped. Direct Intervention. Direct intervention allows users to inspect and act on realizations while or after the system constructs them. It carries the principles of direct manipulation over to OpenTI, namely continuous representation of the objects of interest, physical actions rather than complex syntax, and rapid, incremental, and reversible operations with immediately visible effects (Shneiderman, 1981). Applied to realizations, these principles require interfaces that make the components r = (O,E,β) visible and editable, and that propagate user edits back into r to regenerate the dependent operations. Users can thereby correct misaligned choices that become apparent only in light of intermediate results. Such interfaces differ in the control they afford and the expertise they demand. At one end, users can be exposed to raw code, which gives full control but requires programming experience and imposes significant cognitive load even for proficient users confronted with unfamiliar implementations (Xie et al., 2024). At the other end, intervention purely through follow-up natural language is accessible to non-programmers but requires users to articulate misalignments precisely and forces the system to ground these descriptions back into specific components of r, introducing miscommunication risks. Intermediate representations balance these extremes. Examples include editable natural language explanations of code (Chen et al., 2025e; Tian et al., 2023) and editable assumptions and requirements (Kazemitabaar et al., 2024; Freund et al., 2025). Similarly, visual graph representations expose the operations (O,E) as nodes that can be inspected and parameterized without modifying code directly (Xie et al., 2024; Freund et al., 2025), while editable analytical plans expose the methodological structure i m as ordered lists of steps that can be edited, reordered, or marked as optional (Kazemitabaar et al., 2024; Tian et al., 2023). Direct interventions on the operations (O,E) can enable users to adjust thresholds, change column selections, modify aggregation functions, or alter the composition of analytical steps. Complementing these, interven- tions on the data binding β enable users to exclude irrelevant tables, request additional data matching specific criteria, or adjust relationships between retrieved tables. Setlur et al. (2023) demonstrate the effectiveness of steering retrieval through visual previews and filtering support by presenting candidate data sources with metadata tooltips summarizing available attributes and value distributions, while filtering widgets enable refinement by properties such as temporal range or data type. Interaction modes that involve the user throughout the analytical process presuppose some form of direct intervention. Step-wise, system-initiated clarification, and collaborative modes are defined by users acting on the realization as it emerges rather than merely responding to its final output, which requires representations of the emerging realization that users can inspect and act upon. Combining Interaction Means. The means described above are not mutually exclusive. Research on data analysis interfaces has shown that combining natural language with direct manipulation can be more effective than either means alone, as natural language handles open-ended expressions of intent while direct manipulation handles precise, reference-dependent operations such as selecting data points or adjusting parameters (Srinivasan et al., 2021; Srinivasan & Stasko, 2018). Among reviewed systems, such combinations remain largely unexplored, as systems mostly rely on natural language as their sole interaction means. 4 https://mcpui.dev/ 36 4.3 Interpretability Making the realization r = (O,E,β) that a system produces accessible and interpretable enables users to verify the analytical process and its alignment with their intent. To allow end-to-end verification, users must be able to confirm that the expressed insight need ˆ i is correctly interpreted, that the data binding β retrieved appropriate tables from the corpus, and that the operations O and their composition E are analytically sound. Empirical studies into verification behavior confirm that users combine assessing the applied operations with inspecting the data they operate on, switching between these levels throughout the verification process (Gu et al., 2024a; Xie et al., 2024), which indicates that interpretability must address all three components of the insight need to avoid verification gaps. Interpretability aims to foster warranted trust by allowing users to distinguish sound from unsound realiza- tions, instead of merely projecting transparency by exposing more information (Parasuraman & Manzey, 2010). Superficial transparency can induce unwarranted reliance on flawed realizations (Parasuraman & Manzey, 2010), which is consequential when outputs inform decisions or are otherwise relied upon. The conceptual foundations for surfacing (partial) realizations have been established in the study of data provenance and analytical provenance. Data provenance tracks the origin and transformation history of analytical results (Buneman et al., 2001; Cheney et al., 2009; Woodruff & Stonebraker, 1997), providing a foundation for interpretability by making explicit how outputs derive from the data through sequences of operations. Analytical provenance, established in visual analytics (Gotz & Zhou, 2009), captures the trajectory of the analytical process itself, including what operations were considered and applied, making the agent’s decision path explicit. Mapped to realizations, data provenance concerns β and the data flow induced by E, and analytical provenance addresses the operations O and their composition E. Beyond surfacing the realization itself, systems can make the interpretation process that derives r from the user input ˆ i explicit. Disclosing the interpretation, which happens, for instance, through selective resolution, allows users to detect misalignments between the system’s reading of their intent and the underlying need i before inspecting the analytical derivation (Xie et al., 2024; Kazemitabaar et al., 2024; Freund et al., 2025). Not just the contents of the interpretation but also its form matters because it determines who can effectively verify realizations. There is a trade-off between the fidelity with which the underlying realization is presented and the accessibility to users. Raw code preserves full fidelity but requires programming proficiency to read and imposes cognitive load even on proficient users (Xie et al., 2024). Visual graph representations of operations (Xie et al., 2024) abstract the realization into nodes and edges that can be interrogated individually without exposing implementation details. Natural language decompositions of generated code into explanations (Narechania et al., 2021; Tian et al., 2024; 2023; Menon et al., 2025; Nguyen et al., 2025) offer another form, providing a human-readable intermediate representation between the raw derivation and the user. Tian et al. (2024) extend these natural language decompositions by aligning each explanation step with the relevant database elements and displaying intermediate results after each step. Empirical evidence indicates that such step-by-step descriptions paired with sample data enable non-experts to locate errors in the analysis (Narechania et al., 2021). An effective representation remains accessible enough to allow verification while staying structured enough to preserve the analytical content of the realization. Beyond representations of the operations, inspectable intermediate data states (Gu et al., 2024a; Kazemitabaar et al., 2024) address data provenance by allowing users to observe how operations affect the data as the realization is executed, which provides evidence for assessing whether β and E align with the intended interpretation. While interpretations can be surfaced post-hoc, allowing users to inspect a completed realization and decide whether to trust its output, they can also be disclosed concurrently with the analytical process as it un- folds. Such concurrent interpretability is a precondition for the output-driven, step-wise, and collaborative interaction modes discussed in Section 4.1. Sufficient interpretability is required to avoid verification gaps, but presenting excessive information risks overwhelming users (Kazemitabaar et al., 2024). Guo et al. (2024a) find that users particularly seek explana- tions when outputs do not match their expectations, suggesting that interpretability needs are non-uniform across the verification process. Effective interpretability should therefore address the trade-off between comprehensiveness and cognitive load. Progressive disclosure (Shneiderman, 1996), in which users access a summary-level view of the realization by default and navigate to finer levels of detail on demand, offers 37 a possible design paradigm, which has been extensively explored for visual analytics drill-down interfaces. The appropriate default abstraction and the requirements for drill-down depend on user expertise and on the complexity of the analytical task, neither of which the reviewed works study systematically. 5 Evaluations in OpenTI In this section we understand assessing the gap between a system’s theoretical capabilities and its practical utility as the primary objective of evaluation. This requires systematically quantifying a system’s strengths and weaknesses through a robust methodology that mirrors the complexities of real-world data interaction. Formally, an evaluation benchmark B = (T,E ) consists of a corpus of tables T and a set of n evaluation instances E = e 1 ,...,e n . Each evaluation instance e = ( ̄ i,v) comprises an input ̄ i, representing the expression of an insight need ˆ i, and a validation mechanism v : R → [0, 1] that scores the quality of a system’s realization. In most current benchmarks, ̄ i is a single natural language utterance and v reduces to comparing the output against a predetermined reference output. However, more sophisticated instantiations can capture interactive input protocols and nuanced validation criteria that assess not only the final output but also the analytical process by which it was derived. This section first provides an overview of the surveyed benchmarks (Table 2), and then analyzes them along their components. We systematically analyze the benchmarks that we find in our literature review and select a subset of benchmarks relevant for OpenTI that we present in the overview in Table 2. The methodology for the survey and the selection criteria are detailed in Appendix C. Section 5.1 examines the corpora underlying the benchmarks, Section 5.2 analyzes inputs, and Section 5.3 investigates validation mechanisms. Each section lays out key desiderata for the respective component before analyzing how benchmarks address them. Table 2 gives an overview of the analyzed benchmarks, presenting the characteristics of their corpora, inputs, and validation mechanisms. The benchmarks span a wide range of scope and complexity, from single-table question answering over web tables to multi-step analytical workloads over relational and file-based corpora. 5.1 Corpora The corpus T in a benchmark B defines the data environment on which systems are evaluated. In current benchmarks, corpora range from collections of thousands of web-extracted tables to smaller sets of analytical datasets sourced from repositories like Kaggle or GitHub. These corpora differ substantially in their size, the complexity of their tables, and the degree to which they reflect real-world data environments. 5.1.1 Desiderata To simulate an evaluation environment representative of real-world data environments, T should mirror the complexity and scale of real-world data sources. Thus, a corpus suitable for evaluating OpenTI systems should satisfy the following desiderata: • Realism and Scale. Many academic datasets are built on tables extracted from web pages, which are typically small (Nan et al., 2022; Chen et al., 2021). In contrast, representative organizational or scientific tables often have significantly more columns and rows (Li et al., 2017; Hulsebos et al., 2023) and exhibit different distributions of semantic column types (Hulsebos et al., 2023). A benchmark should therefore feature a large-scale corpus (Lai et al., 2026; Mitsopoulou & Koutrika, 2025) con- taining numerous, sizable tables from diverse domains to properly test a system’s ability to operate in a realistic open setting. • Messiness. Real-world data is rarely pristine. A realistic benchmark should include tables charac- terized by a degree of messiness, such as missing values, inconsistent formatting, extraneous informa- tion, or varying schema quality (Lai et al., 2026; Mitsopoulou & Koutrika, 2025; Dutta et al., 2026). This ensures the evaluation tests not only a system’s analytical capabilities but also its essential data preparation and cleaning abilities (Chen et al., 2025b). 38 Table 2: Overview of analyzed benchmarks. Words denotes the average number of words in an input. Insight Types are classified following the taxonomy in Section 2.3: Look - Lookup, Agg - Aggregation, Char - Characterization, Asso - Association, Grp - Grouping, SI - Statistical Inference, Pred - Prediction, Int - Interventional, CF - Counterfactual. A bordered box Grp / Pred denotes instances that require an artifact (e.g., a prediction model) instead of being directed at an insight. Tbl/Inst. details the mean number of tables required per instance, where 1 indicates that only a single relevant table is provided, n indicates that only the set of relevant tables is provided, indicates that a limited set of relevant and irrelevant tables is provided, and indicates an open setting. “-” indicates that the data is not available or the analysis is not applicable to the benchmark. Corpus Inputs Validation Functions Name #Tab Source Supplement Data #Inst. Tbl/Inst. Words Insight Types Function Output Spider 2018 1,056 University Courses, DatabaseAnswers Relations 11,840 1.55 12.2 Look Agg out : Exact matching r : Structural matching Table, Value OTT-QA 2021 8,891 Wikipedia Text Passages, Title 4,372 1 19.9 Look Agg out : Exact matching, Similarity matching Value NQ-Tables 2021 169,898 Wikipedia 11,628 1 8.9 Look Agg out : Exact matching, Similarity matching Value FeTaQA 2022 10,330 Wikipedia Text Passages, Title 10,330 1 1 13.2 Look Agg out : Similarity matching, Human evaluation Text ARCADE 2023 106 Kaggle, GitHub 1,078 1 1 - Look Agg Char out : Similarity matching Value, Table Open-WikiTable 2023 24,680 Wikipedia Page Title 67,023 1 18.1 Look Agg out : Exact matching Value BIRD 2023 611 Kaggle, ML Data Repos- itory, Manual Relations, ColumnDescription 10,962 2 14.1 Look Agg out : Exact matching Table, Value Archer 2024a 68 Spider Relations, Text Passage 1,042 2.17 26.2 Look Agg CF out : Exact matching Table, Value BLADE 2024b 14 Scientific Works Dataset/Column Description 188 1 1 68.0 Asso SI r : Component validation, Structural matching Value DA-Code 2024 1,733 GitHub, Kaggle, Web Files 500 n 3.47 40.6 Look Agg Char Asso Grp SI Pred out : Exact matching Text, Value, Table, Chart InfiAgent-DABench 2024 67 GitHub 257 1 1 27.4 Agg Char Asso SI Pred out : Exact matching Value DA-Dataset 2025 689 Financial, BIRD Relations 735 4.5 - Agg Asso Char out : LLM-as-judge Text Spider 2.0 2025a 418 BigQuery, Snowflake Marketplace, Web Relations 632 3.89 54.0 Look Agg Asso out : Exact matching, Programmatic validation Text, Table TableBench 2025b 886 Wikipedia 886 1 1 20.2 Look Agg Asso Pred out (text): Similarity matching out (chart): Programmatic validation Text, Chart DABstep 2025 3 Financial Files 450 - 22.4 Look Agg CF out (value, list): Exact matching out (text): Similarity matching Text, Value,List MT-RAIG 2025 19,563 Spider, Open-WikiTable 18,532 2.88 36.5 Look Agg Char Asso out : LLM-as-judge Text DAComp - DA 2025b 418 Curated web DBs Relations 100 4.18 90.1 Look Agg Char Asso out : LLM-as-judge Text + Chart MultiTableQA 2026 59,307 Wikipedia 23,785 2.39 14.8 Look Agg out : Exact matching, Similarity matching Value KramaBench 2026 1,636 Scientific Works Files 104 3.33 33.8 Look Agg Asso Pred out : Exact matching, Similarity matching r : Component validation Text, Value,List ConDABench 2026 1,855 TidyTuesday, Scientific Works, Kaggle 1,420 n 1.31 13.8 Look Agg Char Asso Pred out : LLM-as-judge Text, Chart TACO 2026 13,004 Open Data Portals Relations 13,000 - - Look Agg out : Exact matching Table, Value TOPBench 2026 35 Kaggle Column Description, Value Distribution 779 1 1 108.8 Pred out : Exact matching, Similarity matching, LLM- as-judge Text, Table CausalReasoningBenchmark 2026 132 Scientific Works Column Description, Study Context 173 1 1 32.2 Agg SI Int out : Similarity matching, r : Component validation Text, Value 39 • Domain Diversity. In the open setting systems must generalize across heterogeneous sources and topics. A corpus dominated by a narrow domain like sports statistics may overestimate performance by allowing systems to exploit specific world-knowledge or domain-specific patterns (Gan et al., 2021). Corpora should span multiple domains to assess whether systems can transfer capabilities across different subject areas and data conventions. • Structural Diversity. Tables have varied structures, including wide versus tall layouts (Döhmen et al., 2024), normalized versus denormalized schemas (Kohita, 2025), and different levels of meta- data availability. A corpus with structural diversity enables testing different integration challenges, particularly for insight needs that span multiple tables (Kohita, 2025). 5.1.2 Analysis of Corpora Current benchmark corpora exhibit a clear trade-off between the number of tables in the associated corpus |T| and the structural complexity of the tables in these corpora. This becomes apparent when considering the #Tab and Source columns of Table 2 together with the distribution of row and column counts across indi- vidual tables, depicted in Figure 9. Corpora derived from Wikipedia web tables, such as NQ-Tables (Herzig et al., 2021), OTT-QA (Chen et al., 2021), FeTaQA (Nan et al., 2022), Open-WikiTable (Kweon et al., 2023), and MultiTableQA (Zou et al., 2026), offer large scale, often containing tens of thousands of tables. However, the individual tables are predominantly small and clean, with the overwhelming majority featuring fewer than ten columns and rarely exceeding a few hundred rows, intended for presentation rather than analysis. They thereby provide scale at the expense of structural realism and diversity. Additionally, it can be assumed that the overwhelming majority of large language models have been exposed to Wikipedia contents during training (Yu et al., 2024), potentially leading to data contamination and a better contextual understanding of Wikipedia contents (Ni et al., 2025), which can have downstream impacts on the measured performance and the domain generalization capabilities beyond Wikipedia tables. Corpora sourced from relational databases and analytical repositories exhibit more realistic table character- istics at the cost of scale. Relational corpora such as Spider (Yu et al., 2018) and BIRD (Li et al., 2023) extend to far wider row distributions, with individual tables reaching hundreds of thousands of rows, while data-analysis corpora drawn from Kaggle, GitHub, and scientific sources, such as DA-Code (Huang et al., 2024), InfiAgent-DABench (Hu et al., 2024), and KramaBench (Lai et al., 2026), exhibit the widest spread in both rows and columns, more closely reflecting the complexity of tables used in actual analytical work- flows. However, these corpora are substantially smaller, containing hundreds rather than tens of thousands of tables, which limits their utility for evaluations in an open setting for table discovery and retrieval. Kram- aBench (Lai et al., 2026) is a notable exception in coupling raw, realistic data with a data-lake organization spanning multiple formats, though it too remains modest in its number of tables. Beyond the tables themselves, benchmarks contain diverse additional data, reported under Supplement Data in Table 2. Some establish their corpus as relational data (Yu et al., 2018; Li et al., 2023; Lei et al., 2025a; Deng et al., 2026; Xu et al., 2025; Wu et al., 2025a), combining tables with titles and relationships in the form of foreign-key relationships. Next to such relations, many benchmarks provide further contextual metadata associated with tables, like contextual descriptions (Nan et al., 2022; Chen et al., 2021) or documentation files (Huang et al., 2024; Lai et al., 2026; Egg et al., 2025). Such contextual information may be used to bridge the semantic gap between tabular data and user inputs to aid functional capabilities like interpretation, data retrieval (Gomm & Hulsebos, 2025), and analytical composition and execution (Egg et al., 2025). The characteristics of the corpora largely follow from where they are sourced rather than from deliberate design. Realism and scale stand in direct tension, as most large-scale corpora are sourced from Wikipedia, yielding tables that do not resemble those in organizational corpora. Exceptions to this are KramaBench (Lai et al., 2026), which pairs raw scientific data with a data-lake organization spanning multiple formats, but at a modest number of tables, and TACO (Deng et al., 2026), which has a large corpus of 13,000 tables from municipal and federal open data portals but parses and cleans the raw tables into normalized relational databases. Messiness is thus rarely treated as a design property. Synthetically constructed corpora (Yu et al., 2018; Egg et al., 2025; Lei et al., 2025b) are clean by construction, and corpora assembled from existing data are commonly filtered and standardized during curation, leaving only a few benchmarks that deliberately 40 0.0 0.2 0.4 0.6 0.8 1.0 OTT-QA NQ-Tables FeTaQA OpenWikiTables TableBench MT-RAIG MultiTableQA 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative Share of Tables in Corpus Spider BIRD Archer Spider 2.0 DA-Comp 10 0 10 1 10 2 10 3 10 4 10 5 10 6 10 7 Number of Rows 0.0 0.2 0.4 0.6 0.8 1.0 10 0 10 1 10 2 10 3 Number of Columns BLADE DA-Code InfiAgent DABstep KramaBench ConDABench TOPBench CausalReasoning Figure 9: Empirical cumulative distributions of per-table row and column counts for the corpora in Table 2. Steep curves at low counts mark corpora of small tables, while flatter curves indicate diversity in row and column counts. retain dirty data (Li et al., 2023; Huang et al., 2024; Dutta et al., 2026; Lai et al., 2026). The domains a corpus covers are likewise a byproduct of its source. Corpora sourced from Wikipedia or open data portals offer a broad range of domains, whereas the domain coverage is narrow in benchmarks targeting one specific analytical setting (Egg et al., 2025; Xu et al., 2025; Lei et al., 2025b), so that the corpora spanning domains broadly are the same web table corpora with the least structurally realistic tables. 5.2 Inputs The input ̄ i defines part of the environment for evaluated OpenTI systems, comprising the user inputs that are observable. Following the framework established in Section 2, ˆ i is the expressed form of the user’s latent insight need i communicated in user-system interaction, comprising data, methodological, and output components. The input ̄ i defines the protocol under which ˆ i is emulated in the evaluation. Since potential interaction modes range from single inputs to mixed-lead collaborative interactions, requirements on what ̄ i needs to capture and how interaction should be emulated vary significantly between interaction modes. Benchmarks are usually proposed targeting specific interaction modes. In single-input settings, ̄ i can be a standalone natural language utterance; in interactive settings, it may comprise a sequence of exchanges or a protocol for responding to system-initiated clarification. Benchmarks vary considerably in the analytical complexity of insight needs, the degree to which inputs are suitable for evaluations under an open premise, and whether they support interaction beyond single utterances. 5.2.1 Desiderata Inputs suitable for evaluating OpenTI should satisfy the following desiderata: • Data-Independence. In the open setting, users formulate insight needs without knowledge of spe- cific data structures or contents (Voorhees, 2001). Authentic inputs should thus be data-independent, formulated from the user’s conceptual perspective, without using privileged knowledge of the under- lying corpus a user does not have. Data-privileged inputs, by contrast, reference structural elements such as column headers (e.g., “last_name”), specific values not in the public domain (e.g., “the 41 airplane that took off at 2026-05-08T11:48:11+00:00”), or data containers themselves (e.g., “the user table”). Such references provide unrealistic signals linking queries to specific data structures, undermining the premise of an open corpus (Gomm et al., 2025; Suhr et al., 2020; Lee et al., 2021). • Sufficient Specification. As argued in Section 2.1.2, insight needs may exhibit a degree of latency where users do not exhaustively articulate them but provide what they judge necessary and rely on systems to resolve the rest. What counts as sufficient specification therefore depends on the validation mechanism used to score a realization. When paired with a validation mechanism that admits only a single reference output or realization, inputs must fully determine the required data and methodology, since alternative interpretations would produce divergent but plausibly correct responses that the validation cannot accommodate. Validation mechanisms that accept multiple valid realizations allow inputs to leave ambiguities in corresponding aspects. Misalignment between the degree of ambiguity in inputs and the validation mechanism can conflate accuracy with inter- pretation capabilities, yielding unreliable signals (Bhaskar et al., 2023; Saparina & Lapata, 2025a; Gomm et al., 2025). • Analytical Complexity. Real-world analytical workloads often require multi-step reasoning, in- volving sequences of operations that build upon intermediate results (Gu et al., 2024b; Wu et al., 2025b; Lai et al., 2026). Inputs should reflect this complexity to test whether systems can plan and execute extended analytical procedures. • Multi-Table Sources. Real-world questions often require integrating information from multiple tables. Therefore, a significant share of the evaluation instances should require retrieving and inte- grating information from multiple tables (Lai et al., 2026; Wu et al., 2025a; Akhtar et al., 2025). 5.2.2 Input Protocols Depending on how inputs are provided, evaluation protocols fall into two categories: static and user simu- lation. Static protocols fix all inputs upfront in the benchmark itself, while user simulation protocols define the mechanisms by which inputs that depend on the analytical state or system elicitation are determined. In their simplest and most common form, static protocols define a single standalone utterance per evaluation instance to emulate single-input interactions, which allows tight control over the input and a consistent, replicable evaluation environment. Similarly, for step-wise interaction a multi-turn sequence of inputs can be fixed upfront (Yu et al., 2019; Yin et al., 2023; Liu et al., 2023; Li et al., 2025c). While such sequences emulate a multi-step process, the predetermined path does not adapt to the intermediate analytical state a system exposes at each step, trading the realism of a genuinely contingent interaction for replicability. User simulation protocols instead generate inputs dynamically, emulating a user who provides additional inputs proactively based on observations of the analytical state and/or reactively in response to systems requesting inputs. User simulation has a long lineage in information access research (Balog & Zhai, 2024). It generally centers on a model of the user (Balog & Zhai, 2024), which can be based on rule-based and agenda-based simulators (Schatzmann et al., 2007) that generate inputs by reacting to system outputs according to an explicit representation of user goals, or an end-to-end neural architecture (Gür et al., 2018; Lin et al., 2021) that learns the behavior. LLMs have been proposed as end-to-end user models (Sekulić et al., 2022; 2024; Shao et al., 2026). Empirical assessment finds that such models correlate well with human judgments overall (Dou et al., 2025), but cautions against possible goal misalignments, particularly in multi- turn contexts (Zhang et al., 2020b; Mehri et al., 2026), and against unrealistic response patterns (Seshadri et al., 2026), against which scaffolding techniques have been proposed (Mehri et al., 2026). 5.2.3 Analysis of Inputs Current benchmarks overwhelmingly adopt static input protocols for single-input interaction, whereas a smaller set fixes multi-turn inputs upfront to emulate step-wise interaction (Yu et al., 2019; Yin et al., 2023; Li et al., 2025b; Liu et al., 2023; Li et al., 2025c). User simulation remains rare, applied only in ConDABench (Dutta et al., 2026), BIRD-INTERACT (Huo et al., 2025), and COTA (Li et al., 2025b). They all employ an LLM to simulate the user but differ in how they align it with the underlying insight 42 need i. COTA (Li et al., 2025b) grounds the user simulator in an extensive description of the insight need, decomposed into a set of sub-questions, as well as a sample of the table contents. BIRD-INTERACT (Huo et al., 2025) and ConDABench (Dutta et al., 2026) provide the schema of the data and directly expose the desired analytical operations as SQL and Python code to the LLM, while prompting it to use the code to inform interactions but not to share it, risking leakage of information the system did not ask for. As shown in Table 2, the benchmarks focus mostly on descriptive insights, with little coverage of inferential and causal insights. Lookups and aggregations have the broadest coverage, and many benchmarks do not go beyond these two types. Some benchmarks remain restricted to simple min and max aggregations that do not require calculations (Chen et al., 2021; Nan et al., 2022; Herzig et al., 2021), allowing direct inference over the full table to derive answers from the raw data. In comparison, most benchmarks that expand to more complex aggregations, characterization, and association, which require more complex numerical opera- tions, are rooted in data analysis workflows (Huang et al., 2024; Hu et al., 2024; Lei et al., 2025a; Wu et al., 2025b; Gu et al., 2024b) and are mostly absent from the question answering and text-to-SQL benchmarks. At the inferential tier, statistical inference is covered as an explicit analytical objective in BLADE (Gu et al., 2024b), DA-Code (Huang et al., 2024), InfiAgent-DABench (Hu et al., 2024), and CausalReasoningBench- mark (Sawarni et al., 2026). Prediction is covered more widely but to different extents. DA-Code (Huang et al., 2024) and InfiAgent-DABench (Hu et al., 2024) contain inputs for constructing predictive models with- out seeking a specific insight, whereas TableBench (Wu et al., 2025b), KramaBench (Lai et al., 2026), and ConDABench (Dutta et al., 2026) target simple forecasts through regression. TOPBench (Ji et al., 2026) is the first dataset that centers on predictive insights, verbalizing complex predictive needs that require build- ing and applying models to extract the insights. Causal insight needs have been covered on a shallow level as counterfactual insights in Archer (Zheng et al., 2024a) and DABstep (Egg et al., 2025), which explore counterfactual scenarios that are resolved under the application of provided static rules. Only CausalReason- ingBenchmark (Sawarni et al., 2026) targets genuine interventional analysis, requiring formal observational identification strategies such as instrumental variables, regression discontinuity, and difference-in-differences alongside statistical estimation. Beyond the covered insight needs, input definitions across benchmarks vary widely in the data requirements they impose. Many benchmarks, particularly those focused on question answering, simplify the problem setting by requiring evidence from a single table only, abstracting away the real-world complexities of data integration that define the full scope of OpenTI. The Tbl/Inst. column of Table 2 showcases this, with 9 of the 23 benchmarks assuming that either a single relevant table or a set of relevant tables is directly provided to a system, while only 6 target an open setting that necessitates retrieval. The remainder restrict selection to a bounded set, mostly a single database. Similarly, the complexity of the insight needs expressed in the input definitions is unevenly distributed across benchmarks. We report the average words per input definition as a proxy for the complexity of insight needs in the Words column of Table 2, observing a large range from an average of 8.9 words for inputs in NQ-Tables (Herzig et al., 2021) to 108.8 words in TOPBench (Ji et al., 2026), whose intent-rich queries embed the analytical scenario in a narrative description. However, while longer inputs generally allow for more detailed instructions, word count alone fails to capture the latency in expressions of insight needs, which allows insight needs to be compressed into shorter forms under greater latent assumptions and delegation. DABstep (Egg et al., 2025) and Spider 2.0 (Lei et al., 2025a) are deliberately designed so that the expressed insight needs require multiple analytical steps that build on intermediate results, whereas most other benchmarks can be served by one-shot code generation. Next to these structural properties, evaluating OpenTI systems in an open setting requires that realistic inputs are formulated independently of the exact data that serves as evidence while providing sufficient specification of the methodological and data need in the expression of the insight need. We assess the degree to which these desiderata are violated following the LLM-judge setup proposed by Gomm et al. (2025), with results shown in Figure 10, detailed in Appendix D. Benchmarks vary significantly in the data privilege found in inputs. The analysis reveals high data privilege primarily in data-analysis benchmarks like InfiAgent-DABench (70.0%), DABstep (78.4%), and TOPBench (85.0%), whose inputs reference specific columns and files, limiting the degree to which evaluations in open settings capture realistic interactions. On the other end of the spectrum, NQ-Tables (0.0%) and FeTaQA (0.4%) have negligible levels of data privilege. In addition, all benchmarks show high levels of insufficient specification of inputs. All benchmarks but NQ- 43 Tables (31.6%) and FeTaQA (47.4%) contain more than 50% of inputs that are insufficiently specified. While validation mechanisms based on similarity or LLM judgment may absorb some of this ambiguity, the dominance of exact-matching validation in surveyed benchmarks (Section 5.3.2) means that the bulk of insufficiently specified inputs are paired with validation that admits only one valid interpretation, conflating execution accuracy with the ability of systems to guess the intended interpretation. Spider OTT-QA NQ-Tables FeTaQA OpenWikiTables BIRD Archer DA-Code InfiAgent Spider 2.0 TableBench DABstep MT-RAIG DA-Comp-DA MultiTableQA KramaBench TOPBench CausalReasoning 0% 20% 40% 60% 80% 100% Share of Insufficiently Specified InputsShare of Data-Privileged Inputs Figure 10: Share of inputs that are insufficiently specified (i.e., admitting multiple interpretations) and that exhibit data privileges (i.e., leakage of privileged information from the data into the input) across benchmarks. Overall, inputs across the benchmarks remain limited for evaluating OpenTI systems in an open setting. Since most benchmarks are not designed for an open setting, many of their inputs exhibit high levels of data dependence and insufficient specification. The analytical complexity is bounded, with the most widespread focus on lookup and aggregation types, whereas inferential and causal insight types remain only sparsely covered. In contrast, many benchmarks include inputs that require combining evidence from multiple tables to address the insight need, though most do so in a data setting where a small set of tables is provided and the set of relevant tables does not have to be selected from a large corpus. 5.3 Validation Mechanisms The validation mechanism v : R → [0, 1] determines how evaluation assesses the quality of a system’s realization against the user’s insight need. Ideally, validation would assess to which degree the complete realization r = (O,E,β), consisting of the operations performed, their composition, and the data to which they are bound, constitutes a sound derivation of the analytical knowledge that addresses the user’s insight need. In practice, the validation target ranges from the terminal output out(r) produced by the realization to components of the realization r itself. Connecting this to the problem formulation (Section 2.1.5), v acts as an operational stand-in for the validity criterion captured by R valid (i) and the utility function u i . Since the underlying insight need is fixed by the benchmark’s creators, the validation mechanism encodes their judgment of which realizations satisfy the need and how well. An ideal validation mechanism would take the form v(r) = 1[r ∈ R valid (i)] · g(u i (r)), where g maps utility monotonically into (0, 1], with its exact form immaterial since the objective in Equation 1 depends only on the ordering that u i induces. Realizations that fail to address the insight need score zero, while valid realizations are scored by how well they serve it. In practice, validation in existing benchmarks primarily approximates the validity indicator, leaving the quality of realizations and outputs largely unjudged. 44 We separate validation functions that operate on the output level (Section 5.3.2) from those that operate on the realization level (Section 5.3.3), and consider combined functions (Section 5.3.4). Output-level validation compares the final output out(r) against reference outputs or success criteria, without assessing the validity of the analytical process itself. In contrast, realization-level validation examines components of r directly, assessing the analytical process itself rather than only its results. Table 3 provides an overview of the validation functions we discuss in this section. We focus on functions that score the end-to-end validity of an analysis or its output, setting aside metrics that isolate a single functional capability, such as the recall of the retrieval capability (Herzig et al., 2021; Zou et al., 2026). Table 3: Overview of validation functions in reviewed benchmarks grouped by whether they validate outputs or the realizations themselves. Formalizations correspond to the specific functions employed by the benchmarks. Validation FunctionBenchmarks Output-level Exact matching matches the output, or sub-output out j (r), against a reference output out ∗ under an equivalence function 1[·]. v(r) = 1[out(r) = out ∗ ]Spider 2018, OTT-QA 2021, NQ-Tables 2021, Open-WikiTable 2023, BIRD 2023, Archer 2024a, Spider 2.0 2025a, DA-Code 2024, Multi- TableQA 2026, TACO 2026, TOPBench 2026, KramaBench 2026 v(r) = 1[out j (r) = out ∗ j ∀j ∈ 1,...,n]InfiAgent-DABench 2024 Similarity matching measures the similarity of the output and a reference with a similarity measure sim. v(r) = sim(out(r), out ∗ )FeTaQA 2022, ARCADE 2023, TableBench 2025b, KramaBench 2026, TOPBench 2026, CausalReasoningBenchmark 2026, NQ-Tables 2021, OTT-QA 2021, MultiTableQA 2026 v(r) = max 1[out(r) = out ∗ ], sim(out(r), out ∗ ) DABstep 2025 LLM-as-judge prompts a language model to score the output against the reference following instructions instr or a set of criteria C. v(r) = LLM(out(r), out ∗ , instr)DA-Dataset 2025, ConDABench 2026, TOPBench 2026 v(r) = 1 |C| P c∈C LLM(c, out(r))MT-RAIG 2025, DAComp 2025b Programmatic validation checks the output against a set of per-instance programmatic test cases tests. v(r) = 1[check(out(r),τ)∀τ ∈ tests]Spider 2.0 2025a, TableBench 2025b Human evaluation asks annotators to judge the output against a set of criteria C. v(r) = 1 |C| P c∈C human(out(r),c)FeTaQA 2022 Realization-level Program comparison compares the code p(·) of the realization with that of a reference realization. v(r) = sim(p(r),p(r ∗ ))none among the benchmarks in Table 2 Structural matching compares the operations of the realization r = (O,E,β) and their composition against a reference realization r ∗ = (O ∗ ,E ∗ ,β ∗ ) directly. v(r) = 1 |O ∗ | P o∈O 1[o∈ O ∗ ]Spider 2018, BLADE 2024b Component validation checks the realization against a set of per-instance criteria C. v(r) = 1 |C| P c∈C check(r,c)BLADE 2024b, KramaBench 2026 v(r) = 1[check(r,c)∀c∈ C]CausalReasoningBenchmark 2026 5.3.1 Desiderata Validation mechanisms suitable for evaluating OpenTI should satisfy the following desiderata: • Scalability and reproducibility. Human evaluation offers flexibility in assessing complex, open- ended outputs but is costly, slow, and difficult to reproduce consistently. Instead, a practical valida- tion mechanism should be automated to ensure reproducibility, scalability, and objectivity, requiring minimal human intervention (Gu et al., 2024b). 45 • Multi-modal coverage. Analytical results can be presented in different modalities like natural language, numerical results, tables, and visualizations. The validation mechanism should thus be able to handle the modalities relevant to the evaluated insight needs (Zhang et al., 2025b; Huang et al., 2024). • Robustness and faithfulness. Metrics should reliably quantify what they claim to measure, providing signal rather than noise. This is particularly challenging for open-ended analytical outputs, where traditional text similarity metrics such as BLEU, ROUGE, and BERTScore have been found to correlate poorly with actual response quality and to not reliably separate factually accurate responses from those that are fluent yet factually incorrect (Wolff & Hulsebos, 2025; Wang et al., 2024a). Validation mechanisms must be calibrated to the specific characteristics of analytical outputs to yield meaningful performance estimates. • Acceptance of valid variation. A single insight need may admit multiple valid realizations pro- ducing different but equally valid outputs (Zeng et al., 2025), whether through alternative analytical methods, different but appropriate data selections, or varied presentation choices. Validation mecha- nisms should accommodate this variation rather than penalizing correct responses that diverge from a single gold standard (Gu et al., 2024b; Zeng et al., 2025). This requires either curating sets of valid reference outputs or employing validation approaches capable of assessing correctness beyond exact matching. 5.3.2 Output-Level Validation Output-level validation assesses the final output out(r) produced by executing a realization r against one or more reference outputs or a set of success criteria. This disregards the underlying analytical procedure captured in r, which simplifies automated evaluation but makes it impossible to distinguish correct reasoning from coincidentally correct answers (Liu et al., 2024b). Exact matching. Exact matching compares the output against a desired reference output under an equiv- alence function. Since validity is only confirmed if the system’s output exactly matches the reference output, the output must follow the exact form of the desired output. Different approaches have been explored to constrain the output space. Multiple-choice questions represent the most restrictive approach, limiting valid outputs to a predefined set of options (Lu et al., 2022; Li et al., 2025b). This makes complex analytical questions easily verifiable, but the few choices may allow systems to take shortcuts and guess instead of de- riving the correct output (Balepur et al., 2025). Format-prompting similarly constrains outputs by requiring responses to follow specific templates, such as @answer[value], which can be parsed and verified auto- matically (Hu et al., 2024). These constraints simplify validation at the cost of limiting the expressiveness of responses and potentially leaking information about expected answers. Another approach is to parse the outputs into a standardized format before matching. For instance, Huang et al. (2024) parse plotting scripts to extract numerical data and plotting parameters into numpy and JSON formats for comparison. Exact matching also often requires some cleaning of outputs to account for permutations in lists, different format- ting (e.g., “16€” vs. “16.00”), and further insignificant variations (Egg et al., 2025). Exact matching is the most common validation function used across the analyzed benchmarks. Many benchmarks from tabular question answering are designed to allow only a single value that exactly matches the output (Pasupat & Liang, 2015; Lu et al., 2022; Zhang et al., 2023c; Herzig et al., 2021; Kweon et al., 2023; Chen et al., 2021; Zhao et al., 2022; Zou et al., 2026; Contalbo et al., 2025; Katsis et al., 2022), whereas benchmarks rooted in text-to-SQL generation often employ execution accuracy (Yu et al., 2018; Li et al., 2023; Wang et al., 2025d; Kumar et al., 2024; Chen et al., 2025c), defining an equivalence function that compares the execution results of SQL statements with expected results, sometimes in a column- or row-order invariant manner. Execution accuracy cannot differentiate between semantically distinct queries that incidentally yield the same result (Klopfenstein et al., 2026). Exact matching extends to insight needs that admit multiple valid outputs by crediting a match against any one of them, though this extension is not employed in any analyzed benchmark. To make exact matching more fine-grained, Hu et al. (2024) propose decomposing the output into multiple components, matching each against its own reference value and aggregating the results into a score. 46 Similarity-based matching. Similarity-based matching relaxes the exact-equivalence requirement by ap- plying a similarity measure to compare a system’s output to a reference output. Different similarity measures have been proposed for individual output modalities, ranging from numeric tolerances that absorb floating- point differences (Egg et al., 2025), through fuzzy matching of tables that tolerates additional rows and columns (Yin et al., 2023) and image similarity for rendered charts (Li et al., 2025b), to lexical and seman- tic overlap measures such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2019) for free-form text (Nan et al., 2022; Wu et al., 2025b). However, these similarity metrics for free-form text have been found to correlate poorly with human judgments of free-form outputs (Wolff & Hulsebos, 2025; Wang et al., 2024a). For predictive insight needs, similarity-based matching can be applied to the model used for making the prediction, for instance by calculating the relative performance gap to a well-performing baseline model (Huang et al., 2024; Jing et al., 2025). Similarity-based matching generalizes to sets of reference outputs analogously to exact matching, crediting the best-matching reference. LLM-as-Judge. To handle genuinely open-ended outputs, recent benchmarks increasingly employ large language models as evaluators, prompted with the output, a reference output, and instructions that specify the evaluation criteria (Gu et al., 2024b; Chen et al., 2025g; Xu et al., 2025; Wolff & Hulsebos, 2025; Lai et al., 2023; Lei et al., 2025b; Sahu et al., 2024; Dutta et al., 2026; Jing et al., 2025). The flexibility of these instructions enables specifying conditions that accept semantically equivalent outputs, multiple valid realizations, or fine-grained success criteria, offering flexibility beyond what surface metrics afford. Judges are structured in different ways to sharpen this signal, decomposing the evaluation criteria into fine-grained rubrics combined into a weighted score (Lei et al., 2025b), augmenting the instructions with instance-specific context and criteria (Kim et al., 2025), or decomposing the output itself into atomic claims verified individually against the tables (Seo et al., 2025). TOPBench (Ji et al., 2026) further guards against hallucination-induced judgment errors through a multi-step approach that directly matches quotes from the result to expected outputs. A single judge can also span several output modalities at once to account for different ways of presenting the analytical knowledge (Dutta et al., 2026). Rather than scoring outputs in isolation, judges are also prompted to compare two outputs against each other, either by expressing a preference between two analyses (Wu et al., 2024b) or by rating one relative to the other on aspects such as readability, analytical depth, and visualization (Lei et al., 2025b). While offering greater flexibility, LLM judges may also introduce biases, cost, complexity, and reproducibility concerns (Egg et al., 2025). While Wolff & Hulsebos (2025) find a high accuracy of LLM-judges for simple, descriptive insight needs, Hu et al. (2024) only find a 67% consistency between human experts and LLM-judges for more complex insight needs beyond simple description, underscoring the need to validate the alignment of LLM judges with human judgment for the specific setting in which they are applied (Zhu et al., 2025b; Wang et al., 2024a). Programmatic validation. Instead of relying on predefined desired outputs, programmatic validation introduces flexible, task-specific success criteria in the form of test cases capturing conditions the output must satisfy (Chen et al., 2025g). These test cases are usually expressed in code and can check aspects like verifying that a value falls within acceptable bounds (Chen et al., 2025g), or that a derived output exhibits specific features (Lei et al., 2025a; Wu et al., 2025b), or that the output matches any of multiple patterns. Programmatic validation provides unambiguous and reproducible assessment but requires outputs amenable to automated testing and test cases specified per evaluation instance. Human evaluation. Instead of automated scoring functions, human annotators are employed to rate outputs. Human evaluation can be applied to judge the outputs of individual evaluation instances against explicit criteria (Nan et al., 2022; Yu et al., 2019) or for indicating a pairwise preference for one analysis over another (Wu et al., 2024b; Lei et al., 2025b). Human evaluation makes the evaluation more flexible and can adapt to instance-specific particularities or faults outside anticipated criteria, but it introduces cost in terms of scalability and reproducibility, confining human evaluation to relatively small samples (Nan et al., 2022; Yu et al., 2019; Zhang et al., 2023c; Wu et al., 2024b) or to a fallback where automated matching is brittle, as demonstrated in CRT-QA (Zhang et al., 2023c), where it absorbs formatting errors that exact matching would otherwise penalize. Output-level validation treats the realization as a black box, meaning systems can arrive at correct outputs through flawed analysis or bypass data evidence entirely (Liu et al., 2024b), and partially correct analyses receive no credit. Realization-level validation addresses these gaps by inspecting components of r directly. 47 5.3.3 Realization-Level Validation Instead of narrowly focusing on the final output, realization-level validation assesses the analytical process captured in r itself. It thereby allows distinguishing correct from incorrect analysis, regardless of the result, and allows more fine-grained scoring that can give partial credit to partially correct analysis. The central challenge in realization-level validation lies in surfacing components of r in a form that allows automated checking, either by requiring systems to declare specific artifacts alongside their output or by extracting them from the executed realization. Program comparison. Similar to similarity-based matching at the output-level, program comparison directly applies a similarity function to compare the operations and their composition captured in r against those in a reference realization r ∗ , where both r and r ∗ are represented as code programs. The similarity measure can be algorithmic, such as CodeBLEU (Ren et al., 2020), or learned, such as CodeBERTScore (Zhou et al., 2023). Learned measures can be more permissive of syntactic variation by operating on learned representations of code (Zhou et al., 2023). Among surveyed works, algorithmic n-gram measures such as ROUGE and BLEU (Wu et al., 2025a; Kumar et al., 2024) are applied to compare generated with reference SQL, while COTA (Li et al., 2025b) applies a learned embedding measure to general analysis code. Program comparison is constrained by its reliance on predefined reference realizations, which conflicts with valid variations in structuring and composing realizations where even semantically equivalent programs can diverge substantially in surface form (Pinna et al., 2025). It further requires that the realization is fully expressed as a code artifact, limiting the applicability to systems that emit such programs. Structural matching. Rather than comparing realizations as textual programs, structural matching oper- ates directly on the abstraction of realizations into operations and their composition, measuring the similarity with a reference realization at that level. In text-to-SQL, structural matching has a long lineage in the form of component matching, which decomposes both the predicted and reference SQL into clauses and compares them, either as an exact set match (Yu et al., 2018; Lee et al., 2021) or with partial credit over parsed query trees (Kumar et al., 2024), though such comparisons are tied to the fixed clause structure of SQL. BLADE (Gu et al., 2024b) extends structural matching beyond SQL to general data transformations repre- sented in a graph, with nodes corresponding to discrete data transformations (e.g., filter, groupby, derive,. . . ) and edges expressing data flows between them. It introduces two similarity measures. Value-based matching executes the transformations and compares intermediate column values at corresponding nodes, matching se- quences of operations that produce equivalent data regardless of how they are expressed. Graph isomorphism matching compares the topology and node labels of the graph, accepting structurally equivalent transfor- mations that may produce different values. These similarity measures capture complementary aspects, with value-based matching allowing syntactic diversity given identical results and graph matching tolerating dif- ferences in values under a consistent analytical structure. Generally, structural matching provides more flexibility than program comparison, but requires the realizations to be surfaced in a form that supports the abstraction. BLADE (Gu et al., 2024b) addresses this by employing an LLM to convert transformation code into transform units, which introduces an additional source of error and reliability concerns. Component validation. Component validation decomposes the realization into a set of components and validates each individually against a set of per-instance requirements, aggregating the results into a compos- ite score. Aggregation is commonly an average over the requirements, which allows partial credit, though CausalReasoningBenchmark (Sawarni et al., 2026) instead credits a realization only if it satisfies every requirement. Unlike program comparison and structural matching, the realization is not compared to a ref- erence realization, but checks are applied to determine whether specific analytical artifacts are present and correct. Component validation approaches vary in what components they extract from the realization and how they perform checks on top of them. Components may be defined as stages of a predefined analytical pipeline (Chen et al., 2025g), or task-specific key functionalities that a correct analysis must contain (Lai et al., 2026). This decomposition can also surface how errors propagate through dependent operations by aggregating individual judgments at different steps of the analysis (Lei et al., 2025b). In contrast, Gu et al. (2024b) require agents to declare specific analytical decisions (independent, dependent, and control vari- ables) alongside their code, matching these declarations against a reference through LLM-judged semantic equivalence. Statistical models are similarly extracted from code into natural-language specifications and matched semantically. CausalReasoningBenchmark (Sawarni et al., 2026) similarly compares a set of con- 48 ditions within the realizations with annotations on the desired realization. Through the decomposition of realizations into components, component validation yields fine-grained validation scores that not only in- dicate the end-to-end performance but also surface detailed information on error cases. Depending on the design of the decomposition and checks, component validation can support different valid realizations (Gu et al., 2024b). Component validation inherits the reliability concerns of LLM-as-judge approaches. It further requires expensive curation of per-instance component sets, which must be validated to ensure completeness and correctness. The cost of curation scales with the complexity of the tasks and the number of components per task, which has so far limited the scale of benchmarks. 5.3.4 Combining Validation Signals Validation mechanisms are not mutually exclusive. A benchmark may combine signals from different targets or different mechanisms to capture a broader or more robust validation signal than any single function (Pinna et al., 2025), or to support different output modalities with appropriate functions (Wu et al., 2025b; Egg et al., 2025). Functions can either be aggregated into a single, mixed-target composite score, or deployed separately to collect distinct, multi-criteria validation signals. Mixed-target composites apply and aggregate the results of different validation scores into a single score, acting as an ensemble of validation functions. Output- and realization-level functions can be combined to capture complementary failure modes, where outputs may be correct despite a divergent realization or the realization may be sound despite differences in outputs (Pinna et al., 2025). A composite score simplifies comparisons to a single score at the cost of diagnostic granularity, as a mid-range score does not identify which component failed. Multi-criteria reporting maintains separate scores from distinct validation functions rather than collapsing them into one and is commonly employed among the surveyed benchmarks (Lai et al., 2026; Ji et al., 2026; Gu et al., 2024b). This enables a fine-grained attribution of performance while complicating cross-system comparisons, and can deliberately isolate distinct failure modes by scoring realization-level and output-level signals apart to separate errors in the analysis from errors in its execution (Sawarni et al., 2026). Assessed against the desiderata, scalability and reproducibility are met through automatic evaluation pro- tocols that do not rely on human judgment, although the shift towards LLM-based judging reintroduces variance over directly calculated metrics. Coverage of output modalities is narrow, as many benchmarks simplify evaluation by only assessing raw output values or tables, while most others are restricted to text outputs. Only DAComp (Lei et al., 2025b) evaluates multi-modal outputs that combine text and charts, leaving outputs that combine different modalities mostly uncovered. Robustness and faithfulness are han- dled by confining outputs to specific formats and by replacing flawed similarity metrics with LLM-based assessments, the faithfulness and robustness of which varies by the specific setup and evaluation design. Overall, validation remains concentrated at the output level. Mechanisms that inspect the realization ap- pear in a minority of benchmarks and are concentrated in text-to-SQL component matching, with genuine assessment of open-ended analytical processes confined to a few works such as BLADE (Gu et al., 2024b), KramaBench (Lai et al., 2026), and CausalReasoningBenchmark (Sawarni et al., 2026). Support for multi- ple valid variations is similarly shallow, as most mechanisms that admit more than one valid realization do so only through output equivalence, where execution accuracy credits results that coincide with the refer- ence, whereas crediting genuinely divergent yet valid realizations or outputs is restricted to component- and rubric-based approaches (Gu et al., 2024b; Lei et al., 2025b; Lai et al., 2026). Validation across benchmarks thus predominantly approximates the validity criterion 1[r ∈ R valid (i)] while the actual utility u i (r) remains largely unmeasured. Even where mechanisms produce intermediate scores, these mostly arise from aggre- gating validity checks over components of the output or realization, reflecting partial validity. The utility of different valid realizations remains largely unaddressed, only partially surfacing in rubric-based judges that score aspects of data selection, methodological rigor, and the effectiveness of the presentation explicitly (Lei et al., 2025b; Lai et al., 2026), leaving two systems producing valid realizations of substantially different utility largely indistinguishable under current benchmarks. 49 6 A Research Agenda for OpenTI The preceding sections surface a landscape of related but fragmented work making progress towards OpenTI. Yet, none of the reviewed systems addresses OpenTI end-to-end, and significant gaps remain before such systems can deliver practical value. This section puts forward a research agenda towards closing these gaps and advancing OpenTI towards real-world applications. Successfully implementing (data) systems and democratizing access to insights requires that systems are both reliable and trustworthy (Shneiderman, 2020), and that they provide accessible means for extracting insights (Jagadish et al., 2014). The gaps that separate current work from these ends come from simplifying assumptions in current methods, missing integration between capabilities, evaluation infrastructure that lags behind the systems we wish to build, and a limited perspective on the role of humans in the insight extraction process. We organize the research agenda following these requirements. To holistically address OpenTI end-to-end, systems must reliably retrieve relevant tables from the corpora available to them and address complex analytical needs that reach beyond descriptive insights into inferential and causal ones (Section 6.1). Further, we argue that OpenTI systems need to be trustworthy so that results can be relied upon and that new user interaction paradigms are required to align the analysis with the actual insight need a user has (Section 6.2). Lastly, we take a step back to rethink how systems are built and evaluated, arguing that realizing OpenTI requires new paradigms for both (Section 6.3). 6.1 Addressing Data and Methodological Needs The insight needs a system can satisfy are bounded by the data it can locate, corresponding to the data need i d , and by the analysis it can perform, corresponding to the methodological need i m . 6.1.1 Retrieving Compatible and Relevant Data at Scale Locating sufficient tables from a large corpus is a prerequisite for any downstream operation. As discussed in Section 3.3.2, existing retrieval methods have made progress on alignment between user inputs and individual tables, yet several challenges remain unaddressed. What makes data relevant? Current retrieval methods predominantly operationalize relevance as se- mantic similarity between textual representations of data needs and tables, while accounting for structural compatibility (Section 3.3.2). However, relevance and the fit of tables to the specific task are multidimen- sional. In particular, when multiple sets of tables satisfy semantic and structural criteria, the realizations they enable may exhibit different utility u i . Data quality literature has established such multidimensional relevance frameworks for tabular data (Wang & Strong, 1996; Batini et al., 2009), highlighting aspects like completeness, consistency, and timeliness. Future work should thus define and operationalize quality-aware relevance criteria and integrate them with retrieval. How to design indexes that support the discovery of relevant data? Retrieval methods predomi- nantly build indexes as flat collections of independent table, column, or row representations that are scored against the input (Section 3.3.2). Such indexes treat each entry in isolation and remain agnostic to rela- tionships between tables, data quality criteria, and the analytical use the retrieved data will serve. Instead, future work should explore indexes that are aware of the data they are serving, designed for surfacing relevant data rather than just similar data. How can retrieval scale to heterogeneous real-world corpora? Real-world data environments such as organizational data lakes exhibit large numbers of tables, heterogeneous schemas, inconsistent naming conventions, varying quality, and potentially overlapping content (Nargesian et al., 2019). Maintaining retrieval performance requires methods that are robust to the messiness and ambiguity in such environments. Scaling also amplifies the challenge of distinguishing more from less relevant data. Progress on this is limited by evaluation, since current benchmarks have corpora that trade scale against structural realism (Section 5), leaving methods largely untested against the heterogeneity and ambiguity of the environments they are meant to operate in. 50 6.1.2 Addressing Inferential and Causal Insights Insights span descriptive, inferential, and causal tiers (Section 2.3), yet reviewed systems and evaluations remain mainly concerned with descriptive insights. Inferential and causal insights require deriving knowledge that reaches beyond the observed data, justified against assumptions about the process that produced the data. Current models show a limited capacity to do so across inferential and causal insights. LLMs mainly rely on causal relationships learned from world knowledge (Kiciman et al., 2025) but show limitations in reliably inferring them from evidence (Jin et al., 2024; Liu et al., 2024b). While recent reasoning models narrow this gap, they remain susceptible to bias in instructions (Fu et al., 2026). They also show limitations in building predictive models, with reported tendencies to overfit validation signals (Chan et al., 2024) and difficulties in identifying methodological errors (Kumar et al., 2025). Evaluation is covered unevenly and mostly on a surface level. Within inferential and causal types, prediction is covered most widely, though mostly as a modeling task rather than focused on deriving an insight (Section 5), with TOPBench (Ji et al., 2026) the first to center on predictive insights and CausalReasoningBenchmark (Sawarni et al., 2026) the first to target interventional analysis, while statistical inference appears mostly as one objective among many (Gu et al., 2024b; Huang et al., 2024) and counterfactuals remain limited to simplified rule-based instances (Egg et al., 2025; Zheng et al., 2024a). How can systems establish and verify the data and analysis assumptions? Since the validity of inferential and causal analysis relies on assumptions about the process that generated the data, systems should make these explicit and verify them against the data where possible, and surface the rest to the user to judge. The assumptions differ by tier, as prediction rests on the sample being representative of the targets and free of leakage, statistical inference on the distributional and sampling conditions of the applied test, and causal estimation on identification conditions such as the absence of unobserved confounding. Benchmarks like BLADE (Gu et al., 2024b) take a first step in this direction by requiring systems to declare analytical decisions such as control variables, yet none of the surveyed systems establishes or verifies such assumptions. Treating assumptions as first-class components of the analytical process could also extend interpretability from the provenance of results to their analytical validity (Section 6.2.2). Progress on this is also required to enable realization-level validation of inferential and causal results, which should establish whether a predictive model generalizes beyond its training sample rather than reproducing a stored value, and whether an inferential or causal estimate rests on justified assumptions and appropriate methodology. How to select a methodology that matches the insight need and the data? Inferential and causal needs require choosing among methods whose validity depends on properties of the selected data. Predic- tive needs require selecting a model and features, while interventional needs demand formal identification strategies such as instrumental variables, regression discontinuity, and difference-in-differences, depending on the structural conditions (Sawarni et al., 2026; Wang et al., 2025b). In OpenTI, however, the appropriate methodology depends on retrieval and the data binding β it surfaces, that is, whether the corpus supplies the features a model needs or a variable that serves as a valid instrument. Future work should reason jointly over the methodological need i m and the data need i d , retrieving data that enables valid methods rather than fitting methods to whatever data was retrieved. 6.2 Making Systems Trustworthy and Cooperative Beyond addressing the full spectrum of insight needs end-to-end, OpenTI systems should do so in a trust- worthy manner and ensure that they address the actual need a user has. We thus argue that they need to produce robust realizations whose results users can rely upon, and that systems should be designed for cooperative interactions with users to align realizations with the users’ needs. 6.2.1 Ensuring Robustness of Results and Realizations OpenTI systems are intended to provide factual information grounded in data to inform decisions. To make outputs trustworthy for real-world decision-making, systems need to either produce reliable results or transparently communicate their limitations. Automation bias creates the risk of unwarranted reliance on system outputs when the underlying realization is flawed (Parasuraman & Manzey, 2010), for instance, building on incomplete data, inappropriate methods, or compounding errors across operations. 51 How to detect and handle unanswerable insight needs? Not all insight needs are resolvable since corpora may lack sufficient data, inputs may be underspecified beyond what conventional or selective resolu- tion can address, or the request may be logically ill-formed. Work on unanswerability is limited and largely confined to text-to-SQL settings (Li et al., 2025b; Ren et al., 2026; Wang et al., 2023), where unanswerabil- ity is typically reduced to schema-level coverage gaps. The open setting introduces additional challenges of insufficient data quality, insufficient system capabilities to solve a task (e.g., a system that can only inter- act with data through SQL is limited to the operations supported by the specific SQL dialect), and data that supports a result but lacks statistical validity, for instance, when samples are too small or confounders undermine the intended interpretation. Future research should develop taxonomies of unanswerability in OpenTI and equip systems with the capacity to detect, categorize, and communicate such unanswerability accordingly. How can the uncertainty of realizations and outputs be quantified? The reliability of produced realizations may vary substantially. A realization chains multiple operations, each introducing potential uncertainties and errors, for instance from imprecise retrieval and noisy data to inappropriate analytical choices. Errors may compound across the realization, such that individually reasonable steps may yield unreliable results (Zhao et al., 2025). None of the surveyed systems provides systematic means to quantify this uncertainty, presenting outputs without confidence assessment or caveats about analytical limitations. Future work should propose methods that can quantify and propagate uncertainties, assess whether data and methodology support the conclusions, and surface these assessments to users (Kirchhof et al., 2025). How to detect when realizations fail to serve the insight need? A realization can be methodologically sound yet fail to deliver the insight the user actually needs. Users may express their insight need ˆ i in a way that requests analytical knowledge that does not itself fully satisfy the underlying insight need i. For instance, a user interested in understanding the effect of one variable on another may ask for the correlation between them, yet actually require an analysis that establishes the direction and significance of the effect while accounting for confounders to satisfy their underlying insight need. This divergence may be driven by insufficient methodological knowledge, as users articulate high-level analytical goals without identifying the concrete analytical operations that would satisfy them (Grammel et al., 2010; Amar et al., 2005), or by cognitive biases that influence how needs are framed, for instance, in users who seek analytical knowledge that confirms a prior belief (Lehner et al., 2008; Cook & Smallman, 2008), where a system that faithfully answers may introduce or reinforce misinterpretations rather than contributing factual insights. Systems should thus recognize when an accurate response to ˆ i would not serve the underlying need, communicate this divergence, and propose better-suited analyses, requiring reasoning about the latent need beyond its surface expression. 6.2.2 Enabling Cooperative Human-System Interaction Section 4 establishes insight extraction as a cooperative process in which users deliberately delegate unspec- ified aspects of their insight need to the system and both parties carry obligations of revealment. Research into analytical systems has traditionally placed the burden of alignment on the user, who is expected to form an accurate mental model of what the system can do and how it interprets inputs. Following the obligation for mutual revealment formulated by Radlinski & Craswell (2017) and the division of labor argued for by Gomm et al. (2025), we instead regard this alignment as a mutual responsibility in which the system main- tains and refines a model of the user and the context their insight needs arise in, just as the user builds a model of the system’s data and capabilities. Progressing towards such mutual alignment requires work on interaction modes, interpretability mechanisms, and contextual knowledge. What interaction modes best serve users? Complex analytical workloads may require iterative ex- change, as argued in Section 4.1. Yet, system-initiated and collaborative interaction modes remain largely conceptual. Future research should investigate which interaction modes are most effective for different combi- nations of task complexity and user expertise, and develop systems that support richer interaction protocols. This requires extensive user studies and novel approaches to validate the accuracy of systems under these interaction protocols. 52 How to make realizations accessible and interpretable to users? Users verify system outputs by assessing both the operations performed and the data involved (Gu et al., 2024a; Xie et al., 2024). Making the realization r explicit, interpretable, and actionable for intervention is essential to delegate control to users, fostering trust and enabling users to judge the reliability of realizations (Shneiderman, 2020). We argue for future work drawing on data provenance and lineage research (Buneman et al., 2001; Cheney et al., 2009), surfacing data flows and operations at appropriate levels of abstraction. This requires balancing comprehensiveness with cognitive effort (Kazemitabaar et al., 2024) and developing representations that serve both verification and direct intervention. How to adapt insight extraction to its context? Tabular data mostly exists in organizational settings where analytical tasks are deeply contextualized, with domain-specific vocabulary, conventions, implicit constraints, policies, and regulations (Cao & Zhang, 2007; Egg et al., 2025) shaping how insight needs should be interpreted. Operating within this context reduces the degree to which insight needs must be explicitly specified, as shared understanding resolves ambiguities that would otherwise require specification. Instead of burdening users with accounting for the system’s missing contextual knowledge through redundant specification, which users embedded in a domain may not even recognize as necessary since much of this context is tacit (Polanyi, 1967), we argue that systems should acquire and maintain accurate models of organizational and domain context, facilitating efficient interactions by reducing the requirements on input specification. Beyond the organizational level, systems should also adapt to the specific user since different users bring distinct expertise, expectations, and analytical perspectives (Zhang et al., 2020a), implying that similar insight needs may require different realizations depending on who expresses them. Similarly, users may regard their interaction histories as shared context, under which they express their insight needs (Yin et al., 2023). Such contextual knowledge could be informed by organizational documentation, domain ontologies, interaction histories, or user profiles. How to determine appropriate output presentations? The output need i o is frequently left unspec- ified by users, yet the choice of output modality, like textual summaries, tables, charts, or a combination thereof, may substantially affect the utility of a realization. Surveyed systems either rely on explicit output specification in the input (Hu et al., 2024; Huang et al., 2024) or default to a fixed modality. Future work should explore inferring appropriate output presentations from the analytical context, the nature of the results, and user expectations, making the output need a first-class component of the realization process. 6.3 Rethinking How Systems Are Built and Evaluated The preceding directions ask what systems should be able to do. We now shift the focus to how the paradigms that current systems inherit from the narrow task framings they are built and benchmarked for need to adapt to support this, spanning how analytical work is orchestrated, what models carry it out, and how systems are evaluated. 6.3.1 Dynamically Orchestrating Analytical Workloads For simple insight needs, the analytical process can often be determined from the input alone. For instance, a lookup usually only requires locating the right table and extracting a value. However, as the complexity of insight needs increases, the realization that satisfies them becomes increasingly harder to specify upfront. Intermediate results may reveal that an initially chosen analytical method is unsuitable for the data, that required variables are distributed across tables not yet retrieved, or that the initial plan has a flaw. User inputs throughout processing also require mechanisms to adjust the analytical process. While reactive systems demonstrate initial capabilities in this direction (Zhang et al., 2024b; Wang et al., 2024b), their adaptivity remains largely confined to selecting among predefined operations, with data bindings β fixed upfront and limited capacity for backtracking and revising the analytical strategy. Advancing OpenTI towards complex workloads requires orchestration that can dynamically adapt all components of a realization r = (O,E,β) in response to intermediate results and interactions. How to acquire data during the analytical process? Most existing systems operating in an open setting separate retrieving data from the downstream usage of that data in a two-stage design (Kong et al., 2023; Chen et al., 2021), requiring all relevant tables to be identified before any data operations take place. 53 Yet complex insight needs may reveal data requirements only as the analysis unfolds, for instance, when in- termediate results require linking tables or additional reference data not initially anticipated. While research into dynamic retrieval in retrieval augmented generation (Jiang et al., 2023; Asai et al., 2024) highlights the potential of retrieving data when it becomes relevant, initial steps by Wang & Li (2025) in applying this paradigm to OpenTI fall short of systematically representing and utilizing intermediate analytical context for retrieval, or of addressing data compatibility with intermediate results. How can user input be integrated during processing? As discussed in Section 4.1, interaction modes beyond single-input remain underexplored. From an orchestration perspective, this requires systems to recognize the risk of producing misaligned realizations when proceeding with an interpretation that is uncertain due to ambiguity or data challenges, and to collect user input instead (Gomm et al., 2025; Saparina & Lapata, 2025b). This demands mechanisms to estimate confidence in interpretations and to incorporate intermediate user inputs into the ongoing workflow. How should orchestration adapt to the task? The different types of insights (Section 2.3) have fundamentally different analytical requirements. A lookup may be best served by direct retrieval and a single SQL query, while a prediction may require integrating data from different tables, pre-processing it, building, validating, and applying a model. Current systems apply uniform orchestration strategies regardless of task characteristics. Future work should explore how to route insight needs to appropriate orchestration strategies based on the complexity and requirements of the analytical workload, potentially drawing on task classification or complexity estimation as an initial step. 6.3.2 Superseding and Complementing Language Models Surveyed systems rely almost exclusively on language models for orchestration, reasoning, and code gener- ation. Yet language models are architecturally mismatched with core aspects of insight extraction. Tables are two-dimensional structures that exhibit column-order invariance (Cong et al., 2023), contain data of various types, and stand in complex relationships to each other. Linearization into text does not capture these attributes, leading to limited performance in understanding tabular data, particularly as table size increases (Wolff & Hulsebos, 2025). Beyond tables, the unfolding analytical process, comprising partial results, transformed tables, and the evolving directed graph of operations, is inherently non-sequential, yet must be communicated to language models through linear context windows. Which architectures fit tabular data and the analytical process? Initial efforts towards table- native models through modified training and architectures (Herzig et al., 2020; Li et al., 2024; Su et al., 2024) have shown limited improvements over general-purpose language models on downstream tasks. Large improvements have been made in recent in-context learning for regression and classification on tabular data, so-called Tabular Foundation Models (TFMs) (Hollmann et al., 2023; Qu et al., 2025; Hollmann et al., 2025). Yet, these advancements have only transferred to insight extraction insofar as TFMs have been proposed as tools for language model agents (Cong et al., 2025). Future research should thus explore architectures and methods that better fit the structure of the analytical process and the data it is applied to. To what degree can single models cover the analytical process? The functional capabilities examined in Section 3.3 span locating, understanding, and transforming tabular data as well as orchestrating, reasoning within, and compiling outputs from the analytical process. It remains an open question to what degree these skills can be effectively covered by single models, and where specialization or composition of dedicated components yields more capable systems. 6.3.3 Designing Realistic and Faithful Evaluations Evaluation methodology must faithfully assess the systems we aim to build. As shown in the analysis in Sec- tion 5, current benchmarks exhibit corpora that trade off scale against structural realism, input specifications suffer from ambiguities and data privileges, tasks remain concentrated on descriptive insights with limited coverage of inferential and causal types, and validation mechanisms predominantly rely on output-level ex- act matching. These limitations mean that benchmark performance may not reflect system capabilities under realistic conditions. Addressing these gaps requires not only more representative datasets but also methodological advances in how system behavior is assessed. 54 How to design robust realization-level verification? Benchmarking systems on their outputs alone risks conflating correct realizations with coincidentally correct answers, which has been shown to happen when systems bypass data evidence and instead guess a response (Liu et al., 2024b). Moreover, for open- ended analytical tasks where multiple valid realizations exist (Gomm et al., 2025; Saparina & Lapata, 2025a), output-level validation against singular gold answers penalizes legitimate alternative interpretations. Realization-level verification, which explicitly examines operations O, their composition E, and data bindings β, offers more fine-grained diagnostic signal and enables partial credit for correct substeps (Chen et al., 2025g). However, this requires systems to expose realizations in standardized forms and demands validation methodology capable of assessing analytical soundness. How to evaluate interactive systems and multimodal outputs? Current evaluations mostly assume a standalone input scored against a fixed reference output (Section 5), an assumption that breaks on both sides as OpenTI advances. On the input side, the cooperative interaction modes argued for in Section 6.2.2 replace the standalone utterance with a composition of inputs provided at different times, where the system’s state at each step shapes the subsequent user input. This interdependence between system and input complicates both the design of evaluation instances and the attribution of performance. Initial benchmarks emulate such interaction through LLM-based user simulation (Dutta et al., 2026; Huo et al., 2025; Li et al., 2025b), yet user models remain prone to goal misalignment and unrealistic response patterns (Seshadri et al., 2026; Mehri et al., 2026), calling for methods that faithfully emulate user behavior. On the output side, realizations increasingly produce results that combine text, tables, and visualizations, demanding validation mechanisms that assess these modalities jointly rather than in isolation (Dutta et al., 2026). 7 Conclusion Most of the knowledge that makes tabular data valuable does not reside in its cells but has to be discovered and derived, imposing barriers to locating and analyzing the relevant data. In this work, we posit Open Tabular Insight Extraction (OpenTI) as the task of making this knowledge accessible to those who need it, treating the problem end-to-end, from the expressed need to presenting the extracted knowledge. We develop OpenTI as an emerging field centered on insights rather than the surface-level answers that narrower framings such as table question answering, text-to-SQL, and data analysis largely optimize for. By formulating the problem around the insight needs that users hold, the realizations that describe how analytical knowledge is extracted from tabular data, and a notion of their utility that reaches beyond literal accuracy, we establish a basis on which we conceptualize systems that address OpenTI as agents that navigate a combinatorially vast realization space under partial observability. On this basis, we recast the human side of the task under a framing of cooperative interaction and formalize the evaluation of such systems. Interleaving this conceptualization with a structured survey lets us take stock of where the field stands. We find that many of the pieces OpenTI requires already exist across communities but remain fragmented and, where present, are mostly assembled around narrow answer-centered targets. Capabilities such as data retrieval, output contextualization, cooperative interaction, and inferential and causal analysis remain thin, and evaluation methodology lags behind the systems we ultimately wish to build. We intend the conceptualization and vocabulary developed here to give the communities working towards these ends a common ground from which to advance, making it possible to understand scattered efforts as facets of one problem and to transfer results across the conventions that have kept them apart. Real- izing OpenTI will require assembling these facets into systems that operate end-to-end, yet we believe it is paramount that such systems are designed and evaluated around the insights they produce rather than accuracy alone. A result that is correct on its face can still mislead when it is interpreted outside the context that gives it meaning, as when an output accurately answers the question a user posed while missing the context for the insight they actually seek. The goal we consider most important to keep in view is therefore that OpenTI systems leave people better informed, producing results that are robust and, just as crucially, contextualized so that users arrive at factual, data-driven conclusions rather than confidently mistaken ones. Realizing this makes the cooperation between user and system a first-class concern in system design rather than a peripheral one. Pursued this way, OpenTI aims to make the knowledge held in tables as accessible as the knowledge we already retrieve from documents, democratizing access to data-driven insights in a trustworthy manner. 55 References GROBID, 2008. URL https://github.com/grobidOrg/grobid. tex.archiveprefix: swh tex.eprint: 1:dir:dab86b296e3c3216e2241968f0d63b68e8209d3c. Nikhil Abhyankar, Vivek Gupta, Dan Roth, and Chandan K. Reddy. H-STAR: LLM-driven Hybrid SQL- Text Adaptive Reasoning on Tables. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 8841–8863, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/ v1/2025.naacl-long.445. URL https://aclanthology.org/2025.naacl-long.445/. Katrin Affolter, Kurt Stockinger, and Abraham Bernstein. A comparative survey of recent natural language interfaces for databases. The VLDB Journal, 28(5):793–819, October 2019. ISSN 0949-877X. doi: 10. 1007/s00778-019-00567-8. URL https://doi.org/10.1007/s00778-019-00567-8. Rishita Agarwal, Himanshu Singhal, Peter Baile Chen, Manan Roy Choudhury, Dan Roth, and Vivek Gupta. REaR: Retrieve, Expand and Refine for Effective Multitable Retrieval, November 2025. URL http: //arxiv.org/abs/2511.00805. arXiv:2511.00805 [cs]. Mubashara Akhtar, Chenxi Pang, Andreea Marzoca, Yasemin Altun, and Julian Martin Eisenschlos. TANQ: An Open Domain Dataset of Table Answered Questions. Transactions of the Association for Computational Linguistics, 13:461–480, 2025. doi: 10.1162/tacl_a_00749. URL https://aclanthology.org/2025. tacl-1.23/. Place: Cambridge, MA. J.E. Allen. Mixed-initiative interaction. IEEE Intelligent Systems and their Applications, 14(5):14–16, September 1999. ISSN 2374-9423. doi: 10.1109/5254.796083. URL https://ieeexplore.ieee.org/ abstract/document/796083. R. Amar, J. Eagan, and J. Stasko. Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization, 2005. INFOVIS 2005., p. 111–117, October 2005. doi: 10.1109/INFVIS.2005.1532136. URL https://ieeexplore.ieee.org/document/1532136. ISSN: 1522-404X. Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to Re- trieve, Generate, and Critique through Self-Reflection. International Conference on Learning Represen- tations, 2024:9112–9141, May 2024. URL https://proceedings.iclr.c/paper_files/paper/2024/ hash/25f7be9694d7b32d5c670927b8091e1-Abstract-Conference.html. Kumaripaba Athukorala, Dorota Głowacka, Giulio Jacucci, Antti Oulasvirta, and Jilles Vreeken. Is ex- ploratory search different? A comparison of information search behavior for exploratory and lookup tasks. Journal of the Association for Information Science and Technology, 67(11):2635–2651, 2016. ISSN 2330- 1643. doi: 10.1002/asi.23617. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.23617. _eprint: https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.23617. Muhammad Imam Luthfi Balaka, David Alexander, Qiming Wang, Yue Gong, Adila Krisnadhi, and Raul Castro Fernandez. Pneuma: Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End System. In Proceedings of the ACM on Management of Data, volume 3 of 3, p. 1–28, New York, NY, USA, April 2025. Association for Computing Machinery. doi: 10.1145/3725337. URL https://doi.org/10.1145/3725337. arXiv:2504.09207 [cs]. Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3394– 3418, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.169. URL https://aclanthology.org/2025.acl-long.169/. 56 Krisztian Balog and ChengXiang Zhai. User Simulation for Evaluating Information Access Systems. Foun- dations and Trends® in Information Retrieval, 18(1-2):1–261, June 2024. ISSN 1554-0669, 1554-0677. doi: 10.1561/1500000098. URL http://arxiv.org/abs/2306.08550. arXiv:2306.08550 [cs.HC]. Carlo Batini, Cinzia Cappiello, Chiara Francalanci, and Andrea Maurino. Methodologies for data quality assessment and improvement. ACM Comput. Surv., 41(3):16:1–16:52, July 2009. ISSN 0360-0300. doi: 10.1145/1541880.1541883. URL https://dl.acm.org/doi/10.1145/1541880.1541883. Leilani Battle and Alvitta Ottley. What Do We Mean When We Say “Insight”? A Formal Synthesis of Existing Theory. IEEE Transactions on Visualization and Computer Graphics, 30(9):6075–6088, Septem- ber 2024. ISSN 1941-0506. doi: 10.1109/TVCG.2023.3326698. URL https://ieeexplore.ieee.org/ document/10294215. Nicholas J Belkin. Anomalous states of knowledge as a basis for information retrieval. Canadian journal of information science, 5(1):133–143, 1980. Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. Benchmarking and Improving Text- to-SQL Generation under Ambiguity. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 7053–7074, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.436. URL https://aclanthology.org/2023.emnlp-main.436/. Alex Bogatu, Alvaro A. A. Fernandes, Norman W. Paton, and Nikolaos Konstantinou. Dataset Discovery in Data Lakes. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), p. 709–720, April 2020. doi: 10.1109/ICDE48307.2020.00067. URL https://ieeexplore.ieee.org/abstract/document/ 9101607. Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. Deep Neural Networks and Tabular Data: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 35(6):7499–7519, June 2024. ISSN 2162-2388. doi: 10.1109/TNNLS.2022.3229161. URL https: //ieeexplore.ieee.org/document/9998482. Pia Borlund.The concept of relevance in IR. Journal of the American Society for In- formation Science and Technology, 54(10):913–925, 2003.ISSN 1532-2890.doi: 10. 1002/asi.10286. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.10286. _eprint: https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.10286. Allaa Boutaleb, Bernd Amann, Rafael Angarita, and Hubert Naacke. Exploring Multi-Table Re- trieval Through Iterative Search. In EurIPS 2025 Workshop: AI for Tabular Data, November 2025. URL https://openreview.net/forum?id=d64whXpOgf&filter=excludedInvitations%3AEurIPS.c% 2F2025%2FWorkshop%2FAITD%2FSubmission41%2F-%2FChat&nesting=3&sort=date-desc. Matthew Brehmer and Tamara Munzner. A Multi-Level Typology of Abstract Visualization Tasks. IEEE Transactions on Visualization and Computer Graphics, 19(12):2376–2385, December 2013. ISSN 1941-0506. doi: 10.1109/TVCG.2013.124. URL https://ieeexplore.ieee.org/abstract/document/ 6634168. Leo Breiman.Statistical Modeling:The Two Cultures.Statistical Science, 16(3): 199–231, August 2001.ISSN 0883-4237, 2168-8745.doi:10.1214/s/1009213726. URL https://projecteuclid.org/journals/statistical-science/volume-16/issue-3/ Statistical-Modeling--The-Two-Cultures-with-comments-and-a/10.1214/s/1009213726.full. Peter Buneman, Sanjeev Khanna, and Tan Wang-Chiew. Why and Where: A Characterization of Data Provenance. In Jan Van den Bussche and Victor Vianu (eds.), Database Theory — ICDT 2001, p. 316–330, Berlin, Heidelberg, 2001. Springer. ISBN 978-3-540-44503-6. doi: 10.1007/3-540-44503-X_20. Michael J. Cafarella, Alon Halevy, Daisy Zhe Wang, Eugene Wu, and Yang Zhang. WebTables: exploring the power of tables on the web. Proc. VLDB Endow., 1(1):538–549, August 2008. ISSN 2150-8097. doi: 10.14778/1453856.1453916. URL https://dl.acm.org/doi/10.14778/1453856.1453916. 57 Longbing Cao and Chengqi Zhang. The Evolution of KDD: Towards Domain-driven Data Mining. In- ternational Journal of Pattern Recognition and Artificial Intelligence, 21(04):677–692, June 2007. ISSN 0218-0014. doi: 10.1142/S0218001407005612. URL https://w.worldscientific.com/doi/abs/10. 1142/S0218001407005612. Yihan Cao, Shuyi Chen, Ryan Liu, Zhiruo Wang, and Daniel Fried. API-Assisted Code Generation for Question Answering on Varied Table Structures. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 14536– 14548, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.897. URL https://aclanthology.org/2023.emnlp-main.897/. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering, October 2024. URL http://arxiv.org/abs/ 2410.07095. arXiv:2410.07095. Remco Chang, Caroline Ziemkiewicz, Tera Marie Green, and William Ribarsky. Defining Insight for Visual Analytics. IEEE Computer Graphics and Applications, 29(2):14–17, March 2009. ISSN 1558-1756. doi: 10.1109/MCG.2009.22. URL https://ieeexplore.ieee.org/abstract/document/4797511. Adriane Chapman, Elena Simperl, Laura Koesten, George Konstantinidis, Luis-Daniel Ibáñez, Emilia Kacprzak, and Paul Groth. Dataset search: a survey. The VLDB Journal, 29(1):251–272, Jan- uary 2020. ISSN 0949-877X. doi: 10.1007/s00778-019-00564-x. URL https://doi.org/10.1007/ s00778-019-00564-x. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to Answer Open- Domain Questions. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1870–1879, Vancou- ver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1171. URL https://aclanthology.org/P17-1171/. Kaiwen Chen, Yueting Chen, Nick Koudas, and Xiaohui Yu. Reliable End-to-End Text-to-SQL Gener- ation. In Proceedings 29th International Conference on Extending Database Technology. OpenProceed- ings.org, 2026. doi: 10.48786/EDBT.2026.44. URL https://openproceedings.org/2026/conf/edbt/ paper-177.pdf. Ke Chen, Peiran Wang, Yaoning Yu, Xianyang Zhan, and Haohan Wang. Large Language Model-based Data Science Agent: A Survey, November 2025a. URL http://arxiv.org/abs/2508.02744. arXiv:2508.02744 [cs]. Mengshi Chen, Yuxiang Sun, Tengchao Li, Jianwei Wang, Kai Wang, Xuemin Lin, Ying Zhang, and Wenjie Zhang. Empowering Tabular Data Preparation with Language Models: Why and How?, August 2025b. URL http://arxiv.org/abs/2508.01556. arXiv:2508.01556 [cs]. Peter Baile Chen, Yi Zhang, and Dan Roth. Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, p. 2687–2699, Bangkok, Thailand, June 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.148. URL https://aclanthology.org/2024.acl-long.148/. Peter Baile Chen, Fabian Wenz, Yi Zhang, Devin Yang, Justin Choi, Nesime Tatbul, Michael Cafarella, Çağatay Demiralp, and Michael Stonebraker. BEAVER: An Enterprise Benchmark for Text-to-SQL, January 2025c. URL http://arxiv.org/abs/2409.02038. arXiv:2409.02038 [cs]. Peter Baile Chen, Yi Zhang, Mike Cafarella, and Dan Roth. Can we Retrieve Everything All at Once? ARM: An Alignment-Oriented LLM-based Retrieval Method. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 30298–30317, Vienna, Austria, July 2025d. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1463. URL https://aclanthology.org/2025.acl-long.1463/. 58 Si-An Chen, Lesly Miculicich, Julian Martin Eisenschlos, Zifeng Wang, Zilong Wang, Yanfei Chen, Yasuhisa Fujii, Hsuan-Tien Lin, Chen-Yu Lee, and Tomas Pfister. TableRAG: million-token table understanding with language models. In Proceedings of the 38th International Conference on Neural Information Pro- cessing Systems, volume 37 of NIPS ’24, p. 74899–74921, Red Hook, NY, USA, December 2024b. Curran Associates Inc. ISBN 979-8-3313-1438-5. Wei-Hao Chen, Weixi Tong, Amanda Case, and Tianyi Zhang. Dango: A Mixed-Initiative Data Wrangling System using Large Language Model. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, p. 1–28, New York, NY, USA, April 2025e. Association for Computing Machinery. ISBN 979-8-4007-1394-1. doi: 10.1145/3706598.3714135. URL https://dl.acm.org/doi/ 10.1145/3706598.3714135. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. Open Question Answering over Tables and Text. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=MmCRswl1UYl. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching Large Language Models to Self-Debug. In The Twelfth International Conference on Learning Representations, October 2023. URL https://openreview.net/forum?id=KuPixIqPiq. Yongrui Chen, Junhao He, Linbo Fu, Shenyu Zhang, Rihui Jin, Xinbang Dai, Jiaqi Li, Dehai Min, Nan Hu, Yuxin Zhang, Guilin Qi, Yi Huang, and Tongtong Wu. Pandora: A Code-Driven Large Language Model Agent for Unified Reasoning Across Diverse Structured Knowledge, April 2025f. URL http://arxiv. org/abs/2504.12734. arXiv:2504.12734 [cs]. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. In The Thirteenth International Conference on Learning Representations, 2025g. URL https://openreview.net/forum?id=6z4YKr0GK6. James Cheney, Laura Chiticariu, and Wang-Chiew Tan. Provenance in Databases: Why, How, and Where. Foundations and Trends in Databases, 1(4):379–474, June 2009. ISSN 1931-7883. doi: 10.1561/1900000006. URL https://doi.org/10.1561/1900000006. Yongdong Chi, Hanqing Wang, Yun Chen, Yan Yang, Jian Yang, Zonghan Yang, Xiao Yan, and Guanhua Chen. Pi-SQL: Enhancing Text-to-SQL with Fine-Grained Guidance from Pivot Programming Languages. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, p. 25120–25144, Suzhou, China, Novem- ber 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025. findings-emnlp.1369. URL https://aclanthology.org/2025.findings-emnlp.1369/. Bhavya Chopra, Ananya Singha, Anna Fariha, Sumit Gulwani, Chris Parnin, Ashish Tiwari, and Austin Z. Henley. Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportu- nities, October 2023. URL http://arxiv.org/abs/2310.16164. arXiv:2310.16164 [cs]. Herbert H. Clark and Susan E. Brennan. Grounding in communication. In Perspectives on socially shared cognition, p. 127–149. American Psychological Association, Washington, DC, 1991. ISBN 978-1-55798- 121-9. doi: 10.1037/10096-006. E. F. Codd. A relational model of data for large shared data banks. Commun. ACM, 13(6):377–387, June 1970. ISSN 0001-0782. doi: 10.1145/362384.362685. URL https://dl.acm.org/doi/10.1145/362384. 362685. Jacob Cohen. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46, April 1960. ISSN 0013-1644. doi: 10.1177/001316446002000104. URL https://doi.org/ 10.1177/001316446002000104. 59 Tianji Cong, Madelon Hulsebos, Zhenjie Sun, Paul Groth, and H. V. Jagadish. Observatory: Characterizing Embeddings of Relational Tables. Proceedings of the VLDB Endowment, 17(4):849–862, December 2023. ISSN 2150-8097. doi: 10.14778/3636218.3636237. URL https://dl.acm.org/doi/10.14778/3636218. 3636237. Tianji Cong, Julian Martin Eisenschlos, Daniel Gomm, Leo Grinsztajn, Andreas C. Mueller, Anupam Sanghi, Jan-Micha Bodensohn, Vadim Borisov, Michael Cochez, Katharina Eggensperger, Floris Geerts, Myung Jun Kim, Andreas Kipf, Xue Li, Olga Ovcharenko, Paolo Papotti, Lennart Purucker, Sebastian Schelter, Immanuel Trummer, Gaël Varoquaux, Liane Vogel, Carsten Binnig, Madelon Hulsebos, and Frank Hutter. Unlocking the Full Potential of Data Science Requires Tabular Foundation Models, Agents, and Humans, May 2025. URL https://openreview.net/forum?id=aXMPvmBAm5. Michele Luca Contalbo, Sara Pederzoli, Francesco Del Buono, Venturelli Valeria, Francesco Guerra, and Matteo Paganelli. GRI-QA: a Comprehensive Benchmark for Table Question Answering over Envi- ronmental Data. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pile- hvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, p. 15764–15779, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.814. URL https://aclanthology.org/2025.findings-acl.814/. Maia B. Cook and Harvey S. Smallman. Human Factors of the Confirmation Bias in Intelligence Anal- ysis: Decision Support From Graphical Evidence Landscapes. Human Factors, 50(5):745–754, October 2008. ISSN 0018-7208. doi: 10.1518/001872008X354183. URL https://journals.sagepub.com/action/ showAbstract. W. S. Cooper. A definition of relevance for information retrieval. Information Storage and Retrieval, 7(1):19– 37, June 1971. ISSN 0020-0271. doi: 10.1016/0020-0271(71)90024-6. URL https://w.sciencedirect. com/science/article/pii/0020027171900246. Susan B. Davidson and Juliana Freire. Provenance and scientific workflows: challenges and opportunities. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, SIGMOD ’08, p. 1345–1350, New York, NY, USA, June 2008. Association for Computing Machinery. ISBN 978-1-60558- 102-6. doi: 10.1145/1376616.1376772. URL https://dl.acm.org/doi/10.1145/1376616.1376772. Randall Davis. Meta-rules: Reasoning about control. Artificial Intelligence, 15(3):179–222, December 1980. ISSN 0004-3702. doi: 10.1016/0004-3702(80)90043-0. URL https://w.sciencedirect.com/science/ article/pii/0004370280900430. Randall Davis and Bruce C Buchanan. Meta-Level Knowledge: Overview and Applications. In Proceeding of the International Joint Conference on Artificial Intelligence, 1977. URL https://w.ijcai.org/ Proceedings/77-2/Papers/079.pdf. Tijl De Bie, Luc De Raedt, José Hernández-Orallo, Holger H. Hoos, Padhraic Smyth, and Christopher K. I. Williams. Automating data science. Communications of the ACM, 65(3):76–87, February 2022. ISSN 0001-0782. doi: 10.1145/3495256. URL https://dl.acm.org/doi/10.1145/3495256. Ewa Deelman, Dennis Gannon, Matthew Shields, and Ian Taylor. Workflows and e-Science: An overview of workflow system features and capabilities. Future Generation Computer Systems, 25(5):528–540, May 2009. ISSN 0167-739X. doi: 10.1016/j.future.2008.06.012. URL https://w.sciencedirect.com/science/ article/pii/S0167739X08000861. Chao Deng, Ju Fan, Yuyu Luo, Qinliang Xue, Meihao Fan, Yuxin Zhang, Min Zhang, Xiaofeng Jia, and Jing Zhang. TACO: A Benchmark for Open-Domain Text-to-SQL with Ambiguous and Cross-Database Queries. In Proceedings of the VLDB Endowment, volume 19, 2026. doi: 10.14778/3797919.3797942. URL https://doi.org/10.14778/3797919.3797942. Amol Deshpande, Zachary Ives, and Vijayshankar Raman. Adaptive query processing. Found. Trends databases, 1(1):1–140, January 2007. ISSN 1931-7883. 60 Rui Ding, Shi Han, Yong Xu, Haidong Zhang, and Dongmei Zhang. QuickInsights: Quick and Automatic Discovery of Insights from Multi-Dimensional Data. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD ’19, p. 317–332, New York, NY, USA, June 2019. Association for Computing Machinery. ISBN 978-1-4503-5643-5. doi: 10.1145/3299869.3314037. URL https://dl.acm. org/doi/10.1145/3299869.3314037. Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, Patrick Ng, and Zhiguo Wang. PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 255–273, Albuquerque, New Mexico, April 2025. Association for Com- putational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.13. URL https: //aclanthology.org/2025.naacl-long.13/. Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 35212– 35290, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332- 6. doi: 10.18653/v1/2025.emnlp-main.1786. URL https://aclanthology.org/2025.emnlp-main.1786/. Avik Dutta, Priyanshu Gupta, Hosein Hasanbeig, Rahul Pratap Singh, Harshit Nigam, Sumit Gulwani, Arjun Radhakrishna, Gustavo Soares, and Ashish Tiwari. ConDABench: Interactive Evaluation of Language Models for Data Analysis. In First NeurIPS Workshop on Multi-Turn Interactions in Large Language Models, 2026. URL https://openreview.net/forum?id=1Ynbsc8e7F#discussion. Till Döhmen, Radu Geacu, Madelon Hulsebos, and Sebastian Schelter. SchemaPile: A Large Collection of Relational Database Schemas. Proc. ACM Manag. Data, 2(3):172:1–172:25, May 2024. doi: 10.1145/ 3654975. URL https://dl.acm.org/doi/10.1145/3654975. Alex Egg, Martin Iglesias Goyanes, Friso Kingma, Andreu Mora, Leandro von Werra, and Thomas Wolf. DABstep: Data Agent Benchmark for Multi-step Reasoning, June 2025. URL http://arxiv.org/abs/ 2506.23719. arXiv:2506.23719 [cs]. Xi Fang, Weijie Xu, Fiona Anting Tan, Jiani Zhang, Ziqing Hu, Yanjun Qi, Scott Nickleach, Diego Socol- insky, Srinivasan Sengamedu, and Christos Faloutsos. Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding – A Survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=IZnrCGF9WI. Juliana Freire, David Koop, Emanuele Santos, and Cláudio T. Silva. Provenance for Computational Tasks: A Survey. Computing in Science & Engineering, 10(3):11–21, May 2008. ISSN 1558-366X. doi: 10.1109/ MCSE.2008.79. URL https://ieeexplore.ieee.org/abstract/document/4488060. Stephen N. Freund, Brooke Simon, Emery D. Berger, and Eunice Jun. Flowco: Mixed-Initiative Authoring of Reliable End-to-End Data Analyses via Dataflow Graphs and LLMs. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, UIST ’25, p. 1–20, New York, NY, USA, September 2025. Association for Computing Machinery. ISBN 979-8-4007-2037-6. doi: 10.1145/3746059. 3747636. URL https://dl.acm.org/doi/10.1145/3746059.3747636. Zhizhang Fu, Guangsheng Bao, Hongbo Zhang, Chenkai Hu, and Yue Zhang. Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Process. IEEE Transactions on Audio, Speech and Language Processing, 34:2986–2999, 2026. ISSN 2998-4173. doi: 10.1109/TASLPRO.2026. 3688949. URL https://ieeexplore.ieee.org/abstract/document/11499439. Yujian Gan, Xinyun Chen, and Matthew Purver. Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 61 p. 8926–8931, Online and Punta Cana, Dominican Republic, November 2021. Association for Compu- tational Linguistics. doi: 10.18653/v1/2021.emnlp-main.702. URL https://aclanthology.org/2021. emnlp-main.702/. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided Language Models. In Proceedings of the 40th International Conference on Machine Learning, June 2023. URL https://proceedings.mlr.press/v202/gao23f/gao23f.pdf. Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G. Karahalios. DataTone: Managing Ambiguity in Natural Language Interfaces for Data Visualization. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, UIST ’15, p. 489–500, New York, NY, USA, November 2015. Association for Computing Machinery. ISBN 978-1-4503-3779-3. doi: 10.1145/2807442. 2807478. URL https://dl.acm.org/doi/10.1145/2807442.2807478. Carlos Gemmell and Jeff Dalton. ToolWriter: Question Specific Tool Synthesis for Tabular Data. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 16137–16148, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.1003. URL https://aclanthology.org/ 2023.emnlp-main.1003/. Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning : Theory & Practice. The Morgan Kaufmann Series in Artificial Intelligence. Morgan Kaufmann, Amsterdam, January 2004. ISBN 978-1-55860-856-6. URL https://research.ebsco.com/linkprocessor/plink?id= 6a7bf8a-5f92-3c05-bbbd-30407854aae4. Daniel Gomm and Madelon Hulsebos. Metadata Matters in Dense Table Retrieval. In ELLIS workshop on Representation Learning and Generative Models for Structured Data, February 2025. URL https: //openreview.net/forum?id=rELWIvq2Qy#discussion. Daniel Gomm, Cornelius Wolff, and Madelon Hulsebos. Are We Asking the Right Questions? On Ambiguity in Natural Language Queries for Tabular Data Analysis. In EurIPS 2025 Workshop: AI for Tabular Data, October 2025. URL https://openreview.net/forum?id=Mm1CGvmm9Z. David Gotz and Michelle X. Zhou. Characterizing users’ visual analytic activity for insight provenance. Information Visualization, 8(1):42–55, 2009. doi: 10.1057/ivs.2008.31. URL https://doi.org/10.1057/ ivs.2008.31. tex.eprint: https://doi.org/10.1057/ivs.2008.31. G. Graefe. Volcano/spl minus/an extensible and parallel query evaluation system. IEEE Transactions on Knowledge and Data Engineering, 6(1):120–135, February 1994. ISSN 1558-2191. doi: 10.1109/69.273032. URL https://ieeexplore.ieee.org/document/273032. Lars Grammel, Melanie Tory, and Margaret-Anne Storey. How Information Visualization Novices Construct Visualizations. IEEE Transactions on Visualization and Computer Graphics, 16(6):943–952, November 2010. ISSN 1941-0506. doi: 10.1109/TVCG.2010.164. URL https://ieeexplore.ieee.org/abstract/ document/5613431. Herbert Paul Grice. Logic and coversation. Syntax and semantics, 3:43–58, 1975. Jiawei Gu, Ziting Xian, Yuanzhen Xie, Ye Liu, Enjie Liu, Ruichao Zhong, Mochi Gao, Yunzhi Tan, Bo Hu, and Zang Li. Toward Structured Knowledge Reasoning: Contrastive Retrieval-Augmented Generation on Experience. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, p. 23891–23910, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025. findings-acl.1224. URL https://aclanthology.org/2025.findings-acl.1224/. Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven M. Drucker. How Do Analysts Understand and Verify AI-Assisted Data Analyses? In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, p. 1–22, New York, NY, USA, May 2024a. Association for Computing 62 Machinery. ISBN 979-8-4007-0330-0. doi: 10.1145/3613904.3642497. URL https://dl.acm.org/doi/ 10.1145/3613904.3642497. Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A Merrill, Jeffrey Heer, and Tim Althoff. BLADE: Benchmarking Language Model Agents for Data-Driven Science. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, p. 13936–13971, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.815. URL https://aclanthology. org/2024.findings-emnlp.815/. Jiajing Guo, Vikram Mohanty, Jorge H Piazentin Ono, Hongtao Hao, Liang Gou, and Liu Ren. Investigating Interaction Modes and User Agency in Human-LLM Collaboration for Domain-Specific Data Analysis. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, p. 1–9, Honolulu HI USA, May 2024a. ACM. ISBN 979-8-4007-0331-7. doi: 10.1145/3613905.3651042. URL https: //dl.acm.org/doi/10.1145/3613905.3651042. Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang. DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, p. 36, Vienna, Austria, May 2024b. JMLR.org. doi: 10.5555/3692070.3692738. Yuxiang Guo, Zhonghao Hu, Yuren Mao, Baihua Zheng, Yunjun Gao, and Mingwei Zhou. Birdie: Natu- ral Language-Driven Table Discovery Using Differentiable Search Index. Proceedings of the VLDB En- dowment, 18, April 2025. doi: 10.48550/arXiv.2504.21282. URL http://arxiv.org/abs/2504.21282. arXiv:2504.21282 [cs]. Izzeddin Gür, Dilek Hakkani-Tür, Gokhan Tür, and Pararth Shah. User Modeling for Task Oriented Dia- logues. In 2018 IEEE Spoken Language Technology Workshop (SLT), p. 900–906, December 2018. doi: 10.1109/SLT.2018.8639652. URL https://ieeexplore.ieee.org/abstract/document/8639652. Mingqian He, Yongliang Shen, Wenqi Zhang, Qiuying Peng, Jun Wang, and Weiming Lu. STaR-SQL: Self-Taught Reasoner for Text-to-SQL. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), p. 24365–24375, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1187. URL https://aclanthology.org/2025.acl-long.1187/. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. TaPas: Weakly Supervised Table Parsing via Pre-training. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 4320–4333, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.398. URL https://aclanthology.org/2020.acl-main.398/. Jonathan Herzig, Thomas Müller, Syrine Krichene, and Julian Eisenschlos. Open Domain Question An- swering over Tables via Dense Retrieval. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, p. 512–519, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main.43. URL https://aclanthology.org/ 2021.naacl-main.43. Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=cp5PvcI6w8_. 63 Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular founda- tion model. Nature, 637(8045):319–326, January 2025. ISSN 1476-4687. doi: 10.1038/s41586-024-08328-6. URL https://w.nature.com/articles/s41586-024-08328-6. Zijin Hong, Zheng Yuan, Hao Chen, Qinggang Zhang, Feiran Huang, and Xiao Huang. Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLM. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, p. 10997–11008, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl. 653. URL https://aclanthology.org/2024.findings-acl.653/. Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, CHI ’99, p. 159–166, New York, NY, USA, May 1999. Association for Computing Machinery. ISBN 978-0-201-48559-2. doi: 10.1145/302979.303030. URL https://doi. org/10.1145/302979.303030. Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li, Kun Kuang, Yang Yang, Hongxia Yang, and Fei Wu. InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks. In Proceedings of the 41st International Conference on Machine Learning, p. 19544–19572. PMLR, July 2024. URL https://proceedings.mlr. press/v235/hu24s.html. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large Language Models Cannot Self-Correct Reasoning Yet. In The Twelfth International Confer- ence on Learning Representations, October 2023. URL https://openreview.net/forum?id=IkmD3fKBPQ. Junjie Huang, Wanjun Zhong, Qian Liu, Ming Gong, Daxin Jiang, and Nan Duan. Mixed-modality Repre- sentation Learning and Pre-training for Joint Table-and-Text Retrieval in OpenQA. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, p. 4117–4129, Abu Dhabi, United Arab Emirates, December 2022. Association for Com- putational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.303. URL https://aclanthology.org/ 2022.findings-emnlp.303/. Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, October 2024. doi: 10.18653/v1/2024.emnlp-main. 748. URL https://aclanthology.org/2024.emnlp-main.748/. Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, Kun Shao, and Jun Wang. Deep Research Agents: A Systematic Examination And Roadmap, September 2025. URL http://arxiv.org/abs/2506.18096. arXiv:2506.18096 [cs.AI]. Madelon Hulsebos, Çagatay Demiralp, and Paul Groth. GitTables: A Large-Scale Corpus of Relational Tables. Proc. ACM Manag. Data, 1(1):30:1–30:17, May 2023. doi: 10.1145/3588710. URL https: //dl.acm.org/doi/10.1145/3588710. Nan Huo, Xiaohan Xu, Jinyang Li, Per Jacobsson, Shipei Lin, Bowen Qin, Binyuan Hui, Xiaolong Li, Ge Qu, Shuzheng Si, Linheng Han, Edward Alexander, Xintong Zhu, Rui Qin, Ruihan Yu, Yiyao Jin, Feige Zhou, Weihao Zhong, Yun Chen, Hongyu Liu, Chenhao Ma, Fatma Ozcan, Yannis Papakonstantinou, and Reynold Cheng. BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation via Lens of Dynamic Interactions. In The Fourteenth International Conference on Learning Representations, October 2025. URL https://openreview.net/forum?id=nHrYBGujps. H. V. Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstantinou, Jignesh M. Patel, Raghu Ramakrishnan, and Cyrus Shahabi. Big data and its technical challenges. Commun. ACM, 57(7):86–94, July 2014. ISSN 0001-0782. doi: 10.1145/2611567. URL https://dl.acm.org/doi/10.1145/2611567. 64 Anthony Jameson and Eric Schwarzkopf. Pros and Cons of Controllability: An Empirical Study. In Paul De Bra, Peter Brusilovsky, and Ricardo Conejo (eds.), Adaptive Hypermedia and Adaptive Web-Based Systems, p. 193–202, Berlin, Heidelberg, 2002. Springer. ISBN 978-3-540-47952-9. doi: 10.1007/3-540-47952-X_21. An-Yang Ji, Jun-Peng Jiang, De-Chuan Zhan, and Han-Jia Ye. TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering, June 2026. URL http://arxiv.org/abs/2604. 28076. arXiv:2604.28076 [cs.CL]. Xingyu Ji, Aditya Parameswaran, and Madelon Hulsebos. TARGET: Benchmarking Table Retrieval for Generative Tasks. In NeurIPS 2024 Third Table Representation Learning Workshop, October 2024. URL https://openreview.net/forum?id=gGGvnjFUfL. Changjiang Jiang, Fengchang Yu, Haihua Chen, Wei Lu, and Jin Zeng. TabDSR: Decompose, Sani- tize, and Reason for Complex Numerical Reasoning in Tabular Data. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2025, p. 3172–3196, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.169. URL https://aclanthology.org/2025.findings-emnlp.169/. Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active Retrieval Augmented Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 7969–7992, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.495. URL https://aclanthology.org/ 2023.emnlp-main.495. Ruochun Jin, Xiyue Wang, Dong Wang, Haoqi Zheng, Yunpeng Qi, Silin Yang, and Meng Zhang. TALON: A Multi-Agent Framework for Long-Table Exploration and Question Answering. In Chris- tos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 27397–27413, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1393. URL https://aclanthology.org/2025.emnlp-main.1393/. Zhijing Jin, Jiarui Liu, Zhiheng Lyu, Spencer Poff, Mrinmaya Sachan, Rada Mihalcea, Mona T. Diab, and Bernhard Schölkopf. Can Large Language Models Infer Causation from Correlation? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= vqIH0ObdqL. Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=DSsSPr0RZJ. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in par- tially observable stochastic domains. Artificial Intelligence, 101(1):99–134, May 1998. ISSN 0004-3702. doi: 10.1016/S0004-3702(98)00023-X. URL https://w.sciencedirect.com/science/article/pii/ S000437029800023X. Thomas Kailath. Linear Systems, volume 156. Prentice-Hall Englewood Cliffs, NJ, 1980. R. E. Kalman. Mathematical Description of Linear Dynamical Systems. Journal of the Society for Industrial and Applied Mathematics Series A Control, 1(2):152–192, January 1963. ISSN 0887-4603. doi: 10.1137/ 0301010. URL https://epubs.siam.org/doi/abs/10.1137/0301010. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI Agents That Matter, July 2024. URL http://arxiv.org/abs/2407.01502. arXiv:2407.01502 [cs]. 65 Benjamin Karer, Hans Hagen, and Dirk J. Lehmann. Insight Beyond Numbers: The Impact of Qualitative Factors on Visual Data Analysis. IEEE Transactions on Visualization and Computer Graphics, 27(2):1011– 1021, February 2021. ISSN 1941-0506. doi: 10.1109/TVCG.2020.3030376. URL https://ieeexplore. ieee.org/document/9241426. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 6769–6781, Online, November 2020. Association for Com- putational Linguistics. doi: 10.18653/v1/2020.emnlp-main.550. URL https://aclanthology.org/2020. emnlp-main.550/. Yannis Katsis, Saneem Chemmengath, Vishwajeet Kumar, Samarth Bharadwaj, Mustafa Canim, Michael Glass, Alfio Gliozzo, Feifei Pan, Jaydeep Sen, Karthik Sankaranarayanan, and Soumen Chakrabarti. AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industry. In Anastassia Louk- ina, Rashmi Gangadharaiah, and Bonan Min (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: In- dustry Track, p. 305–314, Hybrid: Seattle, Washington + Online, July 2022. Association for Compu- tational Linguistics. doi: 10.18653/v1/2022.naacl-industry.34. URL https://aclanthology.org/2022. naacl-industry.34/. Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Zachary Henley, Carina Negreanu, and Advait Sarkar. Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST ’24, p. 1–19, New York, NY, USA, October 2024. Association for Computing Machinery. ISBN 979-8-4007-0628-8. doi: 10.1145/3654777.3676345. URL https://dl.acm.org/doi/ 10.1145/3654777.3676345. Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, p. 39–48. Association for Computing Machinery, June 2020. doi: 10.1145/3397271.3401075. URL https://doi.org/10.1145/3397271.3401075. Rohit Khoja, Devanshu Gupta, Yanjie Fu, Dan Roth, and Vivek Gupta. Weaver: Interweaving SQL and LLM for Table Reasoning. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Pro- cessing, p. 28282–28308, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1436. URL https://aclanthology.org/ 2025.emnlp-main.1436/. Emre Kiciman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal Reasoning and Large Language Models: Opening a New Frontier for Causality. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=mqoxLkX210. Heegyu Kim, Jeon Taeyang, SeungHwan Choi, Seungtaek Choi, and Hyunsouk Cho. FLEX: Expert-level False-Less EXecution Metric for Text-to-SQL Benchmark. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4448–4475, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189- 6. doi: 10.18653/v1/2025.naacl-long.228. URL https://aclanthology.org/2025.naacl-long.228/. Michael Kirchhof, Gjergji Kasneci, and Enkelejda Kasneci. Position: Uncertainty Quantification Needs Reassessment for Large Language Model Agents. In Proceedings of the 42nd International Conference on Machine Learning, p. 81665–81677. PMLR, October 2025. URL https://proceedings.mlr.press/ v267/kirchhof25b.html. 66 Rocky Klopfenstein, Yang He, Andrew Tremante, Yuepeng Wang, Nina Narodytska, and Haoze Wu. SpotIt: Evaluating Text-to-SQL Evaluation with Formal Verification. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=iMkvR2ICSE. Denis Kochedykov, Fenglin Yin, and Sreevidya Khatravath. Conversing with databases: Practical Natural Language Querying. In Mingxuan Wang and Imed Zitouni (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, p. 372–379, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-industry.36. URL https: //aclanthology.org/2023.emnlp-industry.36/. Ryosuke Kohita. Exploring Database Normalization Effects on SQL Generation. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM ’25, p. 5788–5796, New York, NY, USA, November 2025. Association for Computing Machinery. ISBN 979-8-4007-2040-6. doi: 10.1145/3746252.3761583. URL https://dl.acm.org/doi/10.1145/3746252.3761583. Vijay R. Konda and John N. Tsitsiklis. Actor-citic agorithms. In Proceedings of the 13th International Conference on Neural Information Processing Systems, NIPS’99, p. 1008–1014, Cambridge, MA, USA, November 1999. MIT Press. Kezhi Kong, Jiani Zhang, Zhengyuan Shen, Balasubramaniam Srinivasan, Chuan Lei, Christos Faloutsos, Huzefa Rangwala, and George Karypis. OpenTab: Advancing Large Language Models as Open-domain Table Reasoners. In The Twelfth International Conference on Learning Representations, October 2023. URL https://openreview.net/forum?id=Qa0ULgosc9. Mayank Kothyari, Dhruva Dhingra, Sunita Sarawagi, and Soumen Chakrabarti. CRUSH4SQL: Collective Retrieval Using Schema Hallucination For Text2SQL. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 14054–14066, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023.emnlp-main.868. URL https://aclanthology.org/2023.emnlp-main.868/. Rahul Kumar, Amar Raja Dibbu, Shrutendra Harsola, Vignesh Subrahmaniam, and Ashutosh Modi. Book- SQL: A Large Scale Text-to-SQL Dataset for Accounting Domain. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pa- pers), p. 497–516, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.28. URL https://aclanthology.org/2024.naacl-long.28/. Smitha Kumar, Michael A. Lones, Manuel Maarek, and Hind Zantout. Navigating Pitfalls: Evaluating LLMs in Machine Learning Programming Education, May 2025. URL http://arxiv.org/abs/2505.18220. arXiv:2505.18220 [cs.CY]. Vishwajeet Kumar, Yash Gupta, Saneem Chemmengath, Jaydeep Sen, Soumen Chakrabarti, Samarth Bharadwaj, and Feifei Pan. Multi-Row, Multi-Span Distant Supervision For Table+Text Question An- swering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8080–8094, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long. 449. URL https://aclanthology.org/2023.acl-long.449/. Sunjun Kweon, Yeonsu Kwon, Seonhee Cho, Yohan Jo, and Edward Choi. Open-WikiTable : Dataset for Open Domain Question Answering with Complex Reasoning over Table. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, p. 8285–8297, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.526. URL https://aclanthology.org/2023.findings-acl.526. Eugenie Lai, Gerardo Vitagliano, Ziyu Zhang, Sivaprasad Sudhir, Om Chabra, Anna Zeng, Anton A. Zabreyko, Chenning Li, Ferdi Kossmann, Jialin Ding, Jun Chen, Markos Markakis, Matthew Russo, Weiyang Wang, Ziniu Wu, Michael J. Cafarella, Lei Cao, Samuel Madden, and Tim Kraska. Kram- aBench: A Benchmark for AI Systems on Data-to-Insight Pipelines over Data Lakes. In The Fourteenth 67 International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id= fZfUdeCC5X. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: a natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of ICML’23, p. 18319–18345, Honolulu, Hawaii, USA, July 2023. JMLR.org. J. Richard Landis and Gary G. Koch. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1):159–174, 1977. ISSN 0006-341X. doi: 10.2307/2529310. URL https://w.jstor. org/stable/2529310. Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. KaggleDBQA: Realistic Evaluation of Text- to-SQL Parsers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 2261–2273, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.176. URL https: //aclanthology.org/2021.acl-long.176/. Jimin Lee, Ingeol Baek, Byeongjeong Kim, Hyunkyung Bae, and Hwanhee Lee. SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL. In Christos Christodoulopou- los, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 19023–19035, Suzhou, China, November 2025. As- sociation for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.962. URL https://aclanthology.org/2025.emnlp-main.962/. Paul Edward Lehner, Leonard Adelman, Brant A. Cheikes, and Mark J. Brown. Confirmation Bias in Complex Analyses. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 38(3):584–592, May 2008. ISSN 1558-2426. doi: 10.1109/TSMCA.2008.918634. URL https://ieeexplore.ieee.org/abstract/document/4490040. Fangyu Lei, Xiang Li, Yifan Wei, Shizhu He, Yiming Huang, Jun Zhao, and Kang Liu. S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering. In Anna Rogers, Jor- dan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 2: Short Papers), p. 1731–1740, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-short.147. URL https://aclanthology.org/2023.acl-short.147/. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview. net/forum?id=XmProj9cPs. Fangyu Lei, Jinxiang Meng, Yiming Huang, Junjie Zhao, Yitong Zhang, Jianwen Luo, Xin Zou, Ruiyi Yang, Wenbo Shi, Yan Gao, Shizhu He, Zuo Wang, Qian Liu, Yang Wang, Ke Wang, Jun Zhao, and Kang Liu. DAComp: Benchmarking Data Agents across the Full Data Intelligence Lifecycle, December 2025b. URL http://arxiv.org/abs/2512.04324. arXiv:2512.04324 [cs]. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Con- ference on Neural Information Processing Systems, NIPS ’20, p. 9459–9474, Red Hook, NY, USA, De- cember 2020. Curran Associates Inc. ISBN 978-1-7138-2954-6. URL https://dl.acm.org/doi/10.5555/ 3495724.3496517. 68 Da Li, Keping Bi, Jiafeng Guo, and Xueqi Cheng. Tailoring Table Retrieval from a Field-aware Hybrid Matching Perspective. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 27693–27704. Association for Computational Linguistics, March 2025a. ISBN 979-8-89176- 332-6. doi: 10.18653/v1/2025.emnlp-main.1409. URL https://aclanthology.org/2025.emnlp-main. 1409/. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. In Advances in Neural Information Processing Systems. Curran Associates Inc., 2023. doi: 10.5555/3666122.3667957. Jinyang Li, Nan Huo, Yan Gao, Jiayi Shi, Yingxiu Zhao, Ge Qu, Bowen Qin, Yurong Wu, Xiaodong Li, Chenhao Ma, Jian-Guang Lou, and Reynold Cheng. Are Large Language Models Ready for Multi-Turn Tabular Data Analysis? In Proceedings of the 42nd International Conference on Machine Learning, p. 34795–34835. PMLR, October 2025b. URL https://proceedings.mlr.press/v267/li25aj.html. Keqian Li, Yeye He, and Kris Ganjam. Discovering Enterprise Concepts Using Spreadsheet Tables. In Pro- ceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, p. 1873–1882, New York, NY, USA, August 2017. Association for Computing Machinery. ISBN 978-1-4503-4887-4. doi: 10.1145/3097983.3098102. URL https://dl.acm.org/doi/10.1145/3097983. 3098102. Liyao Li, Jiaming Tian, Hao Chen, Wentao Ye, Chao Ye, Haobo Wang, Ningtao Wang, Xing Fu, Gang Chen, and Junbo Zhao. LongTableBench: Benchmarking Long-Context Table Reasoning across Real-World Formats and Domains. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, p. 11927–11965, Suzhou, China, November 2025c. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.638. URL https://aclanthology.org/2025.findings-emnlp.638/. Liyao Li, Chao Ye, Wentao Ye, Yifei Sun, Zhe Jiang, Haobo Wang, Jiaming Tian, Yiming Zhang, Ningtao Wang, Xing Fu, Gang Chen, and Junbo Zhao. Table as a Modality for Large Language Models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, October 2025d. URL https://openreview.net/forum?id=kurEZdWU9G. Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-GPT: Table-tuned GPT for Diverse Table Tasks. In Proceedings of the ACM on Management of Data, volume 2 of 3, p. 1–28. Association for Computing Machinery, 2024. doi: 10.1145/3654979. URL https://doi.org/10.1145/3654979. Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, p. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013/. Hsien-chin Lin, Nurul Lubis, Songbo Hu, Carel van Niekerk, Christian Geishauser, Michael Heck, Shu- tong Feng, and Milica Gasic. Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems. In Haizhou Li, Gina-Anne Levow, Zhou Yu, Chitralekha Gupta, Berrak Sisman, Siqi Cai, David Vandyke, Nina Dethlefs, Yan Wu, and Junyi Jessy Li (eds.), Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, p. 445–456, Singapore and Online, July 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.sigdial-1.47. URL https://aclanthology.org/2021.sigdial-1.47/. Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adria De Gispert, and Gonzalo Iglesias. An Inner Table Retriever for Robust Table Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9909–9926, Toronto, Canada, 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.551. URL https://aclanthology.org/ 2023.acl-long.551. 69 Weizhe Lin, Rexhina Blloshmi, Bill Byrne, Adria de Gispert, and Gonzalo Iglesias. LI-RAGE: Late Interac- tion Retrieval Augmented Generation with Explicit Signals for Open-Domain Table Question Answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), p. 1557–1566, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-short.133. URL https://aclanthology.org/2023.acl-short.133. Chuang Liu, Junzhuo Li, and Deyi Xiong. Tab-CQA: A Tabular Conversational Question Answering Dataset on Financial Reports. In Sunayana Sitaram, Beata Beigman Klebanov, and Jason D Williams (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), p. 196–207, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-industry.20. URL https://aclanthology.org/2023.acl-industry.20/. Tianyang Liu, Fei Wang, and Muhao Chen. Rethinking Tabular Data Understanding with Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 450–482, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long. 26. URL https://aclanthology.org/2024.naacl-long.26/. Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data. In Findings of the Association for Computational Linguistics ACL 2024, p. 9215–9235, Bangkok, Thailand and virtual meeting, 2024b. Association for Computational Linguistics. doi: 10.18653/v1/2024. findings-acl.548. URL https://aclanthology.org/2024.findings-acl.548. Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. A Survey of Text-to-SQL in the Era of LLMs: Where are We, and Where are We Going? IEEE Transactions on Knowledge and Data Engineering, p. 1–20, 2025a. ISSN 1558-2191. doi: 10.1109/TKDE.2025.3592032. URL https://ieeexplore.ieee.org/abstract/document/11095853. Yujian Liu, Jiabao Ji, Tong Yu, Ryan A. Rossi, Sungchul Kim, Handong Zhao, Ritwik Sinha, Yang Zhang, and Shiyu Chang. Augment before You Try: Knowledge-Enhanced Table Question Answering via Table Expansion. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, p. 20769–20786, Suzhou, China, November 2025b. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10. 18653/v1/2025.findings-emnlp.1131. URL https://aclanthology.org/2025.findings-emnlp.1131/. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning. In The Eleventh International Conference on Learning Representations, September 2022. URL https://openreview.net/forum?id=DHyHRBwJUTN. Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, and Xiaoyong Du. Large Language Model for Table Processing: A Survey. Frontiers of Computer Science, 19, 2025a. URL https://link.springer. com/article/10.1007/s11704-024-40763-6. Xinyuan Lu, Liangming Pan, Yubo Ma, Preslav Nakov, and Min-Yen Kan. TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoning. In Findings of the Association for Computational Linguistics: NAACL 2025. Association for Computational Linguistics, April 2025b. doi: 10.18653/v1/2025.findings-naacl.244. URL https://aclanthology.org/2025.findings-naacl.244/. Bertram Ludäscher, Ilkay Altintas, Chad Berkley, Dan Higgins, Efrat Jaeger, Matthew Jones, Edward A. Lee, Jing Tao, and Yang Zhao. Scientific workflow management and the Kepler system. Concurrency and computation, 18(10):1039–1065, 2006. ISSN 1532-0626. doi: 10.1002/cpe.994. Feng Luo, Hai Lan, Hui Luo, Zhifeng Bao, Xiaoli Wang, J. Shane Culpepper, and Shazia Sadiq. Decomposition-Driven Multi-Table Retrieval and Reasoning for Numerical Question Answering, March 2026. URL http://arxiv.org/abs/2603.07950. arXiv:2603.07950 [cs]. 70 Jock Mackinlay. Automating the design of graphical presentations of relational information. ACM Trans. Graph., 5(2):110–141, April 1986. ISSN 0730-0301. doi: 10.1145/22949.22950. URL https://dl.acm. org/doi/10.1145/22949.22950. Jock Mackinlay, Pat Hanrahan, and Chris Stolte. Show Me: Automatic Presentation for Visual Analysis. IEEE Transactions on Visualization and Computer Graphics, 13(6):1137–1144, November 2007. ISSN 1941-0506. doi: 10.1109/TVCG.2007.70594. URL https://ieeexplore.ieee.org/abstract/document/ 4376133. Xiao-Qin Mai, Jing Luo, Jian-Hui Wu, and Yue-Jia Luo. “Aha!” effects in a guessing riddle task: An event-related potential study. Human Brain Mapping, 22(4):261–270, 2004. ISSN 1097-0193. doi: 10.1002/hbm.20030. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/hbm.20030. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/hbm.20030. Qingyang Mao, Qi Liu, Zhi Li, Mingyue Cheng, Zheng Zhang, and Rui Li. PoTable: Programming Standardly on Table-based Reasoning Like a Human Analyst, December 2024. URL http://arxiv.org/abs/2412. 04272. arXiv:2412.04272 [cs]. Gary Marchionini. Exploratory search: from finding to understanding. Commun. ACM, 49(4):41–46, April 2006. ISSN 0001-0782. doi: 10.1145/1121949.1121979. URL https://dl.acm.org/doi/10.1145/ 1121949.1121979. Fernando Martínez-Plumed, Lidia Contreras-Ochando, Cèsar Ferri, José Hernández-Orallo, Meelis Kull, Nicolas Lachiche, María José Ramírez-Quintana, and Peter Flach. CRISP-DM Twenty Years Later: From Data Mining Processes to Data Science Trajectories. IEEE Transactions on Knowledge and Data En- gineering, 33(08):3048–3061, August 2021. ISSN 1041-4347. doi: 10.1109/TKDE.2019.2962680. URL https://w.computer.org/csdl/journal/tk/2021/08/08943998/1g6v4EBgP7O. Shuhaib Mehri, Xiaocheng Yang, Takyoung Kim, Gokhan Tur, Shikib Mehri, and Dilek Hakkani-Tür. Goal Alignment in LLM-Based User Simulators for Conversational AI, March 2026. URL http://arxiv.org/ abs/2507.20152. arXiv:2507.20152 [cs]. Rakesh Menon, Kun Qian, Liqun Chen, Ishika Joshi, Daniel Pandyan, Shashank Srivastava, and Yunyao Li. FISQL: Enhancing Text-to-SQL Systems with Rich Interactive Feedback. In Proceedings of the 28th International Conference on Extending Database Technology (EDBT). OpenProceedings.org, 2025. doi: 10.48786/EDBT.2025.86. URL https://openproceedings.org/2025/conf/edbt/paper-300.pdf. Patrick Mikalef, Ilias O. Pappas, John Krogstie, and Michail Giannakos. Big data analytics capabilities: a systematic literature review and research agenda. Information Systems and e-Business Management, 16 (3):547–578, August 2018. ISSN 1617-9854. doi: 10.1007/s10257-017-0362-y. URL https://doi.org/ 10.1007/s10257-017-0362-y. Anna Mitsopoulou and Georgia Koutrika. Analysis of Text-to-SQL Benchmarks: Limitations, Challenges and Opportunities. In Proceedings of the 28th International Conference on Extending Database Technology (EDBT), 2025. Dominik Moritz, Chenglong Wang, Greg L. Nelson, Halden Lin, Adam M. Smith, Bill Howe, and Jeffrey Heer. Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco. IEEE Transactions on Visualization and Computer Graphics, 25(1):438–448, January 2019. ISSN 1941-0506. doi: 10.1109/TVCG.2018.2865240. URL https://ieeexplore.ieee.org/abstract/ document/8440847. Tamara Munzner. Visualization Analysis and Design. A K Peters/CRC Press, New York, December 2014. ISBN 978-0-429-08890-2. doi: 10.1201/b17511. Md Mahadi Hasan Nahid and Davood Rafiei. TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: 71 Human Language Technologies (Volume 1: Long Papers), p. 5725–5737, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.320. URL https: //aclanthology.org/2024.naacl-long.320/. Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. FeTaQA: Free-form Table Question Answering. Transactions of the Association for Computational Linguistics, 10:35–49, 2022. doi: 10.1162/tacl_a_00446. URL https://aclanthology.org/2022.tacl-1.3. Place: Cambridge, MA. Arpit Narechania, Adam Fourney, Bongshin Lee, and Gonzalo Ramos. DIY: Assessing the Correctness of Natural Language to SQL Systems. In Proceedings of the 26th International Conference on Intelligent User Interfaces, IUI ’21, p. 597–607, New York, NY, USA, April 2021. Association for Computing Ma- chinery. ISBN 978-1-4503-8017-1. doi: 10.1145/3397481.3450667. URL https://dl.acm.org/doi/10. 1145/3397481.3450667. Fatemeh Nargesian, Erkang Zhu, Renée J. Miller, Ken Q. Pu, and Patricia C. Arocena. Data lake manage- ment: challenges and opportunities. Proc. VLDB Endow., 12(12):1986–1989, August 2019. ISSN 2150- 8097. doi: 10.14778/3352063.3352116. URL https://dl.acm.org/doi/10.14778/3352063.3352116. Giang Nguyen, Ivan Brugere, Shubham Sharma, Sanjay Kariyappa, Anh Totti Nguyen, and Freddy Lecue. Interpretable LLM-based Table Question Answering. Transactions on Machine Learning Research, April 2025. ISSN 2835-8856. URL https://openreview.net/forum?id=2eTsZBoU2W. Shiwen Ni, Guhong Chen, Shuaimin Li, Xuanang Chen, Siyi Li, Bingli Wang, Qiyao Wang, Xingjian Wang, Yifan Zhang, Liyang Fan, Chengming Li, Ruifeng Xu, Le Sun, and Min Yang. A Survey on Large Language Model Benchmarks, August 2025. URL http://arxiv.org/abs/2508.15361. arXiv:2508.15361 [cs]. C. North. Toward measuring visualization insight. IEEE Computer Graphics and Applications, 26(3): 6–9, May 2006. ISSN 1558-1756. doi: 10.1109/MCG.2006.70. URL https://ieeexplore.ieee.org/ abstract/document/1626178. Matthew J. Page, Joanne E. McKenzie, Patrick M. Bossuyt, Isabelle Boutron, Tammy C. Hoffmann, Cyn- thia D. Mulrow, Larissa Shamseer, Jennifer M. Tetzlaff, Elie A. Akl, Sue E. Brennan, Roger Chou, Julie Glanville, Jeremy M. Grimshaw, Asbjørn Hróbjartsson, Manoj M. Lalu, Tianjing Li, Elizabeth W. Loder, Evan Mayo-Wilson, Steve McDonald, Luke A. McGuinness, Lesley A. Stewart, James Thomas, Andrea C. Tricco, Vivian A. Welch, Penny Whiting, and David Moher. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372:n71, March 2021. ISSN 1756-1833. doi: 10.1136/bmj.n71. URL https://w.bmj.com/content/372/bmj.n71. Vaishali Pal, Andrew Yates, Evangelos Kanoulas, and Maarten de Rijke. MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6322–6334, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.348. URL https://aclanthology.org/2023.acl-long.348/. Feifei Pan, Mustafa Canim, Michael Glass, Alfio Gliozzo, and James Hendler. End-to-End Table Question Answering via Retrieval-Augmented Generation, March 2022. URL http://arxiv.org/abs/2203.16714. arXiv:2203.16714 [cs]. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a Method for Automatic Evaluation of Machine Translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, p. 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115/1073083.1073135. URL https://aclanthology.org/P02-1040/. Raja Parasuraman and Dietrich H. Manzey. Complacency and Bias in Human Use of Automation: An Atten- tional Integration. Human factors, 52(3):381–410, 2010. ISSN 0018-7208. doi: 10.1177/0018720810376055. Place: Los Angeles, CA. 72 Panupong Pasupat and Percy Liang. Compositional Semantic Parsing on Semi-Structured Tables. In Chengqing Zong and Michael Strube (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Process- ing (Volume 1: Long Papers), p. 1470–1480, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15-1142. URL https://aclanthology.org/P15-1142/. Sohan Patnaik, Heril Changwal, Milan Aggarwal, Sumit Bhatia, Yaman Kumar, and Balaji Krishnamurthy. CABINET: Content Relevance-based Noise Reduction for Table Question Answering. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= SQrHpTllXa. Judea Pearl. Causality. University Press, Cambridge, 2nd edition edition, 2009. ISBN 978-1-139-63780-0. Shengmin Piao, Jieun Lee, and Sanghyun Park. LitE-SQL: A Lightweight and Efficient Text-to-SQL Frame- work with Vector-based Schema Linking and Execution-Guided Self-Correction. In Vera Demberg, Ken- taro Inui, and Lluís Marquez (eds.), Findings of the Association for Computational Linguistics: EACL 2026, p. 3593–3608, Rabat, Morocco, March 2026. Association for Computational Linguistics. ISBN 979-8-89176-386-9. doi: 10.18653/v1/2026.findings-eacl.186. URL https://aclanthology.org/2026. findings-eacl.186/. Giovanni Pinna, Yuriy Perezhohin, Luca Manzoni, Mauro Castelli, and Andrea De Lorenzo. Redefining text- to-SQL metrics by incorporating semantic and structural similarity. Scientific Reports, 15(1):22357, July 2025. ISSN 2045-2322. doi: 10.1038/s41598-025-04890-9. URL https://w.nature.com/articles/ s41598-025-04890-9. Peter Pirolli and Stuart Card. The Sensemaking Process and Leverage Points for Analyst Technology as Identified Through Cognitive Task Analysis. In Proceedings of international conference on intelligence analysis, 2005. Michael Polanyi. The tacit dimension. Routledge & Kegan Paul, London, 1967. Mohammadreza Pourreza and Davood Rafiei. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. In Thirty-seventh Conference on Neural Information Processing Systems, November 2023. URL https://openreview.net/forum?id=p53QDxSIc5. Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O. Arik. CHASE-SQL: Multi-Path Reasoning and Prefer- ence Optimized Candidate Selection in Text-to-SQL. In Proceedings of the 13th International Conference on Learning Representations, 2025. doi: 10.48550/arXiv.2410.01943. URL https://openreview.net/ forum?id=CvGqMD5OtX. arXiv:2410.01943 [cs]. Shuofei Qiao, Yanqiu Zhao, Zhisong Qiu, Xiaobin Wang, Jintian Zhang, Zhao Bin, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Scaling Generalist Data-Analytic Agents. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=5PxFqpIYWC. Yang Qin, Chao Chen, Zhihang Fu, Ze Chen, Dezhong Peng, Peng Hu, and Jieping Ye. ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=BAglD6NGy0. Jingang Qu, David Holzmüller, Gaël Varoquaux, and Marine Le Morvan. TabICL: A Tabular Founda- tion Model for In-Context Learning on Large Data. In Proceedings of the 42nd International Confer- ence on Machine Learning, volume 267. PMLR, 2025. doi: 10.48550/arXiv.2502.05564. URL https: //proceedings.mlr.press/v267/qu25d.html. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog? id=qwen3.5. 73 Filip Radlinski and Nick Craswell. A Theoretical Framework for Conversational Search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR ’17, p. 117–126, New York, NY, USA, March 2017. Association for Computing Machinery. ISBN 978-1-4503-4677-1. doi: 10.1145/3020165.3020183. URL https://dl.acm.org/doi/10.1145/3020165.3020183. E. Rahm and H. Do. Data Cleaning: Problems and Current Approaches. IEEE Data(base) Engineering Bulletin, 2000.URL https://w.semanticscholar.org/paper/Data-Cleaning% 3A-Problems-and-Current-Approaches.-Rahm-Do/6a62c0f05abe0516dc4cf2af5bcc8c2583a6db92. Raymond Reiter. On closed world data bases. In Readings in artificial intelligence, p. 119–140. Morgan Kaufmann, 1981. ISBN 978-0-934613-03-3. doi: https://doi.org/10.1016/B978-0-934613-03-3.50014-3. URL https://w.sciencedirect.com/science/article/pii/B9780934613033500143. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis, September 2020. URL http://arxiv.org/abs/2009.10297. arXiv:2009.10297 [cs]. Xuancheng Ren, Shijing Hu, Zhihui Lu, Jiangqi Huang, and Qiang Duan. LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queries, January 2026. URL http://arxiv.org/abs/2601.10398. arXiv:2601.10398 [cs]. Stuart Russell and Eric Wefald. Principles of metareasoning. Artificial Intelligence, 49(1):361–395, May 1991. ISSN 0004-3702. doi: 10.1016/0004-3702(91)90015-C. URL https://w.sciencedirect.com/ science/article/pii/000437029190015C. Stuart J. Russell. Artificial intelligence: a modern approach. Pearson series in artificial intelligence. Pearson, Harlow, United Kingdom, fourth edition., global edition. edition, 2022. ISBN 978-1-292-40113-3. Gaurav Sahu, Abhay Puri, Juan A. Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, Nicolas Chapa- dos, Christopher Pal, Sai Rajeswar, and Issam H. Laradji. InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation. In The Thirteenth International Conference on Learning Representations, October 2024. URL https://openreview.net/forum?id=ZGqd0cbBvm. Alireza Salemi, Surya Kallumadi, and Hamed Zamani. Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation. In Proceedings of the 47th International ACM SI- GIR Conference on Research and Development in Information Retrieval, SIGIR ’24, p. 752–762, New York, NY, USA, July 2024a. Association for Computing Machinery. ISBN 979-8-4007-0431-4. doi: 10.1145/3626772.3657783. URL https://dl.acm.org/doi/10.1145/3626772.3657783. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When Large Language Models Meet Personalization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 7370–7392, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/ v1/2024.acl-long.399. URL https://aclanthology.org/2024.acl-long.399/. Irina Saparina and Mirella Lapata. Disambiguate First, Parse Later: Generating Interpretations for Am- biguity Resolution in Semantic Parsing. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, p. 16825–16839, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-8-89176- 256-5. doi: 10.18653/v1/2025.findings-acl.863. URL https://aclanthology.org/2025.findings-acl. 863/. Irina Saparina and Mirella Lapata. Reasoning About Intent for Ambiguous Requests, 2025b. URL https: //arxiv.org/abs/2511.10453. Version Number: 1. Tefko Saracevic. Relevance: A review of the literature and a framework for thinking on the no- tion in information science. Part I: Behavior and effects of relevance. Journal of the American 74 Society for Information Science and Technology, 58(13):2126–2144, 2007. ISSN 1532-2890. doi: 10.1002/asi.20681. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.20681. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/asi.20681. P. Saraiya, C. North, and K. Duca. An insight-based methodology for evaluating bioinformatics visual- izations. IEEE Transactions on Visualization and Computer Graphics, 11(4):443–456, July 2005. ISSN 1941-0506. doi: 10.1109/TVCG.2005.53. URL https://ieeexplore.ieee.org/document/1432690. Abhraneel Sarma, Kyle Hwang, Jessica Hullman, and Matthew Kay. Milliways: Taming Multiverses through Principled Evaluation of Data Analysis Paths. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, p. 1–15, New York, NY, USA, May 2024. Association for Computing Machinery. ISBN 979-8-4007-0330-0. doi: 10.1145/3613904.3642375. URL https://doi.org/ 10.1145/3613904.3642375. Ayush Sawarni, Jiyuan Tan, and Vasilis Syrgkanis. CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation, May 2026. URL http://arxiv. org/abs/2602.20571. arXiv:2602.20571 [cs.AI]. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. Agenda-Based User Sim- ulation for Bootstrapping a POMDP Dialogue System. In Candace Sidner, Tanja Schultz, Matthew Stone, and ChengXiang Zhai (eds.), Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Pa- pers, p. 149–152, Rochester, New York, April 2007. Association for Computational Linguistics. URL https://aclanthology.org/N07-2038/. Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. Evaluating Mixed-initiative Conversational Search Systems via User Simulation. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM ’22, p. 888–896, New York, NY, USA, February 2022. As- sociation for Computing Machinery. ISBN 978-1-4503-9132-0. doi: 10.1145/3488560.3498440. URL https://dl.acm.org/doi/10.1145/3488560.3498440. Ivan Sekulić, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, An- dre Ferreira Manso, and Roland Mathis. Reliable LLM-based User Simulator for Task-Oriented Dia- logue Systems. In Yvette Graham, Qun Liu, Gerasimos Lampouras, Ignacio Iacobacci, Sinead Madden, Haider Khalid, and Rameez Qureshi (eds.), Proceedings of the 1st Workshop on Simulating Conversa- tional Intelligence in Chat (SCI-CHAT 2024), p. 19–35, St. Julians, Malta, March 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.scichat-1.3. URL https://aclanthology.org/2024. scichat-1.3/. Kwangwook Seo, Donguk Kwon, and Dongha Lee. MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 23142–23172, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025. acl-long.1128. URL https://aclanthology.org/2025.acl-long.1128/. Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odumakinde, Sameer Singh, and Seraphina Goldfarb- Tarrant. Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agen- tic Evaluations. In ICLR Workshop on Algorithmic Fairness Across Alignment Procedures and Agentic Systems, March 2026. URL https://openreview.net/forum?id=m57vJLBHxA. Vidya Setlur, Sarah E. Battersby, Melanie Tory, Rich Gossweiler, and Angel X. Chang. Eviza: A Natural Language Interface for Visual Analysis. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST ’16, p. 365–377, New York, NY, USA, October 2016. Association for Computing Machinery. ISBN 978-1-4503-4189-9. doi: 10.1145/2984511.2984588. URL https://dl.acm. org/doi/10.1145/2984511.2984588. 75 Vidya Setlur, Enamul Hoque, Dae Hyun Kim, and Angel X. Chang. Sneak Pique: Exploring Autocompletion as a Data Discovery Scaffold for Supporting Visual Analysis. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, UIST ’20, p. 966–978, New York, NY, USA, October 2020. Association for Computing Machinery. ISBN 978-1-4503-7514-6. doi: 10.1145/3379337. 3415813. URL https://dl.acm.org/doi/10.1145/3379337.3415813. Vidya Setlur, Andriy Kanyuka, and Arjun Srinivasan. Olio: A Semantic Search Interface for Data Repos- itories. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, p. 1–16, New York, NY, USA, October 2023. Association for Computing Machinery. ISBN 979-8-4007-0132-0. doi: 10.1145/3586183.3606806. URL https://doi.org/10.1145/3586183.3606806. Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, and Diyi Yang. Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=GDYueXtKXT. Zhihui Shao, Shubin Cai, Rongsheng Lin, and Zhong Ming. Enhancing Text-to-SQL with Question Clas- sification and Multi-Agent Collaboration. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Find- ings of the Association for Computational Linguistics: NAACL 2025, p. 4340–4349, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-195-7. doi: 10.18653/v1/2025.findings-naacl.245. URL https://aclanthology.org/2025.findings-naacl.245/. Galit Shmueli. To Explain or to Predict? Statistical Science, 25(3):289–310, 2010. ISSN 0883-4237. URL https://w.jstor.org/stable/41058949. B. Shneiderman. The eyes have it: a task by data type taxonomy for information visualizations. In Proceed- ings 1996 IEEE Symposium on Visual Languages, p. 336–343, September 1996. doi: 10.1109/VL.1996. 545307. URL https://ieeexplore.ieee.org/document/545307/authors. Ben Shneiderman. Direct manipulation: A step beyond programming languages (abstract only). SIGSOC Bull., 13(2-3):143, May 1981. ISSN 0163-5794. doi: 10.1145/1015579.810991. URL https://doi.org/ 10.1145/1015579.810991. Ben Shneiderman. Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy. Interna- tional Journal of Human–Computer Interaction, 36(6):495–504, April 2020. ISSN 1044-7318. doi: 10.1080/10447318.2020.1741118. URL https://doi.org/10.1080/10447318.2020.1741118. _eprint: https://doi.org/10.1080/10447318.2020.1741118. Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84–90, May 2022. ISSN 1566-2535. doi: 10.1016/j.inffus.2021.11.011. URL https://w. sciencedirect.com/science/article/pii/S1566253521002360. Arjun Srinivasan and John Stasko. Natural language interfaces for data analysis with visualization: con- sidering what has and could be asked. In Proceedings of the Eurographics/IEEE VGTC Conference on Visualization: Short Papers, EuroVis ’17, p. 55–59, Goslar, DEU, June 2017. Eurographics Association. doi: 10.2312/eurovisshort.20171133. URL https://doi.org/10.2312/eurovisshort.20171133. Arjun Srinivasan and John Stasko. Orko: Facilitating Multimodal Interaction for Visual Exploration and Analysis of Networks. IEEE Transactions on Visualization and Computer Graphics, 24(1):511–521, Jan- uary 2018. ISSN 1941-0506. doi: 10.1109/TVCG.2017.2745219. URL https://ieeexplore.ieee.org/ abstract/document/8019860. Arjun Srinivasan, Bongshin Lee, and John Stasko. Interweaving Multimodal Interaction With Flexible Unit Visualizations for Data Exploration. IEEE Transactions on Visualization and Computer Graphics, 27(8): 3519–3533, August 2021. ISSN 1941-0506. doi: 10.1109/TVCG.2020.2978050. URL https://ieeexplore. ieee.org/abstract/document/9023002. Robert J Sternberg and Janet E Davidson. The Nature of Insight. The MIT Press, 1995. ISBN 978-0-262- 28437-0. doi: 10.7551/mitpress/4879.001.0001. URL https://direct.mit.edu/books/edited-volume/ 3958/The-Nature-of-Insight. 76 Aofeng Su, Aowen Wang, Chao Ye, Chen Zhou, Ga Zhang, Gang Chen, Guangcheng Zhu, Haobo Wang, Haokai Xu, Hao Chen, Haoze Li, Haoxuan Lan, Jiaming Tian, Jing Yuan, Junbo Zhao, Junlin Zhou, Kaizhe Shou, Liangyu Zha, Lin Long, Liyao Li, Pengzuo Wu, Qi Zhang, Qingyi Huang, Saisai Yang, Tao Zhang, Wentao Ye, Wufang Zhu, Xiaomeng Hu, Xijun Gu, Xinjie Sun, Xiang Li, Yuhang Yang, and Zhiqing Xiao. TableGPT2: A Large Multimodal Model with Tabular Data Integration, November 2024. URL http://arxiv.org/abs/2411.02059. arXiv:2411.02059 [cs]. Alane Suhr, Ming-Wei Chang, Peter Shaw, and Kenton Lee. Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 8372– 8388, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.742. URL https://aclanthology.org/2020.acl-main.742/. Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive Architectures for Language Agents, March 2024. URL http://arxiv.org/abs/2309.02427. arXiv:2309.02427 [cs]. Ruoxi Sun, Sercan Arik, Rajarishi Sinha, Hootan Nakhost, Hanjun Dai, Pengcheng Yin, and Tomas Pfister. SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Data. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, p. 542–550, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-emnlp.39. URL https://aclanthology.org/2023.findings-emnlp.39/. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, October 2018. ISBN 978-0-262-03924-6. Bo Tang, Shi Han, Man Lung Yiu, Rui Ding, and Dongmei Zhang. Extracting Top-K Insights from Multi- dimensional Data. In Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD ’17, p. 1509–1524, New York, NY, USA, May 2017. Association for Computing Machinery. ISBN 978-1-4503-4197-4. doi: 10.1145/3035918.3035922. URL https://dl.acm.org/doi/10.1145/3035918. 3035922. Zirui Tang, Weizheng Wang, Zihang Zhou, Yang Jiao, Bangrui Xu, Boyu Niu, Dayou Zhou, Xuanhe Zhou, Guoliang Li, Yeye He, Wei Zhou, Yitong Song, Cheng Tan, Xue Yang, Chunwei Liu, Bin Wang, Conghui He, Xiaoyang Wang, and Fan Wu. LLM/Agent-as-Data-Analyst: A Survey, October 2025. URL http: //arxiv.org/abs/2509.23988. arXiv:2509.23988 [cs]. Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. Transformer Mem- ory as a Differentiable Search Index. Advances in Neural Information Processing Systems, 35:21831– 21843, December 2022. URL https://proceedings.neurips.c/paper_files/paper/2022/hash/ 892840a6123b5ec99ebaab8be1530fba-Abstract-Conference.html. Robert S. Taylor. The process of asking questions. American Documentation, 13(4):391–396, 1962. ISSN 1936-6108. doi: 10.1002/asi.5090130405. URL https://onlinelibrary.wiley.com/doi/abs/10.1002/ asi.5090130405. _eprint: https://asistdl.onlinelibrary.wiley.com/doi/pdf/10.1002/asi.5090130405. Jaime Teevan, Susan T. Dumais, and Eric Horvitz. Personalizing search via automated analysis of inter- ests and activities. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR ’05, p. 449–456, New York, NY, USA, August 2005. Association for Computing Machinery. ISBN 978-1-59593-034-7. doi: 10.1145/1076034.1076111. URL https://dl.acm.org/doi/10.1145/1076034.1076111. Jiaming Tian, Liyao Li, Wentao Ye, Haobo Wang, Lingxin Wang, Lihua Yu, Zujie Ren, Gang Chen, and Junbo Zhao. Toward real-world Table Agents: capabilities, workflows, and design principles for LLM- based table intelligence. World Wide Web, 29(2):16, January 2026. ISSN 1573-1413. doi: 10.1007/ s11280-025-01399-z. URL https://doi.org/10.1007/s11280-025-01399-z. 77 Yuan Tian, Zheng Zhang, Zheng Ning, Toby Jia-Jun Li, Jonathan K. Kummerfeld, and Tianyi Zhang. Inter- active Text-to-SQL Generation via Editable Step-by-Step Explanations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 16149–16166, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.1004. URL https://aclanthology.org/2023.emnlp-main.1004/. Yuan Tian, Jonathan K. Kummerfeld, Toby Jia-Jun Li, and Tianyi Zhang. SQLucid: Grounding Natural Language Database Queries with Interactive Explanations. In Proceedings of the 37th Annual ACM Sym- posium on User Interface Software and Technology, UIST ’24, p. 1–20, New York, NY, USA, October 2024. Association for Computing Machinery. ISBN 979-8-4007-0628-8. doi: 10.1145/3654777.3676368. URL https://dl.acm.org/doi/10.1145/3654777.3676368. Mohamed Trabelsi, Zhiyu Chen, Shuo Zhang, Brian D. Davison, and Jeff Heflin. StruBERT: Structure-aware BERT for Table Search and Matching. In Proceedings of the ACM Web Conference 2022, W ’22, p. 442–451, New York, NY, USA, April 2022. Association for Computing Machinery. ISBN 978-1-4503-9096-5. doi: 10.1145/3485447.3511972. URL https://dl.acm.org/doi/10.1145/3485447.3511972. John Wilder Tukey. Exploratory Data Analysis, volume 2. Addison-Wesley Publishing Company, 1977. Ellen M. Voorhees. The TREC question answering track. Natural Language Engineering, 7(4):361–378, December 2001. ISSN 1351-3249, 1469-8110. doi: 10.1017/S1351324901002789. URL https://w. cambridge.org/core/product/identifier/S1351324901002789/type/journal_article. Bing Wang, Yan Gao, Zhoujun Li, and Jian-Guang Lou. Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to-SQL. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, p. 5701–5714, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.352. URL https://aclanthology.org/2023.findings-acl.352/. Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro Szekely. Retrieving Complex Tables with Multi- Granular Graph Representation Learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, p. 1472–1482, New York, NY, USA, July 2021. Association for Computing Machinery. ISBN 978-1-4503-8037-9. doi: 10.1145/3404835.3462909. URL https://dl.acm.org/doi/10.1145/3404835.3462909. Jiayi Wang and Guoliang Li. AOP: Automated and Interactive LLM Pipeline Orchestration for Answering Complex Queries. In 15th Annual Conference on Innovative Data Systems Research (CIDR ’25), 2025. Richard Y. Wang and Diane M. Strong. Beyond Accuracy: What Data Quality Means to Data Consumers. Journal of management information systems, 12(4):5–33, 1996. ISSN 0742-1222. doi: 10.1080/07421222. 1996.11518099. Place: Armonk, N.Y. Tianshu Wang, Xiaoyang Chen, Hongyu Lin, Xianpei Han, Le Sun, Hao Wang, and Zhenyu Zeng. DBCopilot: Natural Language Querying over Massive Databases via Schema Routing. In Proceedings of the 28th International Conference on Extending Database Technology (EDBT). OpenProceedings.org, 2025a. doi: 10.48786/EDBT.2025.57. URL https://openproceedings.org/2025/conf/edbt/paper-209.pdf. Xinyue Wang, Kun Zhou, Wenyi Wu, Har Simrat Singh, Fang Nan, Songyao Jin, Aryan Philip, Sa- loni Patnaik, Hou Zhu, Shivam Singh, Parjanya Prashant, Qian Shen, and Biwei Huang. Causal- Copilot: An Autonomous Causal Analysis Agent, April 2025b. URL http://arxiv.org/abs/2504.13263. arXiv:2504.13263 [cs.AI]. Xixi Wang, Miguel Costa, Jordanka Kovaceva, Shuai Wang, and Francisco C. Pereira. Plugging Schema Graph into Multi-Table QA: A Human-Guided Framework for Reducing LLM Reliance. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Associ- ation for Computational Linguistics: EMNLP 2025, p. 5829–5842, Suzhou, China, November 2025c. As- sociation for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp. 311. URL https://aclanthology.org/2025.findings-emnlp.311/. 78 Yuqi Wang, Lyuhao Chen, Songcheng Cai, Zhijian Xu, and Yilun Zhao. Revisiting Automated Evaluation for Long-form Table Question Answering. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 14696–14706, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.815. URL https://aclanthology.org/2024.emnlp-main.815/. Zhensheng Wang, Wenmian Yang, Kun Zhou, Yiquan Zhang, and Weijia Jia. RETQA: A Large-Scale Open-Domain Tabular Question Answering Dataset for Real Estate Sector. In The 39th Annual AAAI Conference on Artificial Intelligence, 2025d. URL https://arxiv.org/abs/2412.10104. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Mi- culicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding. In International Conference on Learning Represen- tations, p. 55587–55610, January 2024b. URL https://proceedings.iclr.c/paper_files/paper/ 2024/file/f53fd88a4340063ecd258c0ae9948b40-Paper-Conference.pdf. Ryen W. White and Resa A. Roth. Exploratory Search: Beyond the Query-Response Paradigm. Synthesis Lectures on Information Concepts, Retrieval, and Services, 1(1):1–98, 2009. ISSN 1947- 945X. doi: 10.2200/s00174ed1v01y200901icr003. URL https://w.mendeley.com/catalogue/ 29c2334d-4da0-3dea-a982-6e7c3ac7d298/. Hadley Wickham. Tidy Data. Journal of Statistical Software, 59:1–23, September 2014. ISSN 1548-7660. doi: 10.18637/jss.v059.i10. URL https://doi.org/10.18637/jss.v059.i10. Cornelius Wolff and Madelon Hulsebos. How well do LLMs reason over tabular data, really? In Proceedings of the 4th Table Representation Learning Workshop, p. 241–250. Association for Computational Linguistics, May 2025. ISBN 979-8-89176-268-8. doi: 10.18653/v1/2025.trl-1.21. URL https://aclanthology.org/ 2025.trl-1.21/. A. Woodruff and M. Stonebraker. Supporting fine-grained data lineage in a database visualization environ- ment. In Proceedings 13th International Conference on Data Engineering, p. 91–102, April 1997. doi: 10.1109/ICDE.1997.581742. URL https://ieeexplore.ieee.org/abstract/document/581742. Michael Wooldridge and Nicholas R. Jennings. Intelligent agents: theory and practice. The Knowledge Engi- neering Review, 10(2):115–152, June 1995. ISSN 1469-8005, 0269-8889. doi: 10.1017/S0269888900008122. URL https://w.cambridge.org/core/journals/knowledge-engineering-review/article/abs/ intelligent-agents-theory-and-practice/CF2A6AAEEA1DBD486EF019F6217F1597. Cheng-Kuang Wu, Zhi Rui Tam, Chao-Chung Wu, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. I Need Help! Evaluating LLM’s Ability to Ask for Users’ Support: A Case Study on Text-to-SQL Generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 2191–2199, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.131. URL https: //aclanthology.org/2024.emnlp-main.131/. Jian Wu, Linyi Yang, Dongyuan Li, Yuliang Ji, Manabu Okumura, and Yue Zhang. MMQA: Evaluating LLMs with Multi-Table Multi-Hop Complex Questions. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id=GGlpykXDCa. Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xinrun Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Guanglin Niu, Tongliang Li, and Zhoujun Li. TableBench: A Com- prehensive and Complex Benchmark for Table Question Answering. In Proceedings of the Thirty- Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Ap- plications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial In- telligence, AAAI’25/IAAI’25/EAAI’25. AAAI Press, March 2025b. ISBN 978-1-57735-897-8. doi: 10.1609/aaai.v39i24.34739. URL https://doi.org/10.1609/aaai.v39i24.34739. 79 Xueqing Wu, Rui Zheng, Jingzhen Sha, Te-Lin Wu, Hanyu Zhou, Tang Mohan, Kai-Wei Chang, Nanyun Peng, and Haoran Huang. DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, November 2024b. URL https://openreview.net/forum?id=NrCPBJSOOc# discussion. Zhenhe Wu, Zhongqiu Li, Jie Zhang, Zhongjiang He, Jian Yang, Yu Zhao, Ruiyu Fang, Bing Wang, Hongyan Xie, Shuangyong Song, and Zhoujun Li. UCS-SQL: Uniting Content and Structure for Enhanced Semantic Bridging In Text-to-SQL. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, p. 8156–8168, Vienna, Austria, July 2025c. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.427. URL https://aclanthology.org/2025.findings-acl.427/. Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. Large Language Mod- els Can Self-Correct with Key Condition Verification. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 12846–12867, Miami, Florida, USA, November 2024c. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.714. URL https://aclanthology.org/2024.emnlp-main.714/. Dongling Xiao, LinZheng Chai, Qian-Wen Zhang, Zhao Yan, Zhoujun Li, and Yunbo Cao. CQR-SQL: Con- versational Question Reformulation Enhanced Context-Dependent Text-to-SQL Parsers. In Yoav Gold- berg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, p. 2055–2068, Abu Dhabi, United Arab Emirates, December 2022. Association for Com- putational Linguistics. doi: 10.18653/v1/2022.findings-emnlp.150. URL https://aclanthology.org/ 2022.findings-emnlp.150/. Liwenhan Xie, Chengbo Zheng, Haijun Xia, Huamin Qu, and Chen Zhu-Tian. WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST ’24, p. 1–14, New York, NY, USA, October 2024. Association for Computing Machinery. ISBN 979-8-4007-0628-8. doi: 10.1145/3654777.3676374. URL https://dl.acm.org/doi/10.1145/3654777.3676374. Wenyi Xu, Yuren Mao, Xiaolu Zhang, Chao Zhang, Xuemei Dong, Mengfei Zhang, and Yunjun Gao. DAgent: A Relational Database-Driven Data Analysis Report Generation Agent, April 2025. URL http://arxiv. org/abs/2503.13269. arXiv:2503.13269 [cs]. Zhuoyan Xu, Haoyang Fang, Boran Han, Bonan Min, Bernie Wang, Cuixiong Hu, and Shuai Zhang. Efficient Table Retrieval and Understanding with Multimodal Large Language Models, February 2026. URL http: //arxiv.org/abs/2602.07642. arXiv:2602.07642 [cs]. Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, and Maosong Sun. MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, p. 11789–11804, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl. 701. URL https://aclanthology.org/2024.findings-acl.701/. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- Act: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR), March 2023. Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning, January 2023. URL https://arxiv.org/abs/2301.13808v3. Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural Lan- guage to Code Generation in Interactive Data Science Notebooks. In Anna Rogers, Jordan Boyd-Graber, 80 and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 126–173, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10.18653/v1/2023.acl-long.9. URL https://aclanthology.org/2023.acl-long. 9/. Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, Chunyang Li, Zheyuan Zhang, Yushi Bai, Yantao Liu, Amy Xin, Kaifeng Yun, Linlu Gong, Nianyi Lin, Jianhui Chen, Zhili Wu, Yunjia Qi, Weikai Li, Yong Guan, Kaisheng Zeng, Ji Qi, Hailong Jin, Jinxin Liu, Yu Gu, Yuan Yao, Ning Ding, Lei Hou, Zhiyuan Liu, Xu Bin, Jie Tang, and Juanzi Li. KoLA: Carefully Benchmarking World Knowledge of Large Language Models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AqN23oqraW. Peiying Yu, Guoxin Chen, and Jingjing Wang. Table-Critic: A Multi-Agent Framework for Collaborative Criticism and Refinement in Table Reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 17432–17451, Vienna, Austria, July 2025. As- sociation for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.853. URL https://aclanthology.org/2025.acl-long.853/. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 3911–3921. Association for Com- putational Linguistics, 2018. doi: 10.18653/v1/D18-1425. URL https://aclanthology.org/D18-1425/. Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 1962–1979, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1204. URL https://aclanthology.org/D19-1204/. Qiuhai Zeng, Claire Jin, Xinyue Wang, Yuhan Zheng, and Qunhua Li. AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2025, p. 10170–10201, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.539. URL https://aclanthology.org/2025.findings-emnlp.539/. Amy X. Zhang, Michael Muller, and Dakuo Wang. How do Data Science Workers Collaborate? Roles, Workflows, and Tools. Proc. ACM Hum.-Comput. Interact., 4(CSCW1):22:1–22:23, May 2020a. doi: 10.1145/3392826. URL https://dl.acm.org/doi/10.1145/3392826. Chi Zhang, Meihui Zhang, Yuxin Yang, Tao Chen, and Zhaojing Luo. AixelAsk: A Stepwise-Guided Retrieval and Reasoning Framework for Large Table QA. Proceedings of the ACM on Management of Data, 3 (6):366:1–366:25, December 2025a. doi: 10.1145/3769831. URL https://dl.acm.org/doi/10.1145/ 3769831. Dan Zhang, Sining Zhoubian, Min Cai, Fengzu Li, Lekang Yang, Wei Wang, Tianjiao Dong, Ziniu Hu, Jie Tang, and Yisong Yue. DataSciBench: An LLM Agent Benchmark for Data Science, February 2025b. URL http://arxiv.org/abs/2502.13897. arXiv:2502.13897 [cs]. Han Zhang, Yuheng Ma, and Hanfang Yang. ALTER: Augmentation for Large-Table-Based Reasoning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for 81 Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 179–198, Albu- querque, New Mexico, 2025c. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.9. URL https://aclanthology.org/2025.naacl-long.9/. Kun Zhang, Xiexiong Lin, Yuanzhuo Wang, Xin Zhang, Fei Sun, Cen Jianhe, Hexiang Tan, Xuhui Jiang, and Huawei Shen. ReFSQL: A Retrieval-Augmentation Framework for Text-to-SQL Generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, p. 664–673, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.48. URL https://aclanthology.org/2023.findings-emnlp.48/. Qin Zhang, Shangsi Chen, Dongkuan Xu, Qingqing Cao, Xiaojun Chen, Trevor Cohn, and Meng Fang. A Survey for Efficient Open Domain Question Answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14447–14465, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.808. URL https://aclanthology.org/2023. acl-long.808/. Siyue Zhang, Anh Tuan Luu, and Chen Zhao. SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, p. 2352–2364, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp. 131. URL https://aclanthology.org/2024.findings-emnlp.131/. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations, September 2019. URL https://openreview.net/forum?id=SkeHuCVFDr. Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Qingfu Zhu, and Wanxiang Che. MURRE: Multi-Hop Table Retrieval with Removal for Open-Domain Text-to-SQL. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, p. 5789–5806, Abu Dhabi, UAE, January 2025d. Association for Computational Linguistics. URL https://aclanthology.org/2025.coling-main.386/. Yue Zhang, Seiji Maekawa, and Nikita Bhutani. Same Content, Different Representations: A Controlled Study for Table QA. In The Fourteenth International Conference on Learning Representations, October 2025e. URL https://openreview.net/forum?id=QTgx2ThiAb. Yunjia Zhang, Jordan Henkel, Avrilia Floratou, and Joyce Cahoon. ReAcTable: Enhancing ReAct for Table Question Answering. Proceedings of the VLDB Endowment, 17(8), 2024b. Zhehao Zhang, Xitao Li, Yan Gao, and Jian-Guang Lou. CRT-QA: A Dataset of Complex Reasoning Question Answering over Tabular Data. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 2131–2153, Singapore, December 2023c. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.132. URL https://aclanthology.org/2023.emnlp-main.132/. Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, 63(10):2011–2027, October 2020b. ISSN 1869-1900. doi: 10.1007/s11431-020-1692-3. URL https://doi.org/10.1007/s11431-020-1692-3. Qiwei Zhao, Dong Li, Yanchi Liu, Wei Cheng, Yiyou Sun, Mika Oishi, Takao Osaki, Katsushi Matsuda, Huaxiu Yao, Chen Zhao, Haifeng Chen, and Xujiang Zhao. Uncertainty Propagation on LLM Agent. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6064–6073, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10.18653/v1/2025.acl-long.302. URL https://aclanthology.org/2025.acl-long.302/. 82 Siyun Zhao, Yuqing Yang, Zilong Wang, Zhiyuan He, Luna K. Qiu, and Lili Qiu. Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely, September 2024a. URL http://arxiv.org/abs/2409.14924. arXiv:2409.14924 [cs]. Xinyang Zhao, View Profile, Xuanhe Zhou, View Profile, Guoliang Li, and View Profile. Chat2Data: An Interactive Data Analysis System with RAG, Vector Databases and LLMs. Proceedings of the VLDB Endowment, 17(12):4481–4484, August 2024b. doi: 10.14778/3685800.3685905. URL https://dl.acm. org/doi/10.14778/3685800.3685905. Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 6588–6600, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.454. URL https://aclanthology.org/2022.acl-long.454/. Yilun Zhao, Lyuhao Chen, Arman Cohan, and Chen Zhao. TaPERA: Enhancing Faithfulness and In- terpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12824–12840, Bangkok, Thailand, 2024c. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.692. URL https://aclanthology.org/2024.acl-long.692. Danna Zheng, Mirella Lapata, and Jeff Pan. Archer: A Human-Labeled Text-to-SQL Dataset with Arith- metic, Commonsense and Hypothetical Reasoning. In Yvette Graham and Matthew Purver (eds.), Pro- ceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 94–111, St. Julian’s, Malta, March 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.eacl-long.6. URL https://aclanthology.org/2024.eacl-long.6/. Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V. Le, and Denny Zhou. Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models.International Conference on Learning Representations, 2024:20279– 20316, May 2024b.URL https://proceedings.iclr.c/paper_files/paper/2024/hash/ 592da1445a51e54a3987958b5831948f-Abstract-Conference.html. Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 13921–13937, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.859. URL https://aclanthology.org/2023.emnlp-main.859/. Wei Zhou, Mohsen Mesgar, Heike Adel, and Annemarie Friedrich. FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 2479–2497, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.137. URL https://aclanthology.org/2024.naacl-long.137/. Wei Zhou, Mohsen Mesgar, Annemarie Friedrich, and Heike Adel. Efficient Multi-Agent Collaboration with Tool Use for Online Planning in Complex Table Question Answering. In Findings of the Association for Computational Linguistics: NAACL 2025, p. 945–968. Association for Computational Linguistics, 2025. ISBN 979-8-89176-195-7. doi: 10.18653/v1/2025.findings-naacl.54. URL https://aclanthology.org/ 2025.findings-naacl.54/. Wei Zhou, Bolei Ma, Annemarie Friedrich, and Mohsen Mesgar. Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation, April 2026. URL http://arxiv.org/abs/2510.09671. arXiv:2510.09671 [cs.CL]. 83 Jun-Peng Zhu, Peng Cai, Kai Xu, Li Li, Yishen Sun, Shuai Zhou, Haihuang Su, Liu Tang, and Qi Liu. AutoTQA: Towards Autonomous Tabular Question Answering through Multi-Agent Large Lan- guage Models. Proceedings of the VLDB Endowment, 17(12):3920–3933, 2024. ISSN 2150-8097. doi: 10.14778/3685800.3685816. Jun-Peng Zhu, Boyan Niu, Peng Cai, Zheming Ni, Jianwei Wan, Kai Xu, Jiajun Huang, Shengbo Ma, Bing Wang, Xuan Zhou, Guanglei Bao, Donghui Zhang, Liu Tang, and Qi Liu. Towards Automated Cross-Domain Exploratory Data Analysis through Large Language Models. Proceedings of the VLDB Endowment, 18(12):5086–5099, August 2025a. ISSN 2150-8097. doi: 10.14778/3750601.3750629. URL https://dl.acm.org/doi/10.14778/3750601.3750629. Junnan Zhu, Jingyi Wang, Bohan Yu, Xiaoyu Wu, Junbo Li, Lei Wang, and Nan Xu. TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 7126–7146, Suzhou, China, November 2025b. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10. 18653/v1/2025.emnlp-main.363. URL https://aclanthology.org/2025.emnlp-main.363/. Jiaru Zou, Dongqi Fu, Sirui Chen, Xinrui He, Zihao Li, Yada Zhu, Jiawei Han, and Jingrui He. RAG over Tables: Hierarchical Memory Index, Multi-Stage Retrieval, and Benchmarking. In ICLR 2026 Work- shop on Logical Reasoning of Large Language Models, 2026. URL https://openreview.net/forum?id= LUW0zHCE2W. 84 A Overview of Notation SymbolDescription Insights and Insight Needs (Section 2.1) iAn insight need, a person’s requirement for analytical knowledge that, interpreted within their contextual understanding, constitutes an insight (Definition 1). i d The data need, specifying what data entities, temporal scope, and domain boundaries are relevant to the insight need. i m The methodological need, specifying what analytical operations and procedures should be applied. i o The output need, specifying how results should be presented. ˆ iThe expression of an insight need i communicated by a user, often only a partial approximation of i. ˆ i s A system’s representation of the insight need, constructed from ˆ i to approximate i. Tabular Data and Corpus (Section 2.1.3) t = (H,D,M )A table, consisting of a header H, a data matrix D, and a metadata set M. T =t 1 ,...,t n A corpus of n tables over which insight needs are expressed. TThe universe of all possible tables, with a corpus T ⊂T . Realizations (Section 2.1.4) r = (O,E,β)A realization, a directed acyclic graph of fully parameterized operations O connected by dataflow edges E with a data binding β (Definition 3). RThe space of all possible realizations, with r ∈R. O =o 1 ,...,o m The set of fully parameterized operations applied in a realization’s derivation. oA single operation, a discrete functional unit ranging from granular primitives to high-level analytical functions. E ⊂ O× OThe set of directed edges connecting operations in a realization, defining the flow of data between them. β ⊂T × OThe data binding connecting specific tables from the corpus to the source operations of a realization. out(r)The analytical knowledge produced as output by executing realization r. Utility and Problem Formulation (Section 2.1.5) u i The realization utility function for insight need i, u i : R → R, scoring how well a realization satisfies i (Definition 4). R valid (i)The set of realizations valid for insight need i, i.e., realizations whose data, operations, and output address i. R feasible (T )The set of realizations feasible given corpus T, i.e., where bound tables are in T. R cand (i,T )The set of candidate realizations, R valid (i)∩R feasible (T ), that both address the insight need and are grounded in available data. r ∗ The candidate realization that maximizes utility (Equation 1); reused in Section 5.3.3 to denote a benchmark’s reference realization. Evaluation (Section 5) B = (T,E )An evaluation benchmark, comprising a corpus of tables T and a set of evaluation instances E. E =e 1 ,...,e n The set of n evaluation instances in a benchmark. e = ( ̄ i,v)An evaluation instance, comprising an input ̄ i and a validation mechanism v. ̄ iThe benchmark input, representing the expression of an insight need ˆ i under the benchmark’s input protocol. vThe validation mechanism scoring a realization, v : R → [0, 1], acting as an opera- tional stand-in for R valid (i) and u i . 85 B Comparison with Related Work Several existing works survey and systematize areas related to OpenTI, yet each addresses only a part of its problem space from within the conventions of particular research communities. A larger group of works exists that systematize systems that perform complex tasks over unseen information without targeting tabular data, spanning open-domain question answering (Zhang et al., 2023b), retrieval augmented generation (Zhao et al., 2024a), and LLM-agents (Kapoor et al., 2024; Sumers et al., 2024). More closely related, other works target tabular data but organize contributions under the labels of the communities they originate from, such as text-to-SQL (Liu et al., 2025a), table question answering (Zhou et al., 2026), the application of LLMs to tabular tasks (Fang et al., 2024; Lu et al., 2025a), and agents for data science and data analysis (Chen et al., 2025a; Tang et al., 2025; Tian et al., 2026). Table 5 compares these works against the dimensions foregrounded in our framing of OpenTI, namely the coverage of the end-to-end scope, the treatment of an open setting in which the corpus is unknown when the need is expressed, an insight-centered rather than answer-centered framing, a formal definition of the problem, and the treatment of user-facing concerns. Table 5: Comparison of this paper with other works that survey related areas.− indicates aspects that are not addressed,◦ ones that are partially addressed or acknowledged for future work, and• signifies that the work foregrounds the aspect as a central part of its contribution. For the end-to-end scope coverage, we indicate the explicit treatment of parts of the end-to-end scope using the abbreviations UI for User Interaction, R for Table Retrieval, DI for Data Integration, DA for Data Analysis, and O for Output Presentation. WorkFocusResearch Communities End-to-EndScope Coverage Open Setting Insight- Centered Formal Definition User-facing Concerns Liu et al. (2025a)Text-to-SQLDB, NLP UIDA −◦− Fang et al. (2024) LLMs for tabular tasks NLP, DB DAO −◦− Lu et al. (2025a)LLMs for tabular tasks NLP, DB DAO −◦ Zhou et al. (2026) Table Question Answering NLP, IR RDA ◦−◦ Tian et al. (2026) LLM-Agents for tabular tasks NLP, DB RDIDA ◦ Chen et al. (2025a) LLM-Agents for tabular tasks NLP, ML/AI RDIDAO −◦ Tang et al. (2025) LLM-Agents for data science DB,NLP, ML/AI RDIDAO ◦−◦ This paperOpen Tabular Insight Extraction DB, HCI, IR, ML/AI, NLP UIRDIDAO • Compared to this paper, these works cover only part of the path from an expressed need to a presented insight, mostly the analysis, while assuming that the relevant tables are supplied rather than discovered from an unknown corpus. The setting they address is thus a closed one, and where openness appears it does so as one task dimension among many rather than the premise of the problem (Zhou et al., 2026; Tang et al., 2025). None of these works takes an insight-centered perspective, treating the tabular tasks as targeting surface-level answer correctness, not as the search for the insights that serve the user. Formalization, where offered, stays confined to isolated subtasks such as translating a question into SQL (Liu et al., 2025a), and the user-facing side of the problem, from interaction to personalization and interpretability, is raised as an open challenge more than it is developed (Lu et al., 2025a; Tian et al., 2026). Instead, our framing of OpenTI treats these dimensions as one problem, spanning the full path, taking the open setting and the insight need as its starting point, grounding both in a formal definition, and placing user-facing concerns as a central aspect. 86 C Literature Review Protocol We collect the works that provide the evidence base for this paper following a structured, multi-stage protocol organized into identification, screening, and selection stages, inspired by the PRISMA guidelines for reporting systematic reviews (Page et al., 2021). We start with a broad keyword search over a curated set of venues (Section C.1) and proceed through manual screening against inclusion criteria (Section C.2), yielding a set of relevant works that informs this paper throughout. From this set we extract the systems and benchmarks that we analyze in Sections 3 and 5 (Section C.3). The sets obtained through this protocol further provide the basis for the analysis of benchmark inputs in Appendix D and for the analysis of citation behavior in Appendix E. Figure 11 summarizes the resulting funnel. Keyword search over 18 venues, 2021 to May 2026 583 matches from 7 queries 399 distinct works 160 relevant works (core set) 58 system works (Section 3) 42 benchmark works (Section 5) 29 systems (Table 1) 23 benchmarks (Table 2) + 8 via back- ward and forward snowballing + 13 via backward and forward snowballing deduplication across queries relevance screening (criteria 1 to 3) ≥ 3 capabilities covered fully or ≥ 4 partially, output beyond code non-English excluded, total score ≥ 3, plus 5 reference datasets Figure 11: Overview of the literature review funnel. A keyword search over 18 venues yields 399 distinct works after deduplication, which relevance screening reduces to a core set of 160 works. From this set, extended through backward and forward snowballing, we extract the works surveyed in Sections 3 and 5 and apply the overview-table criteria that select the 29 systems in Table 1 and the 23 benchmarks in Table 2. C.1 Identification We consider works published between 2021 and May 2026 at the venues most relevant to OpenTI across machine learning, information retrieval, human-computer interaction, database research, and natural lan- guage processing. We begin in 2021, the point from which work relevant to OpenTI accelerates markedly (Figure 1a). The venue set is not exhaustive but deliberately targets the leading venues of the communities we identify in Section 1 as contributing towards OpenTI, selecting the top venues of each to capture the most influential and representative work. To these we add the TRL workshops, which sit precisely at the intersection of tabular data and machine learning that OpenTI occupies and thereby concentrate directly relevant work that the general-purpose venues disperse. The search spans the following 18 venues. • Machine Learning. NeurIPS, ICML, ICLR, JMLR, TMLR • Information Retrieval. SIGIR, ECIR 87 • Human-Computer Interaction. CHI, UIST • Database Research. VLDB, SIGMOD, EDBT • Natural Language Processing. ACL, EMNLP, NAACL, EACL • Tabular. TRL Workshop@NeurIPS/ACL, AI for Tabular Data Workshop@EurIPS We extract the titles and abstracts of all papers published at these venues and organize them in a database. We then run seven keyword searches with diverse queries, mirroring the fragmentation of vocabulary across these communities. The queries target different labels under which communities frame facets of OpenTI, spanning data analysis agents, table understanding and reasoning, table retrieval, table question answering, and text-to-SQL. Table 6 reports the queries together with the number of matches each returns and the number of works from it judged relevant during the screening described below. Deduplicating the matches across queries yields a pool of 399 distinct works that enters screening. Table 6: Keyword queries run over the titles and abstracts, with the number of matches returned and, of these, the number of works judged relevant to the survey during screening (Section C.2). QueryMatches Retained "data analysis" AND ("agent" OR "automated" OR "LLM")9125 "data science" AND "table" AND ("agent" OR "automated" OR "LLM")136 "open-domain" AND "table"8115 "table understanding" OR "table reasoning"11036 "table retrieval"2011 "table question answering" OR "TableQA" OR "(TQA)"6239 "text-to-sql"20654 C.2 Screening and Selection We screen the pool of 399 works by assessing each title and abstract for relevance to the subject of the survey. We retain a work if it informs the treatment of OpenTI, which we operationalize through three criteria. A work qualifies if it (1) describes systems, components of systems, or assessments of systems that fall within the end-to-end scope of OpenTI, (2) proposes a benchmark or evaluation that targets directed insight needs over tabular data such as table question answering, text-to-SQL, or data analysis, or (3) concerns user-system interaction relevant to OpenTI. We further retain works that contribute neither a system nor a benchmark but that inform the conceptual treatment of the subject. Deduplicating the relevant works across queries yields a core set of 160 works, which provide the backbone of the literature review and inform its writing throughout. From this core set we isolate the works that receive detailed treatment in the separate surveys of systems and benchmarks, extracting the works that propose systems spanning the functional capabilities of OpenTI agents and the works that propose benchmarks targeting directed insight needs over tabular data. We extend each of these two sets separately through targeted backward and forward snowballing, adding works with substantial contributions that fall outside the venues, date range, or queries defined above. This adds 8 works to the systems and 13 to the benchmarks, yielding 58 works covering methods and 42 works covering benchmarks that form the basis for the surveys in Sections 3 and 5 respectively. C.3 Selection for the Overview Tables From these works we select a subset for detailed presentation in the overview tables of Section 3 (Table 1) and Section 5 (Table 2), applying criteria aimed at presenting the most relevant works concerning the scope of OpenTI. 88 For methods, we analyze each system along the five functional capabilities established in Section 3 and score its coverage of each as fully, partially, or not covered. We include a system if it covers at least three capabilities fully or four at least partially, selecting systems covering a significant extent of the scope of OpenTI. We further require that a system produces some output beyond raw results or code, so that it exercises at least partial freedom in contextualizing and presenting the analytical knowledge it derives. This yields the 29 systems presented in Table 1. For benchmarks, we first exclude non-English datasets, which we cannot fairly assess against English-language ones. We then score the remaining datasets on the degree to which they cover the data need i d , the methodological need i m , and the output need i o , assigning each component a score from 0 to 2 following the rubric in Table 7. We include datasets with a total score of at least 3, ensuring that the presented datasets cover substantial parts of the scope of insight needs, which yields 18 datasets. We supplement these with five datasets that fall below the threshold but are widely used and highly cited, whose inclusion readers would likely expect as points of reference, namely Spider (Yu et al., 2018), BIRD (Li et al., 2023), OTT-QA (Chen et al., 2021), NQ-Tables (Herzig et al., 2021), and FeTaQA (Nan et al., 2022). This results in the 23 benchmarks presented in Table 2. Table 7: Rubrics for scoring the degree to which benchmarks cover data need i d , methodological need i m , and output need i o . Insight types are counted following the taxonomy in Section 2.3. A dataset is presented in Table 2 if its scores sum to at least 3. Need Scored by012 i d Data settingOnly the relevant tables provided per instance A set of tables provided per instance Open setting i m Insight types covered Shallow descriptive only (lookup, aggregation) At least three insight types covered Any inferential or causal type covered i o Output modalitiesSingle bare modality (value or table) Two modalities or text synthesis Outputsinclude charts C.4 Limitations The results and discussion presented in this paper have to be read as claims about the surveyed works, which are limited in scope by the review methodology. The identification stage covers 18 peer-reviewed venues, so works published solely on preprint servers or at venues outside this set enter the review only through snowballing, which may underrepresent the most recent developments in this fast-moving space. Additionally, commercial and organizational systems are excluded, since their internals are not documented in a form that can be assessed against our framework, although several deployed products plausibly attempt parts of OpenTI. Our protocol departs from full PRISMA compliance. Notably, neither the search nor the inclusion criteria were registered ahead of time, so the criteria in Sections C.2 and C.3 were refined alongside our developing understanding of OpenTI rather than fixed in advance, reflecting that OpenTI is currently emerging from research. D Input Classification Methodology The analysis of data privilege and insufficient specification reported in Section 5.2.3 (Figure 10) adopts the LLM-judge setup of Gomm et al. (2025) unchanged, applying their classifiers to the benchmarks in Table 2. We classify the 18 benchmarks whose inputs are self-contained natural language utterances for single-input interaction, excluding five benchmarks for which this does not hold. TACO (Deng et al., 2026) and DA- Dataset (Xu et al., 2025) are excluded because their inputs are not available to us, ARCADE (Yin et al., 2023) and ConDABench (Dutta et al., 2026) because their step-wise inputs are incompatible with the LLM-judge setup, and similarly BLADE (Gu et al., 2024b) because its inputs follow a multiple-choice format over code rather than a natural language expression of an insight need. For each remaining benchmark we randomly sample 500 input definitions, or all inputs for benchmarks with fewer instances, and classify each sampled 89 input individually. The full prompts and implementation are available in the repository accompanying Gomm et al. (2025). Data privilege is assessed along three dimensions: references to structural elements such as column headers, references to non-public values specific to the underlying data, and references to data containers such as files or tables. To stabilize the classification, we apply self-consistency, classifying each input five times and assigning the final label by majority vote requiring at least three identical votes. An input counts as data privileged if at least one dimension is flagged, where for value references both the explicit and the obscure label count as a flag. For this classification we employ gpt-5-mini-2025-08-07, following the original setup by Gomm et al. (2025). Sufficient specification is assessed by two classifiers covering the five specification dimensions of Gomm et al. (2025), with data specification comprising entity, temporal, and domain specification and procedural specification comprising intent and methodological specification. The per-dimension labels are aggregated into boolean flags following their criteria, under which temporal specification also admits underspecification assuming recency or inapplicability, and domain specification admits underspecification assuming a universal domain. An input is fully specified if all five flags hold and insufficiently specified otherwise. We use gpt-5-2025-08-07 as an LLM-judge. The validity of the LLM-judges is thoroughly established by Gomm et al. (2025). To verify that the setup generalizes to our selection of benchmarks, we replicate their annotation study on a stratified sample of 168 annotations over the 18 classified benchmarks, following their validation protocols. For data privilege, a data-science expert independently annotates each sampled input along three dimensions of data privilege (schema, value, and container privilege), and we report the agreement between these annotations and the judge in Table 8. For sufficient specification, we follow the expert-corrected protocol of Gomm et al. (2025), in which the expert corrects the labels emitted by the judge and we report the agreement between the initial and the corrected labels in Table 9. Gomm et al. (2025) adopt this correction-based protocol because resolving underlying ambiguities requires broader world knowledge than a single independent annotation pass affords. We report both raw agreement and Cohen’s κ (Cohen, 1960), which corrects for the agreement expected by chance. Table 8: Agreement between the data privilege judge and an independent expert annotation. We report raw agreement (%) and Cohen’s κ. Privilege Dimension Rawκ Schema Privilege0.881 0.606 Value Privilege0.911 0.585 Container Privilege0.946 0.760 Data Privilege (agg.) 0.893 0.747 Table 9: Agreement between the sufficient specification classifications and expert-corrected labels. We report raw agreement (%) and Cohen’s κ. Specification Dimension Rawκ Entities0.924 0.838 Temporal0.906 0.786 Domain0.953 0.904 Intent0.971 0.861 Methodology0.929 0.850 Full Specification (agg.) 0.953 0.869 In this annotation study, we find a substantial agreement of κ = 0.747 on the aggregated data privilege judg- ments and an almost perfect agreement of κ = 0.869 on the aggregated judgments of sufficient specification, following the scale of Landis & Koch (1977). The raw agreement scores are in line with those reported by Gomm et al. (2025), indicating that the judges generalize to the benchmarks we assess. 90 E Citation Analysis To support the claim that works relevant to OpenTI are organized by the task label under which they frame their contributions instead of the problem they address, and that works converging on similar system designs frequently remain unaware of one another, we analyze how surveyed works cite each other against what label they use to describe the task they target. In this citation analysis we construct a citation graph among the surveyed works, assign each work the task label it uses to frame itself, and measure whether citation is assortative on that label. We analyze the 181 works that we surface in our literature review, including the works identified through forward and backward snowballing. E.1 Citation Graph Extraction We extract the reference list of each work from its PDF with grobid (gro, 2008), yielding 8,473 references across 180 of the 181 works. We match references against the surveyed works by comparing normalized titles. We apply a hierarchical matching procedure, first matching the full title, then matching only the title preceding any subtitle, and then fuzzy matching with a similarity threshold of 0.85. Since many titles begin with a short system name, we accept a match on the shortened title only when the first authors of both works match. In total, we identify 959 edges between the papers, of which 96% come from an exact title match. We additionally employ Semantic Scholar and OpenAlex to compare and complement our extraction, which yield 809 (Semantic Scholar) and 109 (OpenAlex) edges between surveyed papers respectively. We add the 40 edges not extracted through grobid, giving a final citation graph with 999 edges. E.2 Assigning Task Labels We assign each work one of three labels, namely text-to-SQL, table question answering, and data analysis agents. These are the three framings under which the surveyed works predominantly present themselves. Works framed around table retrieval are assigned the label of the downstream task they serve, since retrieval is a functional capability rather than a task in the sense of OpenTI (Section 3.3.2). To label how each paper frames the task it addresses we use a multi-step process. First, we automatically assign a label to each paper by extracting how often the label, or a derivation of it, occurs in the title and the abstract of the papers, assigning the label that appears most often, not assigning a label if two labels appear the same number of times or if none appears. We cross-check the assignment by an independent classifier that labels the same titles and abstracts against a written codebook, using LLM-based classification with a majority vote over three samples, using Qwen3.5-9B (Qwen Team, 2026) as LLM. The two procedures agree on 86.2% of the works, corresponding to a substantial agreement of κ = 0.799. We manually review the 37 works on which they disagree, or for which neither procedure finds a task term, to correct any remaining inconsistencies. This yields 85 table question answering works, 54 text-to-SQL works, and 33 data analysis agent works, while 9 works are not assigned to any task label. E.3 Analysis We measure assortativity over feasible citations as pairs of works (A,B) for which A could have cited B, meaning that B is published before or in the same year as A. Restricting to these feasible pairs removes the differing sizes of the label groups and the age distribution of the works as confounds, and yields 18,243 pairs. We compare the citation rate within labels against the rate across labels and test the ratio between the two by permuting the labels across works over 10,000 draws. E.3.1 Results Works cite within their own label in 10.6% of the feasible pairs and across labels in 1.8%, a ratio of 6.02, significant at p < 0.001. Looking into the surveyed systems specifically, an increase in the number of functional capabilities shared between two systems raises the citation rate within a label from 5.6% to 18.5% 91 while the rate across labels stays at or below 3.0% (Figure 6b), so systems converging on similar designs are no more likely to cite each other across a label boundary than systems sharing no capability at all. Importantly, the absence of a citation is evidence of non-engagement rather than proof that works are unaware of each other. Nonetheless, these results show that literature engages less rigorously with prior works across task labels than within them. Additionally, around a quarter of the works we label under data analysis agents cite none of the surveyed works at all, so the corresponding row of Figure 6a reflects not just the isolation of that literature but also the reach of our review. 92