Paper deep dive
The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators
Tarfah Alrashed, Fatma Ozcan, Per Jacobsson, Tal Neiman, Xianshun Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/21/2026, 3:57:59 AM
Summary
This paper introduces a Multilayered Evaluation Framework to address the failure of traditional Text-to-SQL evaluation metrics (like Execution Accuracy) when applied to AI-augmented SQL queries. AI operators introduce non-determinism, causing valid queries to be falsely rejected by strict execution-based metrics. The proposed framework decouples deterministic relational logic from flexible AI semantics using a SQL Splitter and an LLM-based Autorater, achieving up to 97.2% accuracy on BigQuery and ThalamusDB, significantly outperforming existing baselines.
Entities (10)
Relation Signals (7)
Multilayered Evaluation Framework â evaluateson â ThalamusDB
confidence 95% ¡ We test our approach across both industry (BigQuery) and academic (ThalamusDB) systems.
Multilayered Evaluation Framework â evaluateson â BigQuery
confidence 95% ¡ We test our approach across both industry (BigQuery) and academic (ThalamusDB) systems.
Multilayered Evaluation Framework â uses â SQL Splitter
confidence 95% ¡ our framework first isolates the structural validity... The 'SQL Splitter' is a general architectural component
Multilayered Evaluation Framework â uses â LLM-based Autorater
confidence 95% ¡ paired with an LLM autorater to evaluate prompt intent independently of database execution.
LLM-SQL-Solver â comparedwith â Multilayered Evaluation Framework
confidence 90% ¡ we compare our proposed Multilayered Evaluation Framework against two baselines: Execution Accuracy (EX) and a state-of-the-art LLM metric called LLM-SQL-Solver
Execution Accuracy â failson â AI-augmented SQL
confidence 90% ¡ Current metrics, which rely on exact query results and deterministic execution, systematically fail against the flexible, non-deterministic outputs of AI operators.
SemBench â usedin â Multilayered Evaluation Framework
confidence 90% ¡ We used SemBench [4]... To produce the queries needed to test our evaluation methods
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:SQL has been augmented with AI operators, enabling modern data analytics platforms to derive insights from both structured and unstructured data. We observe that while current Text-to-SQL systems can successfully generate these AI-augmented queries, reliably evaluating their correctness remains a critical open challenge. Current metrics, which rely on exact query results and deterministic execution, systematically fail against the flexible, non-deterministic outputs of AI operators. In this paper, we formalize these unique evaluation failure modes and introduce a Multilayered Evaluation Framework that decouples deterministic database logic from flexible AI semantics. We test our approach across both industry (BigQuery) and academic (ThalamusDB) systems. We demonstrate that traditional Execution Accuracy severely penalizes valid queries, achieving as low as a 25% detection rate for correct translations. Furthermore, even a state-of-the-art LLM-based autorater falsely rejects 32% of accurate queries due to the complexity of judging both relational and AI components simultaneously. By validating the standard relational logic and the AI operations separately, our framework achieves state-of-the-art overall accuracy across both platforms (up to 97.2%), proposing a reliable standard for benchmarking AI-powered SQL generators.
Tags
Links
- Source: https://arxiv.org/abs/2609.21133v1
- Canonical: https://arxiv.org/abs/2609.21133v1
Trouble viewing inline? Open PDF directly â
Full Text
37,993 characters extracted from source content.
Expand or collapse full text
The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators Tarfah Alrashed Google tarfah@google.com Fatma Ăzcan Google fozcan@google.com Per Jacobsson Google pjacobsson@google.com Tal Neiman Google talneiman@google.com Xianshun Chen Google xianshun@google.com Abstract SQL has been augmented with AI operators, enabling modern data analytics platforms to derive insights from both structured and unstructured data. We observe that while current Text-to-SQL sys- tems can successfully generate these AI-augmented queries, reliably evaluating their correctness remains a critical open challenge. Cur- rent metrics, which rely on exact query results and deterministic execution, systematically fail against the flexible, non-deterministic outputs of AI operators. In this paper, we formalize these unique evaluation failure modes and introduce a Multilayered Evaluation Framework that decouples deterministic database logic from flexi- ble AI semantics. We test our approach across both industry (Big- Query) and academic (ThalamusDB) systems. We demonstrate that traditional Execution Accuracy severely penalizes valid queries, achieving as low as a 25% detection rate for correct translations. Furthermore, even a state-of-the-art LLM-based autorater falsely rejects 32% of accurate queries due to the complexity of judging both relational and AI components simultaneously. By validating the standard relational logic and the AI operations separately, our framework achieves state-of-the-art overall accuracy across both platforms (up to 97.2%), proposing a reliable standard for bench- marking AI-powered SQL generators. CCS Concepts ⢠Information systemsâQuery languages; Database manage- ment system engines;⢠Computing methodologiesâNatural language processing;⢠General and referenceâ Evaluation. Keywords Text-to-SQL, AI Operators, Text-to-SQL Evaluation Methods, Se- mantic Correctness ACM Reference Format: Tarfah Alrashed, Fatma Ăzcan, Per Jacobsson, Tal Neiman, and Xianshun Chen. 2018. The Stochastic Shift: A New Evaluation Paradigm for Text- to-SQL with AI Operators. In Proceedings of Make sure to enter the correct Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym âX, Woodstock, NY Š 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X conference title from your rights confirmation email (Conference acronym âX). ACM, New York, NY, USA, 6 pages. https://doi.org/X.X 1 Introduction Modern database management systems are actively evolving be- yond structured tabular data by embedding AI operators directly into SQL. Platforms like Googleâs BigQuery, AlloyDB, Snowflake [8], and ThalamusDB [2] now provide native AI functions and vector search capabilities (e.g.,AI.IF,AI.SEARCH) [10,11]. These opera- tors allow a database to process unstructured data directly within a SQL query. For example, filtering a table of images to find spe- cific contentâsuch as queryingSELECT car_id WHERE AI.IF("is this car red?", car_image)âbridges a semantic gap that was previously impossible in traditional relational databases. Because writing these complex, hybrid queries manually can hinder adop- tion, translating natural language into AI-augmented SQL has emerged as a critical next step. However, this paradigm shift completely breaks current Text- to-SQL evaluation pipelines. Historically, the NLP and database communities optimized to translate natural language into precise, relational code [15,17]. As a result, the highly brittle Exact Match accuracy [13] gave way to Execution Accuracy (EX) [1,6] and Test Suite Accuracy (TSA) [19] to reduce false negatives. However, a core assumption underlies all these Text-to-SQL evaluation metrics: they were designed for an environment where SQL queries execute deterministically over structured, tabular data. Unlike User-Defined Functions (UDFs) that execute mathemati- cally sound, deterministic logic, LLMs rely on flexible, interpretive reasoning [3,10]. Embedded AI functions require a shift away from strict mathematical equivalence and toward ârelaxed semantic cor- rectness.â When evaluated under current metrics, valid queries fail completely. Exact Match protocols fail immediately because an LLM might inject slightly different but functionally equivalent phrasing into an operatorâs prompt (e.g., âis there a cat in this im- age?â vs. âdoes this picture contain a cat?â), instantly triggering a false negative. EX suffers a similar fate; because slightly different prompt phrasing can shift the LLMâs decision boundary on sub- jective, borderline cases, executing these functionally equivalent prompt variations over unstructured data often yields minor, ac- ceptable differences in the returned rows. In short, current metrics enforce rigid binary correctnessâartificially penalizing valid gener- ative reasoning and completely failing to interpret user intent in this hybrid domain. arXiv:2609.21133v1 [cs.DB] 17 Sep 2026 Conference acronym âX, June 03â05, 2018, Woodstock, NYAlrashed et al. To overcome the limitations of strict execution metrics, the re- search community has increasingly explored LLM-based autoraters to evaluate semantic correctness [18]. However, as we observe in our experiments, evaluating a query that combines deterministic database logic and probabilistic AI prompts overloads these au- toraters. This creates an evaluation conflict: the LLM struggles to simultaneously verify strict relational boundaries (e.g., tracking DISTINCTorUNIONconstraints) while performing fuzzy semantic matching on the embedded AI prompts, leading them to falsely reject a large number of valid queries. Consequently, creating reli- able benchmarks for Text-to-SQL with AI operators necessitates a new evaluation methodology that separates the deterministic relational part of a query from the flexible semantic evaluation of its AI prompts. In this paper, we expose the unique failure modes of AI-augmented SQL when evaluated via EX and a state-of-the-art au- torater, and present a novel benchmarking architecture engineered to decouple relational execution from semantic validation. Specifically, this paper makes the following contributions: ⢠We introduce a Multilayered Evaluation Framework that uses a SQL Split module to decouple deterministic relational logic from non-deterministic AI-augmented parts, paired with an LLM autorater to evaluate prompt intent independently of database execution. ⢠We demonstrate that while EX severely penalizes valid AI- augmented queries (detecting as few as 25% of correct trans- lations), and state-of-the-art LLM autoraters struggle with hybrid evaluation (falsely rejecting up to 32% of accurate queries), our decoupled framework achieves state-of-the-art overall accuracy across both platforms (up to 97.2%). â˘We formally categorize the failure modes of Text-to-SQL generation with AI operators and outline a roadmap of open challenges for future generative query builders. â˘We open-source our complete evaluation framework, includ- ing all baseline evaluation scripts and prompting templates: https://figshare.com/s/c75141469fb6dc74b890 2 Background Standard SQL and relational databases were designed for structured, tabular data. Traditional operators rely on deterministic logic to filter, join, or aggregate numerical and categorical fields, but they cannot comprehend unstructured data like free-text, images, or audio. Historically, extracting insights from such data required exporting it out of the database into external machine learning pipelines, introducing significant data-movement overhead and architecture complexity. The integration of LLMs has addressed this limitation by em- bedding semantic understanding directly into the database via AI operators. By bringing generative models to the data, these oper- ators allow users to query unstructured data using the familiar, declarative SQL interface. While LLM inference introduces compu- tational costs, AI operators bridge the gap between strict relational logic and probabilistic reasoning, enabling tasks like semantic fil- tering and classification without requiring external pipelines or task-specific fine-tuning. This in-database approach is gaining rapid support in both academia and industry [12,15]. Academic systems like ThalamusDB pro- vide operators such asNLfilterandNLjointo process multi- modal data, while enterprise platforms like Google Cloudâs Big- Query now include built-in AI functions (e.g.,AI.IF,AI.SCORE, and AI.SEARCH). To illustrate, suppose a user wants to find electric cars with a dead battery based on diagnostic audio recordings. Traditional SQL cannot filter on raw audio content. However, with AI operators, this becomes a straightforward query. In BigQuery, anAI.IFoperator evaluates a natural language prompt directly against the audio: SELECT DISTINCT cars.car_id FROM cars, car_audio WHERE cars.car_id = car_audio.car_id AND cars.fuel_type ='Electric' AND AI.IF(("Return true if the car recording indicates a dead battery, false otherwise.", car_audio.audio_path), connection_id =>'...'); Academic systems offer analogous functions. For example, Tha- lamusDB executes the same intent using the NLfilter command: SELECT DISTINCT cars.car_id FROM cars, car_audio WHERE cars.car_id = car_audio.car_id AND cars.fuel_type ='Electric' AND NLfilter(car_audio.audio_path, "Return true if the car recording indicates a dead battery, false otherwise."); These examples show AI operators combining deterministic filtering on structured data (e.g.,fuel_type = âElectricâ) with stochastic, semantic evaluation of unstructured content. 3 The Breakdown of Current Evaluation Current performance metrics systematically fail when evaluating Text-to-SQL with AI operators. While previous studies have already highlighted flaws in EX for standard SQL [13,19], the introduction of AI operators causes this metric to break down completely. To illustrate why existing pipelines break down, consider this natu- ral language request: âCount of positive reviews for movie taken_3. Return the count as positive_review_cnt.â A benchmark dataset might define the âgold-standardâ reference query as follows: SELECT COUNT(*) AS positive_review_cnt FROM movie.reviews WHERE id ='taken_3' AND AI.IF(('Determine if the following movie review is positive:', reviewText), connection_id =>'...'); -- Execution Result: 9 Now suppose a Text-to-SQL model generates this functionally equivalent query: SELECT COUNT(*) AS positive_review_cnt FROM movie.reviews WHERE id ='taken_3' AND AI.IF(('Is this review positive? Review:', reviewText), connection_id =>'...'); -- Execution Result: 11 Both queries are valid translations of the userâs intent. In fact, analyzing underlying data reveals that the 11 reviews returned by the generated SQL explicitly include all 9 benchmark reviews. Yet, under EX, the primary evaluation metric for modern leaderboards, this functionally correct generation scores a 0%. The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI OperatorsConference acronym âX, June 03â05, 2018, Woodstock, NY This failure occurs because EX assumes execution is strictly deterministic. While standard SQL filters (e.g.,id = âtaken_3â) return absolute binary results, AI operators do not. Slightly dif- ferent prompt phrasing could cause the underlying LLM to shift its decision boundary on subjective, borderline cases. Because the generated query categorized 11 reviews as positive while the bench- mark categorized 9, EX enforces rigid mathematical equivalence (11 != 9) and falsely rejects the generated query. This rigid rejection exposes a fundamental incompatibility: cur- rent benchmarks demand identical data intersections, whereas AI- augmented SQL relies on probabilistic semantic analysis. Evaluat- ing this emerging domain requires a shift toward relaxed semantic correctness [18]. The objective is no longer to achieve strict set equivalence, but to capture the userâs intent. If a generated query accurately targets the semantic goalâeven if the data varies slightly due to LLM samplingâit should be accepted as correct. Automating the measurement of this semantic alignment forms the theoretical foundation of our Multilayered Evaluation Framework. 4 Multilayered Evaluation Framework To address the limitations of existing evaluation methods, we pro- pose a Multilayered Evaluation Framework designed specifically for the dual nature of AI-augmented SQL. As illustrated in Figure 1, our framework operates in three phases: (1)Splitting: Decoupling a queryâs deterministic relational structure from its AI component. (2)Execution: Running both the full queries (benchmark and generated) and their newly isolated split components against the database to gather their respective result sets. (3) Autorating: Feeding the database schema, natural language question, queries, and execution results into an LLM-as-a- judge for independent semantic evaluation. By isolating the standard SQL structure from the AI tasks and capturing their distinct execution behaviors, we prevent valid gener- ative queries from being falsely rejected due to language variations, non-deterministic execution, or the use of alternative AI operators. SQL Splitter. Because current metrics strictly penalize any varia- tion or non-determinism within the AI component, our framework first isolates the structural validity of a relational query from its semantic intent. The âSQL Splitterâ is a general architectural com- ponent that can be implemented in various ways, such as using a traditional Abstract Syntax Tree (AST) parser or an LLM. We use a few-shot LLM-based SQL Splitter. Given the relative simplicity of SemBench queries, it proved highly effective at di- viding benchmark and generated queries into structural and AI components, as shown below: -- Original SQL SELECT p.car_id FROM cars_dataset.car_m as x JOIN cars_dataset.cars AS p ON p.car_id = x.car_id WHERE AI.IF(prompt => ("Returns true if the vehicle image contains both a puncture and paint scratches.", x.image), connection_id =>'...') LIMIT 100; -- Split SQL (Structural Part) SELECT p.car_id FROM cars_dataset.car_m as x JOIN cars_dataset.cars AS p ON p.car_id = x.car_id WHERE TRUE LIMIT 100; -- Split SQL (AI Part) SELECT x.image FROM cars_dataset.car_m as x WHERE AI.IF(prompt => ("Returns true if the vehicle image contains both a puncture and paint scratches.", x.image), connection_id =>'...') LIMIT 100; Independent Autorating. Following the split, an LLM-based Au- torater evaluates semantic similarity independently of strict execu- tion matching. We feed the Autorater the natural language question, the database schema, the full queries (both benchmark and gener- ated), their split components, and their respective execution results. The Autorater outputs a detailed breakdown of the queryâs ac- curacy. First, it scores overall, relational, and AI similarity on a 1 to 5 scale, as this 5-point grading has been shown to match hu- man judgment best [7]. We consider any score over 3 to be correct. This captures relaxed semantic correctness by accepting function- ally equivalent (4) or perfectly aligned (5) queries, while strictly rejecting ambiguous or partially correct outputs (â¤3). Finally, the Autorater flags the primary source of any errors (e.g., relational logic vs. AI prompt text) and provides a clear, written explanation for its ratings. 5 Experiments 5.1 Experimental Setup Model. In our experiment, we used gemini-3.1-pro-preview con- figured via the Google GenAI SDK (targeting the Google Cloud Vertex AI backend) using the default sampling parameters (tem- perature = 1.0, top-P = 0.95, and top-K = 40, with no random seed set). To generate executable SQL, the model relied on in-context learning with schema-specific instructions and programmatic few- shot examples. We enforced a structured output mode during SQL splitting and autorater evaluation to guarantee schema-conforming responses. Finally, the benchmark and generated queries were exe- cuted directly in their respective database engines (BigQuery and ThalamusDB) to get execution data for evaluation. Dataset. We used SemBench [4], a benchmark designed for test- ing database queries on datasets containing text, image, and audio data. SemBench consists of 55 queries across five real-world scenar- ios, covering tasks like filtering, joining, classifying, and ranking multimodal data. Although SemBench includes queries for several systems (including Python-based tools like LOTUS and Palimpzest), we focused exclusively on BigQuery and ThalamusDB because our work targets Text-to-SQL systems with AI operators. Evaluation Baselines. To empirically expose the failure modes of current evaluation methods, we compare our proposed Multilayered Evaluation Framework against two baselines: Execution Accuracy (EX) and a state-of-the-art LLM metric called LLM-SQL-Solver [18]. LLM-SQL-Solver explores whether an LLM can be used to deter- mine the equivalence between two SQL queries under two defined notions: semantic equivalence and relaxed equivalence. It is an entirely prompt-based approach that relies purely on the query text and schema, without any access to a real SQL engine or live execu- tion data. To assist the LLM in this reasoning process, the authors of LLM-SQL-Solver presented two specific prompting techniques to force the model to output its intermediate reasoning before arriving at a final classification: ⢠Miniature & Mull (Semantic Equivalence): This technique evaluates semantic equivalence by asking the LLM to gen- erate a small, imaginary database. The LLM then mentally Conference acronym âX, June 03â05, 2018, Woodstock, NYAlrashed et al. SQL Splitter LLM-based decomposition Benchmark Data Count the number of red Toyotas. NL Query DB Schema + Sample Rows SQL car_idmakeimage 1ToyotaThe brakes squeak loudly in the cold. 2FordThe infotainment screen freezed. 3ToyotaUneven panel gaps on the trunk. Generated SQL SQL Splitter LLM-based decomposition SELECT car_id FROM cars WHERE make = "TOYOTA" Structural SQL (Benchmark) AI SQL (Benchmark) Structural SQL (Generated) AI SQL (Generated) SQL Execution Engine Execution Results Autorater Internal-Structure-Semantic-Interpreter Strict Relational Validator Exact logical and structural match Relaxed AI Validator Matches prompt meaning and intent Relaxed Result Validator Full query verification Structured Verdict "scores": "overall_similarity": 5, "relational_similarity": 5, "ai_similarity": 4 , "analysis": "Primary_error_source": None, "justification": "..." SELECT image FROM cars WHERE AI.IF("Is the color of this car red?", image); SELECT car_id FROM cars WHERE make = "TOYOTA" SELECT image FROM cars WHERE AI.IF("Car in image is red.", image); SELECT car_id FROM cars WHERE make = "TOYOTA" AND AI.IF("Is the color of this car red?", image); SELECT car_id FROM cars WHERE make = "TOYOTA" AND AI.IF("Car in image is red.", image); Figure 1: Overview of the Multilayered Evaluation Framework for AI-augmented SQL, operating in three phases: Splitting (decoupling relational structure from AI components), Execution (running queries and isolated components against the database), and Autorating (independent semantic evaluation via an LLM-as-a-judge). simulates executing both queries on this mock data to find counterexamples yielding different results. â˘Explain & Compare (Relaxed Equivalence): This technique evaluates relaxed equivalence by guiding the LLM to first step through and explain the underlying logic of each query. It then compares these textual explanations to see if the queries share the same pragmatic intention, ignoring minor structural differences. For our evaluation, we used both the Miniature & Mull and Explain & Compare approaches as our LLM baselines, passing them custom few-shot examples tailored to our AI operators. To ensure a fair comparison and isolate the actual impact of our SQL Split method, we also evaluated augmented versions of these baselines that included the execution results for both the benchmark and generated queries. This removes any potential information gap, ensuring that both the LLM baselines and our proposed framework have access to the exact same data during evaluation. Text-to-SQL Generation with AI Operators. To produce the queries needed to test our evaluation methods, we implemented a Text- to-SQL generator. This generator translates a natural language question, a database schema, and a small random sample of rows into an executable query using in-context learning (ICL) with a few-shot prompt. Based on preliminary experiments, we found this prompting approach was strictly necessary to teach the LLM the novel syntax required to write valid AI-augmented SQL. To establish a reliable baseline, we manually verified the generated outputs. This analysis revealed that our generator successfully produced correct translations for 70% of the BigQuery queries and 80% of the ThalamusDB queries. This variation created a realistic mix of correct and flawed queries to test our evaluation framework. 5.2 Experimental Results While our manual verification confirmed good generation quality (70% to 80% success depending on the database platform), current automated evaluation methods scored this much lower across both systems (Table 1). EX failed drastically on both platforms. It correctly identified only 36.0% of valid BigQuery translations and just 25.0% for Tha- lamusDB, leading to terrible overall accuracies (52.8% and 26.7%). It also failed to consistently reject flawed queries. This highlights a critical flaw in relying solely on execution results: in one case, a generated query left out a requiredDISTINCTclause, but because the benchmark database currently had no duplicate values, both queries returned the same outcome. EX incorrectly accepted this flawed query. The standard LLM-SQL-Solver (Explain & Compare) performed somewhat better by focusing on logic. However, evaluating the entire hybrid query at once proved difficult. It struggled to balance correct acceptance and rejection across systemsâfor instance, while it achieved a high 91.7% True Positive Rate on ThalamusDB, it in- correctly accepted many flawed queries, dropping its True Negative Rate to just 66.7%. Adding execution data to this baseline did not fix the problem. For example, it incorrectly accepted a query that usedUNION DISTINCTinstead of the requiredUNION ALLsimply because both operations happened to return the same results on this specific database. We also evaluated the Miniature & Mull variation, which asks the LLM to mentally generate a counterexample database. Because it generated its own mock data, the LLM successfully imagined a database with duplicate values, allowing it to easily catch the missingDISTINCTerror that fooled EX (achieving a perfect 100.0% True Negative Rate on both systems). However, this counterexample generation falsely rejected around 32% of perfectly correct queries The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI OperatorsConference acronym âX, June 03â05, 2018, Woodstock, NY Table 1: Performance comparison of evaluation methods against our proposed framework on SemBench across two AI- augmented database systems (BigQuery and ThalamusDB). Rates reflect the correct acceptance of valid queries (True Positive) and correct rejection of flawed queries (True Negative). BigQueryThalamusDB Evaluation MethodologyTPR TNR AccuracyTPR TNR Accuracy Execution Accuracy (EX)36.0%90.9%52.8%25.0%33.3%26.7% LLM-SQL-Solver (Miniature & Mull) 68.0% 100.0%77.8%66.7% 100.0%73.3% LLM-SQL-Solver (Miniature & Mull w/ Exec Results)56.0%90.0%66.7%58.3% 100.0%66.7% LLM-SQL-Solver (Explain & Compare) 76.0%90.9%80.6%91.7%66.7%86.7% LLM-SQL-Solver (Explain & Compare w/ Exec Results) 84.0%81.8%83.3%91.7%66.7%86.7% Multilayered Eval Framework (Ours)100.0%90.9%97.2%91.7% 100.0% 93.3% across both databases. Furthermore, when we added execution results to this baseline, the real data directly conflicted with the promptâs command to invent counterexamples, causing its overall accuracy to drop even further. Our Multilayered Evaluation Framework solved these issues by separating the relational logic from the AI component. This allowed it to safely use execution data without being tricked by the execution results. By evaluating the parts separately, it correctly caught theUNIONerror that fooled the EX baseline. It consistently provided the best balance, capturing 100% of correct translations on BigQuery and achieving a perfect 100% True Negative Rate on ThalamusDB, ultimately setting a state-of-the-art overall accuracy for both platforms (97.2% and 93.3% respectively). 6 Open Challenges in Text-to-SQL Generation with AI Operators The integration of AI operators unlocks powerful semantic capa- bilities, but it simultaneously introduces new challenges in query generation. While our Multilayered Evaluation Framework estab- lishes a reliable methodology for evaluating these non-deterministic queries, generating them correctly presents failure modes distinct from traditional Text-to-SQL. Based on our preliminary analysis, we outline four critical challenges that future models must resolve. Structured vs. Semantic Misalignment. The fundamental assump- tion in query routing is that deterministic execution over struc- tured data is cheaper, faster, and more consistent than invoking AI operators. Meanwhile, embedding-based vector search occu- pies a middle groundâslower and less precise than relational logic, but much faster than AI operator inference. Generative agents frequently struggle to identify the optimal threshold along this spectrum. For instance, given the prompt âShow me rentals that include laundry,â an agent must determine if a strict check against a booleanhas_laundrycolumn is sufficient, or if it must compute anAI.IFfunction over an unstructured description column. Con- versely, models often over-rely on structured data, such as mapping âthree bedroomsâ to a deterministic filternumber_of_rooms = 3, which erroneously flags houses with three total rooms instead of parsing the unstructured property description. Data Source and Modality Ambiguity. Subjective queries require the generative agent to selectively route logic to the most viable unstructured data source. If a user asks to âShow me pet-friendly rentals,â the agent must independently decide whether the most accurate signal lies in raw property descriptions, user review text, or an image analysis of the property photos. Composing an exe- cution pathâdetermining whether to process these modalities via sequential AI filters or synthesize them in a single multi-modal promptârequires a new form of semantic query optimization. The Open-World vs. Closed-World Divide. Complex queries often cross the boundary between closed-world data (database columns) and open-world knowledge (external facts held by the LLM). A prompt demanding âSedans under 20,000 with a strong reputation for winter reliabilityâ requires an intelligent split. The model must apply standard SQL filters for exact matches (price < 20000 AND type = âsedanâ) alongside an AI function (AI.IF) to evaluate subjective knowledge like vehicle reliability records. Forced Answering of Ambiguous Queries. A common flaw in LLMs is the âforced answering problem,â where models default to gen- erating SQL even if the userâs request is invalid or incomplete [5]. For AI-augmented SQL, this behavior is computationally expen- sive. If a database lacks the data to answer an ambiguous question, allowing the LLM to generate a complex AI queryârather than simply refusing the promptâwastes tokens and severely increases execution latency. Developing mechanisms to reject unanswerable queries, such as those proposed for standard Text-to-SQL in related work [5], before they trigger AI operators is a critical requirement for real-world systems. 7 Related Work LLM-Powered Data Systems and AI-Augmented SQL. The research community is increasingly developing LLM-powered data process- ing systems that merge the reasoning capabilities of foundation models with the robust execution engines of traditional databases. Recent systems include LOTUS from Stanford [12], Palimpzest from MIT [9], DocETL from UC Berkeley [14], and ThalamusDB from Cornell [2]. Commercial products like BigQuery, AlloyDB, Snowflake, and Databricks already support these operations by augmenting standard SQL with AI operators. To drive the adoption of these AI queries by agents and human analysts, a robust natural language interface is required. However, while translating natu- ral language into AI-augmented SQL unlocks powerful analytical capabilities, it introduces fundamental evaluation challenges that Conference acronym âX, June 03â05, 2018, Woodstock, NYAlrashed et al. existing Text-to-SQL evaluation methods are unequipped to handle, motivating our evaluation framework. Evolution of Text-to-SQL Benchmarks. While current benchmarks like Spider [16] and BIRD [6] drove traditional Text-to-SQL progress, evaluating AI-augmented systems requires specialized datasets and relaxed metrics. Recently, SemBench [4] was proposed to assess LLM-powered multi-modal data systems. However, it primarily evaluates the computational cost, latency, and accuracy of the data- base systems themselves. To train and evaluate generative models for the natural language translation task, future benchmarks must contain a large volume of complex AI-operator queries. Critically, these datasets must incorporate mechanisms to measure relaxed semantic correctness. Only by moving beyond the rigid constraints of EX can the community accurately evaluate next-generation AI- powered query engines. LLM-as-a-Judge and Semantic Evaluation. To overcome the brit- tleness of execution-based metrics, recent natural language gen- eration research has increasingly adopted LLM-as-a-Judge tech- niques [18]. These paradigms leverage foundation models to eval- uate the semantic alignment and intention of generative outputs, rather than demanding execution equivalence. Our Multilayered Evaluation Framework extends this concept into the structured query domain by employing a SQL Split technique, isolating the deterministic relational logic from the stochastic AI part, and em- ploying an autorater specifically tuned to assess the semantic intent of embedded AI operators. 8 Conclusion & Future Work To overcome the failure of strict execution metrics and baseline LLMs on AI-augmented SQL, we introduced a Multilayered Evalua- tion Framework that separates deterministic relational logic from non-deterministic AI components. Combining database execution with an independent LLM autorater, our framework achieves state- of-the-art accuracy on BigQuery (97.2%) and ThalamusDB (93.3%). Crucially, it reveals that current generators produce valid queries 70â80% of the timeâa success rate obscured by EX (which detected as few as 25%) and existing LLM autoraters (which falsely rejected up to 32% of valid outputs). With this evaluation framework in place, our future work focuses on building a large-scale, challeng- ing benchmark for Text-to-SQL with AI operators, targeting the failure modes in Section 6. By evaluating intelligent data routing, multi-modal ambiguity, and safe refusal, this dataset will serve as the definitive testing ground for next-generation query engines. References [1]Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R. Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL Easier to Infer from Natural Language Specifications. In Findings of the Association for Computational Linguistics: EMNLP 2021, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, Dominican Republic, 2030â2042. doi:10.18653/v1/2021.findings-emnlp.174 [2]Saehan Jo and Immanuel Trummer. 2024. ThalamusDB: Approximate Query Processing on Multi-Modal Data. Proc. ACM Manag. Data 2, 3, Article 186 (May 2024), 26 pages. doi:10.1145/3654989 [3]Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al.2023. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv preprint arXiv:2310.03714 (2023). [4]Jiale Lao, Andreas Zimmerer, Olga Ovcharenko, Tianji Cong, Matthew Russo, Gerardo Vitagliano, Michael Cochez, Fatma Ăzcan, Gautam Gupta, Thibaud Hottelier, H. V. Jagadish, Kris Kissel, Sebastian Schelter, Andreas Kipf, and Im- manuel Trummer. 2026. SemBench: A Benchmark for Semantic Query Processing Engines. arXiv:2511.01716 [cs.DB] https://arxiv.org/abs/2511.01716 [5]Gyubok Lee, Woosog Chay, Seonhee Cho, and Edward Choi. 2024. Trust- SQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring. arXiv:2403.15879 [cs.AI] https://arxiv.org/abs/2403.15879 [6] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM already serve as a database interface? a big bench for large-scale database grounded text-to-SQLs. In Proceedings of the 37th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NeurIPS â23). Curran Associates Inc., Red Hook, NY, USA, Article 1835, 28 pages. [7]Weiyue Li, Minda Zhao, Weixuan Dong, Jiahui Cai, Yuze Wei, Michael Pocress, Yi Li, Wanyan Yuan, Xiaoyue Wang, Ruoyu Hou, Kaiyuan Lou, Wenqi Zeng, Yutong Yang, Yilun Du, and Mengyu Wang. 2026. Grading Scale Impact on LLM-as-a-Judge: Human-LLM Alignment Is Highest on 0-5 Grading Scale. arXiv:2601.03444 [cs.CL] https://arxiv.org/abs/2601.03444 [8]PaweĹ Liskowski, Benjamin Han, Paritosh Aggarwal, Bowei Chen, Boxin Jiang, Nitish Jindal, Zihan Li, Aaron Lin, Kyle Schmaus, Jay Tayade, et al.2026. Cortex AISQL: A Production SQL Engine for Unstructured Data . In Companion of the International Conference on Management of Data. 400â412. [9]Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baile Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, Rana Shahout, et al.2025. Palimpzest: Optimizing ai-powered analytics with declarative query processing. In Proceedings of the Conference on Innovative Database Research (CIDR). 2. [10]Liana Patel, Carlos Guestrin, and Matei Zaharia. 2026. Towards AI-Native Data Systems with the Semantic Operator Model and LOTUS. IEEE Data En- gineering Bulletin (2026). http://sites.computer.org/debull/A26mar/A26MAR- CD.pdf#page=61 [11]Liana Patel, Siddharth Jha, Parth Asawa, Melissa Pan, Carlos Guestrin, and Matei Zaharia. 2024. Semantic Operators: A Declarative Model for Rich, AI-based Analytics Over Text Data. (2024). arXiv:2407.11418 https://arxiv.org/abs/2407. 11418 [12] Liana Patel, Siddharth Jha, Melissa Pan, Harshit Gupta, Parth Asawa, Carlos Guestrin, and Matei Zaharia. 2025. Semantic Operators: A Declarative Model for Rich, AI-based Data Processing. arXiv:2407.11418 [cs.DB] https://arxiv.org/abs/ 2407.11418 [13] Mohammadreza Pourreza and Davood Rafiei. 2023. Evaluating cross-domain text-to-SQL models and benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 1601â1611. [14] Shreya Shankar, Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. 2025. DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing. Proc. VLDB Endow. 18, 9 (May 2025), 3035â3048. doi:10. 14778/3746405.3746426 [15]Ruoxi Sun, Sercan Ă. Arik, Alex Muzio, Lesly Miculicich, Satya Gundabathula, Pengcheng Yin, Hanjun Dai, Hootan Nakhost, Rajarishi Sinha, Zifeng Wang, and Tomas Pfister. 2024. SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended). arXiv:2306.00739 [cs.CL] https://arxiv.org/abs/2306. 00739 [16]Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Ellen Riloff, David Chiang, Julia Hockenmaier, and Junâichi Tsujii (Eds.). Association for Computational Linguistics, Brussels, Belgium, 3911â3921. doi:10.18653/v1/D18- 1425 [17]John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference on Artificial Intelligence - Volume 2 (Portland, Oregon) (AAAIâ96). AAAI Press, 1050â1055. [18]Fuheng Zhao, Jiayue Chen, Lawrence Lim, Ishtiyaque Ahmad, Divyakant Agrawal, and Amr El Abbadi. 2025. LLM-SQL-Solver: Can LLMs Determine SQL Equivalence?. In 2025 IEEE International Conference on Big Data (BigData). IEEE, 1887â1894. [19] Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic Evaluation for Text-to-SQL with Distilled Test Suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 396â411. doi:10.18653/v1/2020.emnlp-main.29