Paper deep dive
Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL
Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan, Yilin Xiao, Hao Chen, Qinggang Zhang, Xiao Huang, Feiran Huang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern text-to-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm's marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search.
Tags
Links
- Source: https://arxiv.org/abs/2608.28432v1
- Canonical: https://arxiv.org/abs/2608.28432v1
Trouble viewing inline? Open PDF directly →
Full Text
83,902 characters extracted from source content.
Expand or collapse full text
Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL Conference: The 20th ACM International Conference on Web Search and Data Mining; February 15–19, 2027; Hong KongCCS: Information systems Relational database query languagesCCS: Computing methodologies Natural language processing Jiayan Lin Note: All authors contributed equally to this research. Affiliation: Jinan University, Guangzhou, China email: ljiayan@stu2023.jnu.edu.cn , Yujia Liu Affiliation: Jinan University, Guangzhou, China email: yujialiu@stu2023.jnu.edu.cn , Zijin Hong Affiliation: The Hong Kong Polytechnic University, Kowloon, Hong Kong email: zijin.hong@connect.polyu.hk , Zheng Yuan Affiliation: The Hong Kong Polytechnic University, Kowloon, Hong Kong email: yzheng.yuan@connect.polyu.hk , Yilin Xiao Affiliation: The Hong Kong Polytechnic University, Kowloon, Hong Kong email: yilin.xiao@connect.polyu.hk , Hao Chen Affiliation: City University of Macau, Taipa, Macau email: sundaychenhao@gmail.com , Qinggang Zhang Affiliation: Jilin University, Changchun, China email: qinggangzhang@jlu.edu.cn , Xiao Huang Affiliation: The Hong Kong Polytechnic University, Kowloon, Hong Kong email: xiao.huang@polyu.edu.hk and Feiran Huang Affiliation: Beihang University, Beijing, China email: huangfr@buaa.edu.cn © , 2027 Abstract. Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern text-to-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm’s marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search. Keywords: Text-to-SQL, In-context Learning, Large Language Models, Accuracy-Cost Trade-offs 1. Introduction The advent of large language models (LLMs) has driven remarkable progress on text-to-SQL (Hong et al., 2025), a foundational task that translates natural language questions into executable SQL queries over relational databases. In particular, in-context learning (ICL) approaches (Wang et al., 2025b) have substantially pushed execution accuracy on widely used benchmarks such as BIRD (Li et al., 2023) and Spider (Yu et al., 2018) by assembling increasingly elaborate pipelines around the base generator. ICL is particularly suitable for such analysis because its improvements are largely realized through inference-time pipeline design, allowing the marginal benefit and added cost of individual components to be measured while holding the underlying backbone fixed. A modern ICL text-to-SQL system rarely issues a single prompt: it retrieves in-context demonstrations (Shi et al., 2025), links the relevant schema (Li et al., 2025a), samples and selects among multiple candidates (Lee et al., 2025), and refines the chosen query through execution feedback or multi-agent coordination (Wang et al., 2025a). Each successive system stacks more such modules atop the last, propelling state-of-the-art accuracy to new heights on public leaderboards. These increasingly modular pipelines still create a growing design space in which accuracy gains may often arise in practice from very different combinations of inference-time components. Figure 1. Overview of a modular ICL Text-to-SQL pipeline and its accuracy–cost trade-offs. Five recurring modules span pre-processing, SQL generation, and post-processing, but their benefits and inference costs vary across backbones, yielding different cost-efficient configurations under fixed budgets.A schematic comparison of rising Text-to-SQL leaderboard accuracy and increasing token and dollar costs as additional pipeline modules are added. Figure 2. Taxonomy of ICL Text-to-SQL pipelines, organizing five modules across three pipeline stages into 17 paradigm-level configurations and 24 representative methods, including methods that naturally span multiple paradigms within the pipeline. Repeated method names indicate systems that combine design paradigms across modules or use multiple strategies within the same module, providing a unified view of how modern ICL Text-to-SQL systems assemble pipeline components in practice.A circular taxonomy diagram organizing five ICL text-to-SQL modules across preprocessing, SQL generation, and post-processing stages, with branches for 17 paradigm-level configurations and representative methods. However, is the accuracy growth on text-to-SQL leaderboards reliably worth its disproportionately growing cost? Figure 1 illustrates a concerning trend: execution accuracy keeps rising as more modules are stacked, while the cost required to gain each additional point grows disproportionately. Pipeline components, however, are typically reported only through a single end-to-end accuracy number for the system. This aggregate-only reporting practice raises two concerns: 1) The marginal contribution of individual modules is overshadowed by aggregate accuracy, as the same end-to-end number may be reached through entirely different module combinations, leaving unclear which paradigms carry the gain and which are merely along for the ride; 2) Cost is rarely accounted for jointly with accuracy, despite agentic refinement loops and multi-candidate sampling consuming several times, and in some cases an order of magnitude, more tokens and dollars than a single-pass baseline. These concerns make it difficult for practitioners to decide which modules are worth adding for a backbone, and how far the pipeline should go before engineering ceases to pay off. They therefore motivate us to study this recurring class of modular ICL pipelines under controlled settings, where inference-time modules can be varied while holding the backbone fixed to more reliably attribute their marginal accuracy and cost. Studies have begun to interrogate aspects of text-to-SQL pipelines, such as ablations on schema linking (Maamari et al., 2024) or comparisons among candidate selection strategies (Pourreza et al., 2025), while several surveys catalog representative methods and their design choices (Li et al., 2024). However, these efforts either evaluate isolated components within a single system or report each method as a monolith without disentangling its constituent paradigms, leaving open how each module’s contribution shifts across backbones of differing capability and reasoning style. In this context, effectively quantifying the marginal accuracy–cost tradeoff of each design paradigm in ICL text-to-SQL pipelines remains a significant challenge. As a solution, this paper conducts a controlled paradigm-level evaluation that decomposes the ICL text-to-SQL pipeline into recurring modules, instantiates each design choice as a paradigm-level configuration under a unified implementation, and isolates its marginal contribution alongside the incurred cost across backbones spanning diverse capability levels and reasoning styles. Unlike accuracy-only evaluations, we jointly examine execution accuracy and inference cost, revealing how each paradigm’s value shifts with model capability. Overall, our contributions in this paper are listed as follows: • We conduct the first controlled, paradigm-level accuracy–cost evaluation of in-context learning (ICL) text-to-SQL pipelines, instantiating 17 paradigm-level configurations across five recurring modules under a single unified implementation. • We reveal that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other paradigms instead help only under backbone-dependent conditions, exposing the unreliability of aggregate-only accuracy reporting in practice. • We further show that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline, and distill these findings into a tiered cost-efficient configuration guideline validated on five additional backbones across capability levels without per-paradigm search. 2. Pipeline Decomposition We decompose the ICL Text-to-SQL pipeline into three main stages, Pre-Processing, SQL Generation, and Post-Processing, comprising five recurring functional modules: Example Retrieval, Schema Linking, Generation Strategy, Candidate Selection, and Refinement. These yield 17 paradigm-level configurations; Figure 2 summarizes the taxonomy and 24 representative methods. For a question qiq_i with schema iS_i, the five modules are formalized below. (1) Example Retrieval. Retrieves a demonstration set iZ_i from trainD_train using a paradigm-specific representation ρe _e: (1) i=TopKz∈trainsim(ρe(qi),ρe(qz)).Z_i= *TopK_z _trainsim ( _e(q_i), _e(q_z) ). Here, qzq_z denotes the question in example z. E1E_1 Question Similarity directly uses raw question embeddings (Li et al., 2024); E2E_2 Masked Question Similarity masks schema-specific entities before retrieval (Gao et al., 2024a); and E3E_3 SQL Skeleton Similarity uses structural SQL query forms (Ren et al., 2024). (2) Schema Linking. For each schema element u∈iu _i, a paradigm-specific rule ℓs(u∣qi,i)∈0,1 _s(u q_i,S_i)∈\0,1\ determines whether u is retained: (2) ~i=u∈i|ℓs(u∣qi,i)=1⊆i. S_i= \u _i\; |\; _s(u q_i,S_i)=1 \ _i. S1S_1 Retrieval-Based Linking scores relevant schema elements via dense retrieval (Xie et al., 2025); S2S_2 Pseudo-SQL Guided Linking retains elements explicitly referenced by a preliminary SQL draft (Shi et al., 2025); S3S_3 Direct LLM Linking uses S3aS_3a Single-shot (Pourreza and Rafiei, 2023) or S3bS_3b Two-stage (Dong et al., 2023) LLM linking; and S4S_4 Agentic Linking uses iterative proposer-verifier exchanges. (3) Generation Strategy. Generates a pool iY_i of N SQL candidates: (3) i=y^i(k)k=1N=Generateg(qi,~i,i,N).Y_i= \ y_i^(k) \_k=1^N=Generate_g\! (q_i, S_i,Z_i;N ). Disabled retrieval uses i=∅Z_i= , while disabled linking uses ~i=i S_i=S_i. G1G_1 Direct Generation generates SQL directly (Deng et al., 2025); G2G_2 CoT-Enhanced Generation elicits intermediate reasoning (Pourreza et al., 2025); and G3G_3 Decomposition-Based Generation decomposes the question before generating the final SQL (Wang et al., 2025b). They may also be combined via parallel generators. (4) Candidate Selection. Selects one query from iY_i according to selection paradigm c: (4) y^isel=Selectc(y^i(k)k=1N)∈i. y_i^sel=Select_c\! (\ y_i^(k)\_k=1^N ) _i. C1C_1 Execution-Based Voting selects from the largest result-equivalence group (Li et al., 2025b); C2C_2 LLM-Based Preference Selection uses an LLM to choose via C2aC_2a Scoring and Ranking (Gao et al., 2024b) or C2bC_2b Pairwise Comparison (Li et al., 2026). (5) Refinement. Maps the selected SQL y^isel y_i^sel to the final prediction under refinement paradigm r: (5) y^i=Refiner(y^isel,qi,~i). y_i=Refine_r\! ( y_i^sel,q_i, S_i ). R1R_1 Execution-Feedback Refinement rewrites only after an execution error or empty result (Cao et al., 2024); R2R_2 Self-Correction performs prompt-level review without execution (Xie et al., 2024); and R3R_3 Agentic Refinement coordinates multiple revision roles using execution feedback (Wang et al., 2025a). Without refinement, the selected SQL is simply returned unchanged (Hong et al., 2026a; Qu et al., 2025). Our taxonomy covers 24 representative ICL Text-to-SQL methods published between 2023 and 2026 that report on BIRD or Spider and provide sufficient pipeline detail for reliable paradigm-level classification. Methods may span multiple paradigms, especially through parallel generation or combined schema linking signals. 3. Experimental Setup This section describes the experimental setup for our paradigm-level analysis. We first introduce the datasets and backbone scope, then present the evaluation metrics and statistical testing, and finally detail the controlled implementation of the 17 paradigm configurations to ensure consistent and comparable assessment across paradigms and backbones within our unified framework. 3.1. Dataset and Backbones Dataset. We conduct the main paradigm-level analysis on the development split of BIRD (Li et al., 2023), which contains 1,534 question and SQL pairs over 11 databases. BIRD is widely used in recent ICL Text-to-SQL work and provides evidence annotations that are directly relevant to several paradigms studied in this paper. To examine cross-benchmark generalization, we additionally use Spider (Yu et al., 2018) (1,034 samples) to evaluate cross-benchmark transfer of the BIRD-derived pipeline configurations under the same protocol in § 5.2. Backbones. We run the full set of paradigm configurations on four primary backbones spanning different capability levels and reasoning behaviors: GPT-4o-mini, Gemini-2.5-Flash, DeepSeek-V4-Flash (reasoning), and GPT-5.4. To further examine the effect of reasoning behavior on generation strategies, we additionally evaluate Qwen3-235B in thinking mode and o3-mini in § 4.2. Finally, we evaluate the tiered guideline in § 6 on five additional backbones by applying the three prescribed tiers without per-paradigm search. 3.2. Metrics and Statistical Testing Accuracy metrics. For intuitive comparison, we follow the standard evaluation setting in the text-to-SQL community (Yu et al., 2018). Our primary evaluation metric is execution accuracy (EXEX). Given a set =(qi,yi)i=1nD=\(q_i,y_i)\_i=1^n of question and gold-SQL pairs and predicted SQLs y^ii=1n\ y_i\_i=1^n: (6) EX=1n∑i=1n[Exec(y^i)=Exec(yi)],EX\;=\; 1n _i=1^n 1\! [\, Exec( y_i)= Exec(y_i)\, ], where Exec(⋅) Exec(·) denotes the execution operator on the corresponding database. For configurations that sample multiple candidates, let y^i(k)k=1N\ y_i^(k)\_k=1^N denote the N sampled candidates for example i. We additionally report oracle candidate recall (OCROCR): (7) OCR=1n∑i=1n⋁k=1N[Exec(y^i(k))=Exec(yi)],OCR\;=\; 1n _i=1^n _k=1^N 1\! [\, Exec( y_i^(k))= Exec(y_i)\, ], which upper-bounds the EXEX achievable by any selection strategy over a fixed candidate pool. Cost and resource metrics. To compare paradigms along the accuracy–cost tradeoff, for a paradigm p instantiated in module m, we report cost per EXEX point (CPPCPP): (8) CPP(p)=Cost(p)−Cost(bm)100[EX(p)−EX(bm)],CPP(p)= Cost(p)-Cost(b_m)100 [EX(p)-EX(b_m) ], where bmb_m denotes the matched reference for module m, using the same sampling budget as p (N=5N=5 for Candidate Selection and N=1N=1 otherwise). Measured in USD/p, CPPCPP represents the additional cost to gain one percentage point of EXEX over the matched reference. CPPCPP is reported only when p strictly improves EXEX at strictly higher cost; paradigms that Pareto-dominate the reference are marked separately. We report all costs in USD and also record mean per-question input (IN) and output (OUT) token counts for comparison to better characterize resource overhead beyond aggregate API cost. Statistical testing. All EXEX gains over the matched reference are evaluated using McNemar’s test. We report 95% confidence intervals and determine statistical significance at α=0.05α=0.05; detailed numerical intervals are provided in the supplementary material for reference. 3.3. Controlled Implementation To ensure that quantitative comparisons reflect paradigm-level differences rather than incidental properties of specific systems (Hong et al., 2025), we re-implement the 17 configurations identified in § 2 as paradigm representatives under a unified controlled framework. Rather than reproducing any particular system, each configuration instantiates the defining mechanism of one paradigm while keeping shared implementation settings fixed within corresponding module across all backbones. Figure 3 summarizes these controlled implementations across all five modules. Detailed implementation settings and fallback behavior are provided in the final supplementary material. Baseline and single-swap protocol. We define a global baseline for each backbone with no retrieved demonstrations, the full schema, direct generation (G1G_1) with N=1N=1, and neither Candidate Selection nor Refinement. For each target module, we replace only that module with the paradigm under study while keeping other modules at their trivial settings, so that the observed difference can then be attributed directly to the target paradigm. Candidate Selection is the only exception to the N=1N=1 reference: because selection requires multiple candidates, we use a matched N=5N=5 baseline that returns the first candidate from the same candidate pool, thereby also separating the selector’s contribution from the gain due to multi-sample generation. For Generation Strategy, we additionally evaluate N=5N=5 sampling to examine effect on candidate-pool quality. 4. Single-Module Analysis Controlled Paradigm Implementation 1: qiq_i, iS_i, optional evidence, backbone, target module m, and paradigm setting 2: final prediction y^i y_i 3: Keep all non-target modules at their trivial settings. 4: if m is Example Retrieval then 5: Fix k=3k=3, the training retrieval pool, and the same frozen encoder. 6: i←retrieve with qi,E1,mask schema entities→retrieve,E2,draft SQL→skeleton→retrieve,E3. _i← casesretrieve with q_i,&E_1,\\ mask schema entities ,&E_2,\\ draft SQL ,&E_3. cases 7: else if m is Schema Linking then 8: Fix the full-schema input and always retain primary/foreign keys. 9: ~i←retrieval-based filtering,S1,draft SQL→referenced schema,S2,one-shot table/column selection,S3a,tables→columns,S3b,propose→verify→[revise],S4. S_i← casesretrieval-based filtering,&S_1,\\ draft SQL schema,&S_2,\\ one-shot table/column selection,&S_3a,\\ tables ,&S_3b,\\ propose →[revise],&S_4. cases 10: else if m is Generation Strategy then 11: Fix one backbone generation call at temperature 00. 12: i←generate SQL directly,G1,reason→generate final SQL,G2,decompose→generate final SQL,G3. _i← casesgenerate SQL directly,&G_1,\\ reason final SQL,&G_2,\\ decompose final SQL,&G_3. cases 13: else if m is Candidate Selection then 14: Fix the same N=5N=5 pool iY_i sampled from G1G_1. 15: Execute all candidates in iY_i. 16: y^isel←result-equivalence voting,C1,score/rank with execution outcomes,C2a,pairwise tournament,C2b. y_i^sel← casesresult-equivalence voting,&C_1,\\ score/rank with execution outcomes,&C_2a,\\ pairwise tournament,&C_2b. cases 17: else if m is Refinement then 18: Fix the same inputs and temperature 00; keep the last valid SQL on failure. 19: y^i←execute→[rewrite if error/empty],R1,review without execution feedback,R2,correct with execution feedback→verify→[correct],R3. y_i← casesexecute→[rewrite if error/empty],&R_1,\\ review without execution feedback,&R_2,\\ correct with execution feedback →[correct],&R_3. cases 20: end if 21: Run the remaining pipeline stages at their trivial settings. 22: return y^i y_i Gray italics: shared fixed settings. [⋅][·] denotes a conditional operation. Figure 3. Controlled implementation of the 17 paradigm configurations. For each target module, shared settings are fixed and all non-target modules remain at their trivial settings, so that only the paradigm-specific mechanism itself varies. Table 1. Module-level accuracy–cost results on BIRD dev across four LLMs. EX: execution accuracy (%); Cost: total API spend (USD); CPP: incremental USD per EX point over the matched-budget baseline (N=5N=5 for Candidate Selection, N=1N=1 otherwise; § 3.2). IN/OUT: mean per-question tokens (DeepSeek-V4-Flash reasoning tokens count as OUT). Significance uses McNemar’s test (α=0.05α=0.05; full CIs in the supplement). In CPP, numbers denote significant gains; “∘”, nonsignificant gains; “–”, no EX improvement; and “∗”, a significant gain at no higher cost. / mark the best/second-best results; OCR@5 is the oracle upper bound for N=5N=5. Paradigm GPT-4o-mini DeepSeek-V4-Flash Gemini-2.5-Flash GPT-5.4 EX↑ Cost↓ CPP↓ IN↓ OUT↓ EX↑ Cost↓ CPP↓ IN↓ OUT↓ EX↑ Cost↓ CPP↓ IN↓ OUT↓ EX↑ Cost↓ CPP↓ IN↓ OUT↓ Baselines N=1N=1 49.74 0.394 – 1510 51 56.32 0.547 – 1557 494 59.84 1.035 – 1642 73 61.28 7.305 – 1509 66 N=5N=5 49.67 1.972 – 7550 255 56.45 2.756 – 7787 2523 60.04 5.219 – 8222 374 61.41 35.864 – 7545 301 OCR@5OCR@5 56.91 1.972 – 7550 255 60.56 2.756 – 7787 2523 64.54 5.219 – 8222 374 66.82 35.864 – 7545 301 Example Retrieval E1E_1 50.07 0.462 ∘ 1806 50 56.06 0.613 – 1858 498 60.43 1.270 0.401 1842 110 62.58 8.602 0.995 1834 68 E2E_2 50.65 0.525 0.144 2009 68 56.84 0.717 0.326 2311 512 61.28 1.632 0.417 2207 161 63.69 9.613 0.957 2058 75 E3E_3 49.80 0.867 ∘ 3357 103 57.11 1.063 0.660 3708 622 60.10 2.421 ∘ 3358 228 63.49 14.879 3.417 3325 92 Schema Linking S1S_1 49.87 0.275 ∘ 873 81 55.93 0.360 – 964 357 59.13 1.117 – 857 188 60.63 5.102 – 826 84 S2S_2 52.15 0.574 0.075 1844 163 57.04 0.899 0.491 1871 1157 61.15 1.784 0.575 1855 243 62.39 10.955 3.293 1903 159 S3aS_3a 51.11 0.544 0.109 1936 107 56.19 0.782 – 1852 896 60.89 1.659 0.599 1924 202 60.82 9.672 – 1944 96 S3bS_3b 51.83 0.582 0.090 1528 250 55.87 0.757 – 1634 944 60.76 1.808 0.847 1577 282 59.97 12.806 – 1622 286 S4S_4 52.93 1.260 0.271 4478 249 56.52 1.380 ∘ 4520 953 61.34 2.870 1.224 4524 205 62.06 23.631 20.870 4568 266 SQL Generation G1G_1 49.74 0.394 – 1510 51 56.32 0.547 – 1557 494 59.84 1.035 – 1642 73 61.28 7.305 – 1509 66 G2G_2 53.98 0.503 0.026 1534 163 55.93 0.556 – 1602 493 62.71 1.522 0.170 1538 212 62.84 10.000 1.723 1533 179 G3G_3 51.24 0.534 0.093 1574 187 56.13 0.651 – 1650 689 60.76 1.925 0.976 1590 311 63.10 11.476 2.285 1573 237 Candidate Selection C1C_1 51.11 1.972 ∗ 7550 255 57.30 2.756 ∗ 7787 2523 60.89 5.219 ∗ 8222 374 62.71 35.864 ∗ 7545 301 C2aC_2a 50.52 2.625 0.771 9718 423 59.00 3.602 0.333 10319 3227 61.60 8.045 1.806 10018 896 63.82 53.348 7.249 9964 658 C2bC_2b 51.17 4.056 1.389 14962 666 58.54 4.743 0.953 14962 3562 61.93 10.622 2.858 14846 988 63.89 75.436 15.975 14933 790 Refinement R1R_1 54.63 0.471 0.016 1750 74 58.15 0.620 0.040 1697 596 62.32 1.099 0.026 1671 86 63.30 7.905 0.297 1592 78 R2R_2 51.83 1.030 0.305 3509 242 56.58 1.253 ∘ 3638 1098 60.95 3.211 1.964 3513 416 62.06 21.918 18.680 3518 366 R3R_3 53.26 1.365 0.276 4980 238 59.97 1.878 0.365 5007 1870 63.04 4.281 1.016 5116 502 64.02 28.578 7.769 5251 367 We instantiate each of the 17 paradigm configurations by replacing exactly one module in the global baseline pipeline, holding the remaining four at their trivial settings. This single-swap protocol isolates the marginal contribution of each paradigm. Table 1 reports the accuracy–cost and token statistics across all four backbones, while Figure 4 summarizes the ΔEX results with 95% confidence intervals. We organize the analysis around two questions below: Q1. Are there paradigms whose marginal benefit is robust across all four backbones, making them safe pipeline defaults? Q2. For paradigms whose benefit is not universal, does the variation correlate with model capability or reasoning style? 4.1. Universal Paradigms Across the 17 overall paradigm configurations, R1R_1 (Execution-Feedback Refinement) is the only paradigm whose benefit holds across all four backbones at consistently low cost. It yields gains between +1.83+1.83 and +4.89+4.89 p, with CPPCPP below 0.300.30 USD/p throughout. The mechanism is: a diagnostic execution is run on the candidate query, and the model is guided to rewrite only when execution errors or returns no rows. The call is therefore failure-gated rather than mandatory, which keeps the average overhead small while preserving the upside on the queries that actually need revision. The other two refinement paradigms illustrate why R1R_1’s combination is rare. R2R_2 (Self-Correction) significantly improves EXEX on three backbones but has no significant effect on the reasoning backbone DeepSeek-V4-Flash (+0.26+0.26 p, n.s.); even where it helps, its CPPCPP is far above R1R_1’s. R3R_3 (Agentic Refinement) is positive on all four backbones and posts the highest absolute EXEX in several columns, but at CPPCPP one to two orders of magnitude above R1R_1. The pattern across the three paradigms is consistent: execution feedback supplies information the model did not have at generation time, while prompt-internal correction adds only what the model could in principle have produced on its own. Among universally helpful paradigms, this information asymmetry, rather than reasoning quality, is thus what separates cheap from expensive in this setting. 4.2. Model-Dependent Paradigms The remaining four modules deliver gains that depend on the backbone. Rather than treating each module in isolation, we organize the discussion by backbone characteristics, since the modules’ contributions move together with model capability across tested backbones. A line chart showing execution-accuracy changes for 17 paradigm configurations across GPT-4o-mini, DeepSeek-V4-Flash, Gemini-2.5-Flash, and GPT-5.4, with 95 percent confidence bands and a zero-change baseline. Figure 4. ΔEX (p) relative to the matched-budget reference baseline for 17 paradigm configurations on BIRD dev. Shaded bands show 95% CIs for paired differences, and the dashed line marks ΔEX=0 =0. Statistical significance is assessed using McNemar’s test. Full numerical CIs are provided in the supplementary material for every paradigm on each backbone.A line chart showing execution-accuracy changes for 17 paradigm configurations across GPT-4o-mini, DeepSeek-V4-Flash, Gemini-2.5-Flash, and GPT-5.4, with 95 percent confidence bands and a zero-change baseline. Table 2. Generation strategy results on BIRD dev for two additional reasoning backbones. Qwen3-235B uses thinking mode. Parentheses report ΔEX over G1G_1 (p). Units: Cost in USD, CPP in USD/p, and IN/OUT in tokens per question. Backbone Paradigm EX↑ Cost↓ CPP↓ IN↓ OUT↓ Qwen3-235B (Thinking) G1G_1 57.82 4.55 – 1555 1134 G2G_2 58.02 (+0.20+0.20) 5.22 3.426 1529 1327 G3G_3 57.30 (−0.52-0.52) 5.84 – 1596 1495 o3-mini G1G_1 54.11 5.67 – 1509 463 G2G_2 53.46 (−0.65-0.65) 5.83 – 1546 477 G3G_3 53.06 (−1.04-1.04) 7.08 – 1538 665 Capability gradient: schema linking and example retrieval exhibit opposing trends. On GPT-4o-mini, the weakest backbone in our set, Schema Linking delivers its largest gains across the four backbones: S4S_4 (Agentic Linking) adds +3.19+3.19 p, and S2S_2, S3aS_3a, S3bS_3b all clear +1.3+1.3 p. The benefit shrinks sharply on stronger backbones. On GPT-5.4, only S2S_2 and S4S_4 yield positive gains, both at substantially higher CPPCPP than pre-processing paradigms typically incur on cheaper backbones. Schema linking acts as a capability prosthesis: its value falls as the backbone’s schema-grounding ability rises. Recent studies (Maamari et al., 2024; Yuan et al., 2025) report that sufficiently strong models can generate correct SQL even without explicit schema linking; we extend the observation to a four-backbone, cost-aware setting. Example Retrieval shows the opposite trend, though weakly. E2E_2 (Masked Question Similarity) is the best retrieval paradigm on every non-reasoning backbone, and its gain scales with baseline EXEX, from +0.91+0.91 p on GPT-4o-mini to +2.41+2.41 p on GPT-5.4. One reading is that stronger non-reasoning backbones extract reusable structure from demonstrations, but DeepSeek-V4-Flash departs from this trend in ways we cannot attribute to capability alone in this case. These trends suggest backbone-dependent preprocessing benefits. Reasoning split: external reasoning scaffolds help mainly non-reasoning backbones. Among the four primary backbones tested, Generation Strategy varies more than any other module. G2G_2 (CoT-Enhanced Generation) delivers +4.24+4.24 p on GPT-4o-mini at CPP=0.026CPP=0.026 USD/p, the second-largest single-module gain after R1R_1. The gain shrinks on Gemini-2.5-Flash and GPT-5.4, and turns negative on DeepSeek-V4-Flash (−0.39-0.39 p), with CPPCPP on reasoning-capable backbones one to two orders of magnitude higher. G3G_3 (Decomposition-Based Generation) follows a similar pattern, with negligible effect on DeepSeek-V4-Flash; the exception is GPT-5.4, where G3G_3 slightly exceeds G2G_2 (+1.83+1.83 vs. +1.56+1.56 p) at higher CPPCPP. Figure 5. Per-question input (IN) and output (OUT) token consumption across the 17 paradigm configurations for two backbones, GPT-4o-mini and DeepSeek-V4-Flash, on BIRD. Dashed horizontal lines mark the corresponding baseline token levels for direct comparison across configurations. Table 3. Cross-module interactions across four backbones. EX is combined execution accuracy (%), with gray values marking combinations that do not outperform the better of the two individual paradigms. Green and red ℐI denote super- and sub-additive interactions. The G1×RjG_1× R_j rows are omitted because they reduce to RjR_j and have ℐ=0I=0 by construction. Cost is reported in USD. All interaction values are measured in percentage points relative to the matched-budget reference baseline for each backbone. GPT-4o-mini DeepSeek-V4-Flash Gemini-2.5-Flash GPT-5.4 Combination EX↑ I Total Cost↓ EX↑ I Total Cost↓ EX↑ I Total Cost↓ EX↑ I Total Cost↓ G2×R1G_2× R_1 56.84 −2.02-2.02 0.612 57.82 +0.07+0.07 0.743 65.12 −0.07-0.07 1.971 64.60 −0.26-0.26 10.310 G2×R2G_2× R_2 55.87 −0.20-0.20 1.143 55.22 −0.97-0.97 1.457 63.62 −0.20-0.20 3.856 64.73 +1.11+1.11 24.751 G2×R3G_2× R_3 57.17 −0.33-0.33 1.525 59.84 +0.26+0.26 2.004 66.04 +0.13+0.13 4.748 65.19 −0.39-0.39 32.632 G2×E2G_2× E_2 54.50 −0.39-0.39 0.685 56.39 −0.07-0.07 0.793 64.34 +0.20+0.20 2.217 65.32 +0.07+0.07 12.190 G3×R1G_3× R_1 55.74 −0.39-0.39 0.657 57.89 −0.07-0.07 0.733 63.43 +0.19+0.19 1.956 65.38 +0.26+0.26 12.114 G3×R2G_3× R_2 52.28 −1.05-1.05 1.182 55.67 −0.72-0.72 1.501 59.45 −2.42-2.42 4.523 64.47 +0.59+0.59 26.360 G3×R3G_3× R_3 52.93 −1.83-1.83 1.441 60.17 +0.39+0.39 2.067 64.60 +0.64+0.64 5.304 66.56 +0.72+0.72 34.801 Because DeepSeek-V4-Flash is the only reasoning backbone among the primary four models, we further compare all three generation strategies on Qwen3-235B in thinking mode and o3-mini (Table 2). On Qwen3-235B, G2G_2 changes EXEX by +0.20+0.20 p and G3G_3 by −0.52-0.52 p. On o3-mini, the changes are −0.65-0.65 and −1.04-1.04 p, with higher cost in every case. Across the three reasoning backbones tested, external reasoning scaffolds provide no consistent accuracy gain over direct generation, despite higher cost. We nevertheless treat this split as suggestive rather than universal, since reasoning style, capability, and model family remain confounded. Pool-bound regime: candidate selection saturates regardless of backbone. All three selectors significantly improve EXEX over the N=5N=5 baseline on every backbone, but we do not test selector against selector; we therefore draw no ranking among C1C_1, C2aC_2a, C2bC_2b. More consequentially, taking the best selector per backbone, a substantial gap to the oracle ceiling remains everywhere, reaching 5.745.74 p on GPT-4o-mini. Candidate selection is recall-bound: once the candidate pool is fixed, further gains must come from improving the pool itself or revising the chosen query post-hoc. The fixed-pool recall ceiling, together with the substantial combined inference cost of multi-candidate generation and selection, motivates the exclusion of Candidate Selection from both the stacking analysis in § 5 and the cost-aware configuration guideline in § 6. 4.3. Token-Level Resource Decomposition USD cost alone does not reveal where a paradigm’s resource overhead comes from. Table 1 reports the complete IN/OUT token accounting across all four backbones, while Figure 5 visualizes two contrasting backbones. The source of token overhead closely follows the paradigm mechanism. Example Retrieval and LLM-based Candidate Selection are primarily input-heavy: E3E_3 substantially expands the prompt context, while C2bC_2b reaches about 15K input tokens. In contrast, Generation Strategy mainly shifts resource use toward output consumption through additional reasoning or decomposition. Schema Linking is more heterogeneous: S1S_1 reduces input through schema pruning, whereas S4S_4 incurs substantially larger input context from iterative interaction. Refinement shows a different pattern: R1R_1 remains close to the baseline in both IN and OUT, whereas R2R_2 and R3R_3 incur substantially larger token overhead. Across backbones, token consumption varies more strongly in OUT than IN. For a given paradigm, IN is stable across the four backbones, while the two-backbone visualization shows a pronounced OUT gap: DeepSeek-V4-Flash consistently produces substantially more OUT than GPT-4o-mini, with its reasoning traces contributing to the OUT total. This pattern suggests that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior, especially reasoning. 4.4. Single-Module Conclusion The single-module analysis surfaces one safe default and a structured account of when the rest help and where their resource overhead arises. Only R1R_1 qualifies as a universal, low-cost default; other modules’ contributions are backbone-dependent. The dependence has structure: input-side scaffolding tracks capability, generation-side scaffolding tracks reasoning style, and post-hoc selection saturates against a pool-bound ceiling. Resource use shows a complementary pattern: IN is more closely tied to pipeline structure, whereas OUT is more sensitive to backbone generation behavior. § 5 next asks whether these effects compose when paradigms are stacked, and how a fixed budget should be allocated across stages. 5. Cross-Module and -Model Tradeoffs The single-swap protocol discussed in § 4 isolates each paradigm’s marginal contribution, but two research questions remain unsolved: Q3. When two paradigms are used together, do their gains compose additively, or does their combination yield smaller or larger gains than expected across different backbones? Q4. Given a fixed budget, is it more cost-efficient to engineer a richer pipeline on a cheaper backbone, or instead to simply upgrade to a stronger backbone running a lean pipeline? 5.1. Module Interactions We study Q3 through a systematic analysis of module interactions. Since exhaustive coverage is infeasible, we focus on the full 3×33× 3 Generation × Refinement grid across all backbones. We choose this pair because refinement revises the SQL produced by generation, making their interaction particularly direct. For paradigms A and B, we quantify their interaction in percentage points as (9) ℐ=ΔEXpp(A&B)−ΔEXpp(A)−ΔEXpp(B),I\;=\; _p(A\&B)- _p(A)- _p(B), where positive ℐI indicates super-additive gains and negative ℐI indicates sub-additive gains, both relative to the baseline pipeline. Table 3 reports the resulting configurations and interaction terms. Generation × Refinement. Of the 24 nontrivial G2/G3×R1–3G_2/G_3× R_1--3 configurations, 17 achieve higher EXEX than the better individual paradigm. The pattern is strongest on GPT-5.4, where six combinations improve over both individual paradigms; five of six do so on both GPT-4o-mini and Gemini-2.5-Flash. DeepSeek-V4-Flash is the exception, with only one of six combinations improving over both paradigms. This is consistent with the single-module results in § 4: neither G2G_2 nor G3G_3 outperforms G1G_1 on this reasoning backbone, so the resulting combinations rarely outperform refinement alone. The interaction terms reveal how these gains compose across backbones. On GPT-4o-mini, stacking usually outperforms the better individual paradigm, but all six interactions are sub-additive. This may reflect the weaker backbone’s limited capacity to exploit generation and refinement jointly, yielding overlapping gains across stages. GPT-5.4 exhibits the opposite pattern: all six combinations outperform both individual paradigms, and four are super-additive. Generation × Example Retrieval. We further examine whether this backbone-dependent pattern extends to the input side through G2×E2G_2× E_2. The combination outperforms both individual paradigms on the three non-reasoning backbones, but not on DeepSeek-V4-Flash, where G2G_2 does not improve the baseline in isolation. ℐI remains small across all four backbones, ranging from −0.39-0.39 to +0.20+0.20 p, indicating near-additive composition for this module pairing overall. Overall pattern. Results from the two module pairings show that interaction effects differ systematically across backbones. Module stacking improves accuracy on most backbones, but how its gains compose varies with backbone capability. Since results from two-paradigm combinations do not determine the performance of complete multi-module stacks, § 5.2 evaluates backbone-specific stacks and reports end-to-end accuracy and cost jointly. 5.2. Stacking vs. Upgrading Table 4. Module-best pipelines on BIRD dev. Parentheses show ΔEX (p) over the N=1N=1 baseline; API costs are in USD. Stacks follow the cost-aware rule in § 5.2 and favor paradigms with low CPPCPP and broadly positive gains across tested models. Backbone Stack EXEX (%)↑ Cost↓ GPT-4o-mini E2+S2+G2+R1E_2+S_2+G_2+R_1 58.02 (+8.28+8.28) 0.759 DeepSeek-V4-Flash E2+S2+G1+R1E_2+S_2+G_1+R_1 61.15 (+4.82+4.82) 1.584 Gemini-2.5-Flash E2+S2+G2+R1E_2+S_2+G_2+R_1 67.14 (+7.30+7.30) 2.916 GPT-5.4 E2+S2+G2+R1E_2+S_2+G_2+R_1 69.43 (+8.15+8.15) 16.237 Figure 6. Cost–EX tradeoffs on Spider dev. Hollow circles denote N=1N=1 baselines, filled stars denote the module-best pipelines selected on BIRD and applied unchanged to Spider. The horizontal axis reports total cost on a logarithmic scale. Stack construction rule. We exclude Candidate Selection because its gains are bounded by candidate-pool diversity (§ 4.2) and do not justify sampling N>1N>1 candidates from a single generator. For each backbone and each remaining module, we select the nontrivial paradigm with the lowest CPPCPP whose EXEX gain is positive on that backbone and on a majority of backbones, retaining the trivial paradigm if none qualifies. This selects S2S_2 rather than the cheaper S1S_1 on GPT-4o-mini: although S1S_1 gains +0.13+0.13 p on that backbone, its gain is negative on the other three. The resulting stacks in Table 4 largely follow the marginal analysis: all use R1R_1, while DeepSeek-V4-Flash retains G1G_1 because neither nontrivial generation paradigm improves its EXEX. The only substantive departure from the marginal-EXEX ranking occurs in GPT-5.4’s generation module, where the rule selects G2G_2 for its lower CPPCPP despite G3G_3’s slightly higher EXEX overall. BIRD tradeoff. Table 4 summarizes the resulting stacks on BIRD. Stacking improves EXEX on every backbone, but by different margins: GPT-4o-mini, Gemini-2.5-Flash, and GPT-5.4 each gain more than 77 p, while DeepSeek-V4-Flash gains noticeably less, consistent with its weaker response to pipeline engineering observed in § 4. At moderate budgets, engineering a richer pipeline on a mid-tier backbone outperforms upgrading to a frontier backbone running lean: Gemini-2.5-Flash’s stack Pareto-dominates GPT-5.4’s lean baseline, achieving substantially higher EXEX at lower cost. This reframes choice as a backbone–pipeline tradeoff rather than a preference for the strongest model. At the high-budget end, GPT-5.4’s stack attains the highest EXEX, but its 2.292.29 p advantage over Gemini-2.5-Flash’s stack comes at more than 5×5× the total cost. Spider generalization. To test cross-benchmark transfer, we apply the BIRD-derived stacks unchanged to Spider. As shown in Figure 6, all four stacks improve EXEX over their respective lean baselines, with gains ranging from +3.68+3.68 to +6.77+6.77 p. The backbone–pipeline tradeoff persists on Spider: the DeepSeek-V4-Flash and Gemini-2.5-Flash stacks both Pareto-dominate GPT-5.4’s lean baseline. The GPT-4o-mini stack nearly matches that baseline (80.17%80.17\% vs. 80.27%80.27\%) at $0.315 rather than $1.879, while the DeepSeek-V4-Flash and Gemini-2.5-Flash stacks surpass it, reaching 82.01%82.01\% and 84.72%84.72\% at lower costs of $0.679 and $1.172, respectively. GPT-5.4’s stack achieves the highest EXEX of 87.04%87.04\%, but its 2.322.32 p advantage over Gemini-2.5-Flash’s stack requires 5.62×5.62× the total cost overall. 5.3. Cross-Module and -Model Conclusion Cross-module composition is backbone-dependent, and at moderate budgets the cost-efficient frontier runs through stacked mid-tier pipelines rather than lean configurations on frontier backbones. Marginal results remain useful for stack construction, but interactions shape how module gains combine. Applying the BIRD-derived stacks unchanged to Spider shows that the resulting backbone–pipeline tradeoff extends beyond BIRD. Table 4 reports a separate stack for each backbone; § 6 distills these results into a unified guideline and evaluates it on five diverse additional models. 6. Cost-Efficient Configuration Guideline The per-backbone stacks in § 5.2 are backbone-specific selections derived from per-paradigm evaluation and therefore cannot be assumed in advance when configuring a new system. The practical question is instead whether the recurring patterns across these stacks can be compressed into a small set of configurations that requires only coarse knowledge of the backbone, rather than a new paradigm search for every model. This section distills the preceding single-module and stacking results into such a prescriptive guideline, examines the empirical patterns behind its recommendations, and directly evaluates its transfer to five additional validation backbones spanning diverse capability levels and reasoning behaviors. 6.1. From Stacks to a Tiered Guideline We distill the single-module and stacking results in our study into three clear incremental tiers for practical use. Tier 1 is the bare backbone running a single-pass baseline. Tier 2 adds R1R_1 (Execution-Feedback Refinement), the only paradigm that improves all four primary backbones at consistently low cost. Tier 3 further adds E2E_2 (Masked Question Similarity) and S2S_2 (Pseudo-SQL Guided Linking), together with a backbone-conditional generation strategy: G2G_2 (CoT-Enhanced) for non-reasoning backbones and G1G_1 (trivial) for reasoning-capable ones, following the split identified in § 4.2. Table 5. Guideline validation on additional backbones. EXEX (%) and total API cost in USD are measured on BIRD; parenthesized values report ΔEX in percentage points over Tier 1. T1–T3 denote three progressively richer configurations, ranging from the lean baseline to the full cost-aware pipeline. EXEX (%) Cost Backbone T1 T2 T3 T1 T2 T3 Llama-3.1-8B-Instruct 37.61 40.22 (+2.61)(+2.61) 44.07 (+6.46)(+6.46) 0.05 0.06 0.09 Qwen3-32B 50.59 52.80 (+2.21)(+2.21) 55.48 (+4.89)(+4.89) 0.43 0.52 0.93 o3-mini 54.11 55.87 (+1.76)(+1.76) 59.97 (+5.87)(+5.87) 5.67 6.40 12.67 Qwen3-235B 57.82 59.06 (+1.24)(+1.24) 61.80 (+3.98)(+3.98) 4.55 5.76 11.36 Claude Sonnet 4.5 62.71 65.25 (+2.54)(+2.54) 70.34 (+7.63)(+7.63) 8.62 9.65 18.94 These choices are not obtained by simply taking the highest-accuracy paradigm in each module. They preserve the components that recur across the cost-efficient stacks in § 5.2. All four stacks contain E2E_2, S2S_2, and R1R_1, whereas Generation Strategy is the main component that changes with backbone reasoning behavior: the three non-reasoning stacks use G2G_2, while DeepSeek-V4-Flash retains G1G_1. This recurring structure in turn motivates a fixed tier definition with the generation choice conditioned on the backbone. The resulting guideline recommends a tier from the backbone profile and accuracy–cost preference, without requiring per-paradigm search for a new backbone. Figure 7 summarizes the resulting tier construction and backbone-conditioned selection rule. 6.2. Patterns Behind the Guideline Pipeline engineering is most cost-effective on small and mid-tier non-reasoning backbones. The Tier 1 to Tier 3 improvement is largest on the smallest backbone in our study, while a mid-tier non-reasoning backbone at Tier 3 costs a fraction of a frontier backbone at Tier 1 (§ 5.2). For example, Gemini-2.5-Flash at Tier 3 reaches higher EX than GPT-5.4 at Tier 1 while costing substantially less in our setting. The guideline therefore favors Tier 3 for these backbones, while using Tier 2 as the cost-efficient default for reasoning backbones and reserving Tier 3 on frontier backbones for accuracy-first settings; § 6.3 tests this pattern on five backbones. [Tier Construction] Bare-backbone baseline. Use the single-pass baseline with all modules at their trivial settings. ↪ T1: Bare backbone (N=1N=1). Universal refinement rule. R1R_1 is the only paradigm that improves all tested backbones at consistently low cost. ↪ T2: T1+R1T1+R_1. Backbone-conditioned stacking rule. E2+S2E_2+S_2 persist across backbone stacks, while generation follows the reasoning split: G∗=G2,non-reasoning backbone,G1,reasoning backbone.G^*= casesG_2,&non-reasoning backbone,\\ G_1,&reasoning backbone. cases ↪ T3: T2+E2+S2+G∗T2+E_2+S_2+G^*. [Candidate Selection] Cost-efficient setting. Candidate Selection faces a fixed-pool recall ceiling and N>1N>1 sampling overhead, so it is excluded from the cost-efficient tiers. Accuracy-first option. When cost is secondary, Candidate Selection may be added for further accuracy, subject to the candidate-pool recall ceiling. [Tier Selection] Reasoning T2 is the cost-efficient default; consider T3 when extra accuracy justifies the cost. Small / mid-tier non-reasoning Prefer T3. Frontier Use T3 when maximum EX is prioritized; otherwise consider a cheaper backbone with T3. Figure 7. Evidence-backed cost-efficient configuration guideline distilled from the single-module and stacking analyses. The three tiers progressively incorporate paradigms supported by robust accuracy–cost evidence, while Candidate Selection is treated separately due to its sampling overhead and fixed-pool recall ceiling. The recommended tier further depends on backbone capability and reasoning behavior. 6.3. Validation on Additional Backbones To test whether the guideline transfers beyond the four primary backbones from which its tier structure was derived, we directly apply the three prescribed tiers to five additional validation backbones spanning different capability levels and reasoning behaviors, without conducting any per-paradigm search or adapting the tier composition to individual models. All five validation backbones consistently improve from Tier 1 to both Tier 2 and Tier 3. As shown in Table 5, Tier 2 gains range from +1.24 to +2.61 p, while Tier 3 gains range from +3.98 to +7.63 p. The improvement is observed across capability levels and both non-reasoning and reasoning settings, indicating that the fixed tier compositions continue to transfer beyond the four primary backbones used for stack construction. The corresponding costs also follow the intended tier ordering, with Tier 2 adding limited overhead and Tier 3 trading a larger cost increase for higher EXEX across five validation backbones. 7. Related Work Empirical Analysis of Text-to-SQL. Several studies analyze specific components or configurations of Text-to-SQL systems. Li et al. (2024) introduce NL2SQL360, a multi-angle evaluation framework that compares NL2SQL methods across data domains, SQL characteristics, and evaluation settings, and further explores the selection of solutions for different application needs. At the component level, Maamari et al. (2024) revisit explicit schema linking and show that stronger LLMs can often operate directly over the full schema when it fits within the context window, avoiding errors caused by filtering necessary elements. In contrast, Yuan et al. (2025) improve schema linking by formulating the selection of relevant schema elements as a constrained knapsack optimization. For generation, Tai et al. (2023) also show that iterative least-to-most prompting can be unnecessary and that overly detailed reasoning may propagate errors. Recent systems increasingly combine multiple such mechanisms into elaborate inference-time pipelines. CHASE-SQL (Pourreza et al., 2025) combines multi-path candidate generation with preference-based pairwise selection; DeepEye-SQL (Li et al., 2026) organizes linking, diverse generation, verification, and candidate selection through a workflow; and Agentar-Scale-SQL (Wang et al., 2025b) orchestrates internal reasoning, iterative refinement, and parallel generation and selection for test-time scaling. These systems demonstrate the benefit of increasingly sophisticated pipeline orchestration, but still primarily optimize integrated system designs and evaluate components within their own pipelines. Our study complements these efforts by isolating each module’s marginal accuracy–cost contribution under one controlled implementation across multiple backbones; the schema-linking and CoT trends further extend prior observations to this cost-aware setting. Cost-Aware LLM Systems. Cost-aware LLM deployment has been studied at three complementary layers. At the model-selection layer, FrugalGPT (Chen et al., 2024), SATER (Shen et al., 2025), and RouteLLM (Ong et al., 2025) use routing or cascading to navigate tradeoffs between performance and cost. At the inference-resource layer, LLMLingua (Jiang et al., 2023) and Sketch-of-Thought (Aytes et al., 2025) reduce prompt or reasoning token usage, while recent work studies compute-optimal and budget-aware allocation of test-time reasoning resources (Snell et al., 2025; Lin et al., 2026). At the workflow layer, CATP LLM (Wu et al., 2025) explicitly balances task performance against tool-execution cost. Within Text-to-SQL, Dönder et al. (2025) propose N-rep, which reduces inference cost through multiple schema representations without CoT or self-consistency. These efforts optimize model choice, inference resources, or workflow execution; we instead directly ask which modules of the Text-to-SQL pipeline justify their added cost. 8. Conclusion We studied the ICL Text-to-SQL pipeline as a modular object in its own right, isolating the marginal contribution and cost of each design paradigm under a unified controlled framework across four backbones. Execution-feedback refinement is the only paradigm that improves all four primary backbones at consistently low cost, making it a safe first addition; most others pay off only under backbone-dependent conditions. Token accounting suggests that input demand is closely tied to pipeline structure, whereas output demand is sensitive to backbone generation behavior. Building on this analysis, we found that a fixed budget is often better spent engineering a richer pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. The BIRD-derived stacks also transfer to Spider, where all four improve accuracy and the backbone–pipeline tradeoff persists. We distill these observations into a backbone-dependent, tiered guideline that transfers to five additional validation backbones, offering practitioners a more practical, cost-aware starting point for configuring ICL Text-to-SQL systems than aggregate accuracy alone in practical settings. 9. Ethical Considerations This work evaluates text-to-SQL methods on the publicly available BIRD benchmark (Li et al., 2023). Nevertheless, deployed text-to-SQL systems, especially in real world database applications, can expose sensitive database contents or execute harmful queries when model outputs are incorrect or maliciously prompted. Practical deployments should therefore enforce least-privilege access, use read-only execution whenever possible, validate generated SQL, sandbox query execution, and maintain proper human oversight and audit logs. References Aytes et al. (2025) S. A. Aytes, J. Baek, and S. J. Hwang Sketch-of-thought: efficient LLM reasoning with adaptive cognitive-inspired sketching. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §7. Cao et al. (2024) Z. Cao, Y. Zheng, Z. Fan, X. Zhang, W. Chen, and X. Bai RSL-sql: robust schema linking in text-to-sql generation. arXiv preprint arXiv:2411.00073. Cited by: §2. Chen et al. (2024) L. Chen, M. Zaharia, and J. Zou FrugalGPT: how to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research (TMLR). Cited by: §7. Deng et al. (2025) M. Deng, A. Ramachandran, C. Xu, L. Hu, Z. Yao, A. Datta, and H. Zhang ReFoRCE: a text-to-SQL agent with self-refinement, format restriction, and column exploration. In ICLR 2025 Workshop: VerifAI: AI Verification in the Wild, Cited by: §2. Dönder et al. (2025) Y. D. Dönder, D. Hommel, A. W. Wen-Yi, D. Mimno, and U. E. S. Jo Cheaper, better, faster, stronger: robust text-to-SQL without chain-of-thought or fine-tuning. arXiv preprint arXiv:2505.14174. Cited by: §7. Dong et al. (2023) X. Dong, C. Zhang, Y. Ge, Y. Mao, Y. Gao, L. Chen, J. Lin, and D. Lou C3: zero-shot text-to-sql with chatgpt. arXiv:2307.07306. Cited by: §2. Dubey et al. (2024) A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Gao et al. (2024a) D. Gao, H. Wang, Y. Li, X. Sun, Y. Qian, B. Ding, and J. Zhou Text-to-sql empowered by large language models: a benchmark evaluation. In International Conference on Very Large Data Bases (VLDB), Cited by: §2. Gao et al. (2024b) Y. Gao, Y. Liu, X. Li, X. Shi, Y. Zhu, Y. Wang, S. Li, W. Li, Y. Hong, Z. Luo, J. Gao, L. Mou, and Y. Li Xiyan-sql: a multi-generator ensemble framework for text-to-sql. arXiv preprint arXiv:2411.08599. Cited by: §2. Hong et al. (2026a) Z. Hong, H. Chen, Z. Yuan, Q. Zhang, L. Zhuang, Q. Liao, F. Huang, Y. Song, and X. Huang ErrorLLM: modeling sql errors for text-to-sql refinement. In Conference on Knowledge Discovery and Data Mining (KDD), Cited by: §2. Hong et al. (2026b) Z. Hong, H. Wu, S. Dong, J. Dong, Y. Xiao, Y. Zhang, Z. Wang, F. Huang, L. Li, H. Yang, and X. Huang Benchmarking llms’ mathematical reasoning with unseen random variables questions. In Conference on Artificial Intelligence (AAAI), Hong et al. (2024) Z. Hong, Z. Yuan, H. Chen, Q. Zhang, F. Huang, and X. Huang Knowledge-to-SQL: enhancing SQL generation with data expert LLM. In Findings of Association for Computational Linguistics (ACL), Hong et al. (2025) Z. Hong, Z. Yuan, Q. Zhang, H. Chen, J. Dong, F. Huang, and X. Huang Next-generation database interfaces: a survey of LLM-based text-to-SQL. IEEE Transactions on Knowledge and Data Engineering (TKDE). Cited by: §1, §3.3. Jiang et al. (2023) H. Jiang, Q. Wu, C. Lin, Y. Yang, and L. Qiu LLMLingua: compressing prompts for accelerated inference of large language models. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §7. Lee et al. (2025) D. Lee, C. Park, J. Kim, and H. Park MCS-SQL: leveraging multiple prompts and multiple-choice selection for text-to-SQL generation. In International Conference on Computational Linguistics (COLING), Cited by: §1. Li et al. (2026) B. Li, C. Chen, Z. Xue, Y. Mei, and Y. Luo DeepEye-sql: a software-engineering-inspired text-to-sql framework. In Conference on Management of Data (SIGMOD), Cited by: §2, §7. Li et al. (2024) B. Li, Y. Luo, C. Chai, G. Li, and N. Tang The dawn of natural language to SQL: are we fully ready?. In International Conference on Very Large Data Bases (VLDB), Cited by: §1, §2, §7. Li et al. (2025a) B. Li, J. Zhang, J. Fan, Y. Xu, C. Chen, N. Tang, and Y. Luo Alpha-SQL: zero-shot text-to-SQL using monte carlo tree search. In International Conference on Machine Learning (ICML), Cited by: §1. Li et al. (2023) J. Li, B. Hui, G. Qu, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, X. Zhou, C. Ma, G. Li, K. Chang, F. Huang, R. Cheng, and Y. Li Can LLM already serve as a database interface? a Big bench for large-scale database grounded text-to-SQLs. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1, §3.1, §9. Li et al. (2025b) Z. Li, X. Wang, J. Zhao, S. Yang, G. Du, X. Hu, B. Zhang, Y. Ye, Z. Li, R. Zhao, and H. Mao Pet-sql: a prompt-enhanced two-round refinement of text-to-sql with cross-consistency. In International Conference on Database Systems for Advanced Applications (DASFAA), Cited by: §2. Lin et al. (2026) J. Lin, X. Zeng, J. Zhu, S. Wang, J. Shun, J. Wu, and D. Zhou Plan and budget: effective and efficient test-time scaling on reasoning large language models. In International Conference on Learning Representations (ICLR), Cited by: §7. Maamari et al. (2024) K. Maamari, F. Abubaker, D. Jaroslawicz, and A. Mhedhbi The death of schema linking? text-to-SQL in the age of well-reasoned language models. In NeurIPS 2024 Third Table Representation Learning Workshop (NeurIPS), Cited by: §1, §4.2, §7. Ong et al. (2025) I. Ong, A. Almahairi, V. Wu, W. Chiang, T. Wu, J. E. Gonzalez, M. Kadous, and I. Stoica RouteLLM: learning to route LLMs from preference data. In International Conference on Learning Representations (ICLR), Cited by: §7. Pourreza et al. (2025) M. Pourreza, H. Li, R. Sun, Y. Chung, S. Talaei, G. T. Kakkar, Y. Gan, A. Saberi, F. Ozcan, and S. O. Arik Chase-sql: multi-path reasoning and preference optimized candidate selection in text-to-sql. In International Conference on Learning Representations (ICLR), Cited by: §1, §2, §7. Pourreza and Rafiei (2023) M. Pourreza and D. Rafiei Din-sql: decomposed in-context learning of text-to-sql with self-correction. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Qu et al. (2025) G. Qu, J. Li, B. Qin, X. Li, N. Huo, C. Ma, and R. Cheng SHARE: an SLM-based hierarchical action CorREction assistant for text-to-SQL. In Association for Computational Linguistics (ACL), Cited by: §2. Ren et al. (2024) T. Ren, Y. Fan, Z. He, R. Huang, J. Dai, C. Huang, Y. Jing, K. Zhang, Y. Yang, and X. S. Wang PURPLE: making a large language model a better sql writer. In International Conference on Data Engineering (ICDE), Cited by: §2. Shen et al. (2025) Y. Shen, Y. Liu, Z. Huang, R. Yin, X. Zheng, and X. Huang SATER: a self-aware and token-efficient approach to routing and cascading. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §7. Shi et al. (2025) J. Shi, B. Xu, J. Liang, Y. Xiao, J. Chen, C. Xie, P. Wang, and W. Wang Gen-SQL: efficient text-to-SQL by bridging natural language question and database schema with pseudo-schema. In International Conference on Computational Linguistics (COLING), Cited by: §1, §2. Snell et al. (2025) C. Snell, J. Lee, K. Xu, and A. Kumar Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In International Conference on Learning Representations (ICLR), Cited by: §7. Tai et al. (2023) C. Tai, Z. Chen, T. ZHANG, X. Deng, and H. Sun Exploring chain of thought style prompting for text-to-SQL. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §7. Talaei et al. (2024) S. Talaei, M. Pourreza, Y. Chang, A. Mirhoseini, and A. Saberi CHESS: contextual harnessing for efficient sql synthesis. arXiv:2405.16755. Wang et al. (2025a) B. Wang, C. Ren, J. Yang, X. Liang, J. Bai, L. Chai, Z. Yan, Q. Zhang, D. Yin, X. Sun, and Z. Li MAC-sql: a multi-agent collaborative framework for text-to-sql.. In International Conference on Computational Linguistics (COLING), Cited by: §1, §2. Wang et al. (2025b) P. Wang, B. Sun, X. Dong, Y. Dai, H. Yuan, M. Chu, Y. Gao, X. Qi, P. Zhang, and Y. Yan Agentar-Scale-SQL: advancing text-to-SQL through orchestrated test-time scaling. arXiv preprint arXiv:2509.24403. Cited by: §1, §2, §7. Wang et al. (2025c) Y. Wang, P. Liu, and X. Yang LinkAlign: scalable schema linking for real-world large-scale multi-database text-to-SQL. In Empirical Methods in Natural Language Processing (EMNLP), Wang et al. (2026) Z. Wang, Y. Zheng, Z. Cao, X. Zhang, Z. Wei, P. Fu, Z. Luo, W. Chen, and X. Bai AutoLink: autonomous schema exploration and expansion for scalable schema linking in text-to-sql at scale. In Conference on Artificial Intelligence (AAAI), Wu et al. (2025) D. Wu, J. Wang, Y. Meng, Y. Zhang, L. Sun, and Z. Wang CATP-LLM: empowering large language models for cost-aware tool planning. In International Conference on Computer Vision (ICCV), Cited by: §7. Xie et al. (2025) X. Xie, G. Xu, L. Zhao, and R. Guo OpenSearch-sql: enhancing text-to-sql with dynamic few-shot and consistency alignment. In Conference on Management of Data (SIGMOD), Cited by: §2. Xie et al. (2024) Y. Xie, X. Jin, T. Xie, M. Lin, L. Chen, C. Yu, L. Cheng, C. Zhuo, B. Hu, and Z. Li Decomposition for enhancing attention: improving llm-based text-to-sql through workflow paradigm. In Findings of Association for Computational Linguistics (ACL), Cited by: §2. Yao et al. (2026) H. Yao, Z. Hong, H. Chen, Z. Li, Q. Shen, Z. Ying, Q. Feng, H. Gong, and F. Huang Macro graph of experts for billion-scale multi-task recommendation. In Conference on Knowledge Discovery and Data Mining (KDD), Yu et al. (2018) T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Empirical Methods in Natural Language Processing (EMNLP), Cited by: §1, §3.1, §3.2. Yuan et al. (2025) Z. Yuan, H. Chen, Z. Hong, Q. Zhang, F. Huang, Q. Li, and X. Huang Knapsack optimization-based schema linking for llm-based text-to-sql generation. In International Conference on Data Engineering (ICDE), Cited by: §4.2, §7. Table 6. Per-backbone EX (%) on BIRD dev (1,534 examples). ΔEX [95% CI] is the percentage-point gain over the matched-budget baseline (N=5N=5 for Candidate Selection; N=1N=1 otherwise), and CPP is incremental USD/p. “–” denotes undefined CPP, “∗” Pareto domination, and the / keys mark the best/second best results. ID GPT-4o-mini DeepSeek-V4-Flash Gemini-2.5-Flash GPT-5.4 ↑EX ↑ EX [95% CI] ↓CPP ↑EX ↑ EX [95% CI] ↓CPP ↑EX ↑ EX [95% CI] ↓CPP ↑EX ↑ EX [95% CI] ↓CPP Example Retrieval E1 50.07 +0.33 [-0.01, +0.67] 0.207 56.06 -0.26 [-0.57, +0.05] – 60.43 +0.59 [+0.06, +1.12] 0.401 62.58 +1.30 [+0.63, +1.97] 0.995 E2 50.65 +0.91 [+0.26, +1.56] 0.144 56.84 +0.52 [+0.01, +1.03] 0.326 61.28 +1.43 [+0.74, +2.13] 0.417 63.69 +2.41 [+1.42, +3.40] 0.957 E3 49.80 +0.07 [-0.16, +0.29] 7.259 57.11 +0.78 [+0.06, +1.50] 0.660 60.10 +0.26 [-0.18, +0.70] 5.317 63.49 +2.22 [+1.28, +3.15] 3.417 Schema Linking S1 49.87 +0.13 [-0.13, +0.39] – 55.93 -0.39 [-0.75, -0.03] – 59.13 -0.72 [-1.27, -0.16] – 60.63 -0.65 [-1.19, -0.11] – S2 52.15 +2.41 [+1.49, +3.33] 0.075 57.04 +0.72 [+0.08, +1.36] 0.491 61.15 +1.30 [+0.63, +1.98] 0.575 62.39 +1.11 [+0.45, +1.77] 3.293 S3a 51.11 +1.37 [+0.69, +2.05] 0.109 56.19 -0.13 [-0.39, +0.13] – 60.89 +1.04 [+0.42, +1.67] 0.599 60.82 -0.46 [-0.95, +0.03] – S3b 51.83 +2.09 [+1.23, +2.95] 0.090 55.87 -0.46 [-0.88, -0.04] – 60.76 +0.91 [+0.32, +1.51] 0.847 59.97 -1.30 [-1.98, -0.63] – S4 52.93 +3.19 [+2.11, +4.27] 0.271 56.52 +0.20 [-0.09, +0.49] 4.262 61.34 +1.50 [+0.79, +2.21] 1.224 62.06 +0.78 [+0.21, +1.35] 20.870 SQL Generation G1 49.74 0.00 – 56.32 0.00 – 59.84 0.00 – 61.28 0.00 – G2 53.98 +4.24 [+2.96, +5.52] 0.026 55.93 -0.39 [-0.75, -0.03] – 62.71 +2.87 [+1.80, +3.94] 0.170 62.84 +1.56 [+0.80, +2.32] 1.723 G3 51.24 +1.50 [+0.77, +2.23] 0.093 56.13 -0.20 [-0.49, +0.09] – 60.76 +0.91 [+0.32, +1.51] 0.976 63.10 +1.83 [+1.00, +2.65] 2.285 Candidate Selection C1 51.11 +1.43 [+0.74, +2.13] ∗ 57.30 +0.85 [+0.19, +1.51] ∗ 60.89 +0.85 [+0.27, +1.43] ∗ 62.71 +1.30 [+0.63, +1.97] ∗ C2a 50.52 +0.85 [+0.36, +1.34] 0.771 59.00 +2.54 [+1.47, +3.61] 0.333 61.60 +1.56 [+0.80, +2.32] 1.806 63.82 +2.41 [+1.42, +3.40] 7.249 C2b 51.17 +1.50 [+0.79, +2.21] 1.389 58.54 +2.09 [+1.14, +3.04] 0.953 61.93 +1.89 [+1.04, +2.74] 2.858 63.89 +2.48 [+1.48, +3.48] 15.975 Refinement R1 54.63 +4.89 [+3.53, +6.25] 0.016 58.15 +1.83 [+0.99, +2.67] 0.040 62.32 +2.48 [+1.48, +3.48] 0.026 63.30 +2.02 [+1.15, +2.89] 0.297 R2 51.83 +2.09 [+1.23, +2.95] 0.305 56.58 +0.26 [-0.05, +0.57] 2.708 60.95 +1.11 [+0.45, +1.77] 1.964 62.06 +0.78 [+0.21, +1.35] 18.680 R3 53.26 +3.52 [+2.38, +4.66] 0.276 59.97 +3.65 [+2.47, +4.83] 0.365 63.04 +3.19 [+2.08, +4.30] 1.016 64.02 +2.74 [+1.68, +3.80] 7.769 Appendix A Full Taxonomy Mapping Table 7 maps 24 ICL Text-to-SQL methods to paradigms across five pipeline modules. A checkmark indicates documented use of a paradigm, while a dash indicates that such use is not established in the original paper. For Direct LLM Linking and LLM-Based Preference Selection, sub-paradigm distinctions (S3aS_3a vs. S3bS_3b and C2aC_2a vs. C2bC_2b) are recorded within cells rather than assigned separate columns. Multi-paradigm cells. A method may instantiate multiple paradigms within one module. This occurs mainly in Generation Strategy, where systems combine direct, CoT-enhanced, and decomposition-based generators, and in Schema Linking, where retrieval-based and LLM-based signals are combined before SQL generation. Classification protocol. We classify each method according to the primary system reported in the original paper, excluding ablation variants. When the paradigm is not stated explicitly, we infer it from the provided prompts or pseudocode; if neither provides sufficient evidence, the cell is marked as – in the final taxonomy table. Table 7. Taxonomy of representative Text-to-SQL methods. Method Venue Example Retrieval Schema Linking Generation Strategy Candidate Selection Refinement Question Similarity Masked Question Similarity SQL Skeleton Similarity Retrieval- Based Linking Pseudo-SQL Guided Linking Direct LLM Linking Agentic Linking Direct Generation CoT-Enhanced Generation Decomposition- Based Generation Execution- Based Voting LLM-Based Preference Selection Execution- Feedback Refinement Self- Correction Agentic Refinement AutoLink AAAI’26 – – – – – – ✓ – ✓ ✓ ✓ Pairwise Comparison ✓ – – DeepEye-SQL SIGMOD’26 – ✓ – ✓ ✓ Single-shot Direct Linking – ✓ ✓ ✓ ✓ Pairwise Comparison – – ✓ MAGIC AAAI’25 – – – – – – – – – – – – – ✓ – Agentar-Scale-SQL arXiv’25 – ✓ – – – – – ✓ ✓ ✓ – Pairwise Comparison ✓ ✓ – N-rep arXiv’25 – ✓ – – – Single-shot Direct Linking – ✓ – – ✓ Pairwise Comparison – – – XiYan-SQL arXiv’25 – ✓ – ✓ – Single-shot Direct Linking – ✓ – – – Scoring and Ranking ✓ – – Gen-SQL COLING’25 ✓ – – ✓ ✓ – – ✓ – – – – ✓ – – MCS-SQL COLING’25 ✓ ✓ – – – Two-stage Direct Linking – – ✓ – – Scoring and Ranking – – – MAC-SQL COLING’25 – – – – – – ✓ – ✓ ✓ – – – – ✓ PET-SQL DASFAA’25 – ✓ – – ✓ – – ✓ – – ✓ – – – – LinkAlign EMNLP’25 – – – – – – ✓ – – – – – – – – Alpha-SQL ICML’25 – – – ✓ – Single-shot Direct Linking – – ✓ ✓ ✓ – ✓ – – CHESS-SQL ICML-W’25 – – – – – Two-stage Direct Linking – – ✓ – – Scoring and Ranking ✓ – – CHASE-SQL ICLR’25 – – – – – Two-stage Direct Linking – – ✓ ✓ ✓ Pairwise Comparison – – ✓ ReFoRCE ICLR-W’25 – – – – – Single-shot Direct Linking – ✓ – – ✓ – – – ✓ OpenSearch-SQL SIGMOD’25 – ✓ – ✓ – Single-shot Direct Linking – – ✓ – ✓ – ✓ – – RSL-SQL arXiv’24 ✓ – – – ✓ Single-shot Direct Linking – ✓ – ✓ – Pairwise Comparison ✓ – – DEA-SQL ACL-F’24 ✓ ✓ – – – Two-stage Direct Linking – ✓ – – – – – ✓ – PURPLE ICDE’24 – – ✓ ✓ – – – ✓ – – ✓ – – – – SuperSQL VLDB’24 ✓ – ✓ ✓ – – – ✓ – – ✓ – – – – DAIL-SQL VLDB’24 ✓ ✓ – – – – – ✓ – – ✓ – – – – C3-SQL arXiv’23 – – – – – Two-stage Direct Linking – ✓ – – ✓ – – – – ACT-SQL EMNLP-F’23 ✓ – – – – – – – ✓ – – – – – – DIN-SQL NeurIPS’23 – – – – – Single-shot Direct Linking – – ✓ ✓ – – – ✓ – Appendix B Paradigm Implementation Details This section details the controlled implementations of the 17 paradigm-level configurations evaluated in the main paper. We organize them into pre-processing (Example Retrieval and Schema Linking; Figure 8), SQL generation (Generation Strategy; Figure 9), and post-processing (Candidate Selection and Refinement; Figure 10). Box layout. Each module card specifies its shared inputs, backbone, and fallback behavior; maps paradigms to mechanisms and per-query LLM calls; notes auxiliary calls or behavior and their cost treatment; and identifies the trivial baseline used in the main marginal analysis for controlled comparison across paradigms.. Example Retrieval. Shared setup. All three paradigms retrieve from the BIRD training split using a frozen bge-large-en-v1.5 encoder, with ℓ2 _2-normalized embeddings ranked by cosine similarity. Examples drawn from the target database are excluded to prevent leakage, and exact-duplicate questions are removed. The generator receives up to k=3k=3 demonstrations, each rendered as the original question, its optional evidence annotation, and the gold SQL; no demonstration schema is included. Embeddings are precomputed once and cached. Paradigm Retrieval signal Shots E1E_1 Question Similarity Raw question embedding. 3 E2 Masked Question Similarity Question with schema-specific entities masked by the backbone before encoding; retrieval returns the original (unmasked) examples. 3 E3 SQL Skeleton Similarity Structural skeleton of a preliminary SQL draft, matched against skeletons of training gold SQL. 3 Auxiliary LLM calls. E2 issues one backbone call per query to produce the masked question; E3 issues one call to draft the preliminary SQL from which the skeleton is extracted. E1E_1 requires no auxiliary call. All auxiliary calls use the same backbone as the main generator, at temperature 0; on an empty or malformed response, the paradigm falls back to the unmasked question (E2) or to direct generation (E3). Skeleton extraction abstracts literals and non-keyword identifiers to placeholder tokens while preserving SQL keywords and operators. Trivial baseline. Zero-shot generation with no retrieved demonstrations. Schema Linking. Shared setup. Each paradigm receives the question, the full schema with column descriptions, and the optional evidence annotation, and returns a filtered schema for the generator; primary and foreign keys are always retained after filtering. Paradigms differ in who makes the linking decision. Malformed or empty responses fall back to the full schema, so no paradigm gains accuracy from its failure path. All auxiliary calls use the main backbone, so their cost is attributed to that backbone. Paradigm Mechanism Calls S1S_1 Retrieval-Based Embeds question keywords against per-column text, boosts columns whose cells match a keyword, and keeps the top-scoring tables and columns. 1 S2 Pseudo-SQL Guided Drafts a SQL query over the full schema, then keeps the referenced tables and columns. 1 S3a Direct LLM, Single-shot Asks the backbone once to select the needed tables and columns, then validates the names against the schema. 1 S3b Direct LLM, Two-stage Selects relevant tables first, then selects columns only within those tables. 2 S4 Agentic A proposer selects a schema slice, a verifier judges its sufficiency, and an optional second proposer revises it. 1–3 Auxiliary LLM calls. All linking calls run on the main backbone at temperature 0. S1S_1 uses a frozen bge-large-en-v1.5 encoder for its keyword–column matching. S4 issues a variable number of calls: a single proposer call when its proposal is empty or unparsable, two calls when the verifier accepts the slice, and a third only when the verifier reports a concrete deficiency. Trivial baseline. The full schema is passed to the generator unfiltered. Figure 8. Controlled implementation of pre-processing modules: Example Retrieval and Schema Linking. Generation Strategy. Shared setup. All three paradigms receive the question, the schema (linked or full), and the optional evidence, and run on the main backbone at temperature 0 with a single sampled candidate—one generation call per query. Each parses the final SQL string from its own output and passes only that string downstream; when no SQL can be parsed, the query counts as incorrect. Paradigm Mechanism Calls G1G_1 Direct The backbone writes a single SQL query directly, returned in a fenced code block. 1 G2 CoT-Enhanced The backbone emits a step-by-step reasoning trace and the final SQL in one structured response. 1 G3 Decomposition-Based The backbone decomposes the question into sub-questions and sub-queries and returns an assembled final SQL, all in one structured response. 1 Generation behavior. All three paradigms use a single backbone call; they differ only in what that call is asked to emit. In G2, reasoning and SQL are produced together and is therefore charged as additional output tokens rather than obtained from a separate planning call. In G3, the number of sub-questions is chosen by the model, and the final SQL is read directly from its output rather than assembled by executing sub-queries; G3 likewise adds no extra call. SQL is parsed from a structured field where available (G2, G3) or from the fenced block (G1G_1), with a shared fallback that extracts a bare SELECT/WITH statement. Trivial baseline. Direct generation (G1). Figure 9. Controlled implementation of the generation stage: Generation Strategy. Candidate Selection. Shared setup. All selectors operate over the same pool of N=5N=5 candidates, sampled from the direct-generation prompt on the main backbone at temperature 0.8—the same pool used by the N=5N=5 reference baseline. Each selector returns one of the five candidates and never generates a new query. Paradigm Mechanism LLM calls C1C_1 Execution-Based Voting Executes all five candidates, groups them by result equivalence, and takes the earliest candidate in the largest group. 0 C2a Scoring and Ranking Shows the model all five candidates with their execution outcomes and asks it to pick the best. 1 C2b Pairwise Comparison Runs a single-elimination tournament, comparing the running winner against each remaining candidate. N−1N-1 Selection behavior. All selectors first execute the five candidates and run on the main backbone at temperature 0. They differ in cost: C1 issues no LLM call (execution only), C2a a single call independent of N, and C2b N−1N-1 calls (four at N=5N=5), the dominant cost driver among the selectors. Trivial baseline. Return the first candidate from the same N=5N=5 candidate pool. Refinement. Shared setup. Each paradigm receives the selected SQL, the question, the schema, and the optional evidence, and returns a final SQL that replaces the original for evaluation; if refinement yields no valid SQL, the original (or most recent valid revision) is kept. All refinement calls run on the main backbone at temperature 0. Paradigm Mechanism LLM calls R1R_1 Execution-Feedback Executes the SQL first, then asks the model to rewrite it only when execution errors or returns no rows. 0–1 R2 Self-Correction Asks the model to review the SQL against a generic error checklist, without executing it. 1 R3 Agentic A corrector revises the SQL using execution feedback, a verifier reviews the result, and the corrector may revise once more. 1–3 Refinement behavior. The paradigms differ in when they fire, which is what separates their cost. R1 is failure gated: it runs one diagnostic execution and issues a model call only on an error or empty result, so its average call count per query is below one—the reason it adds accuracy at near-zero cost. R2 and R3 instead fire on every query: R2 issues exactly one review call with no execution feedback, while R3 issues two to three calls across its corrector and verifier roles. Each paradigm performs at most one revision round (R3 at most one additional corrector pass). Trivial baseline. The selected SQL is returned unchanged, with no refinement call. Figure 10. Controlled implementation of post-processing modules: Candidate Selection and Refinement. Appendix C API Pricing All USD costs reported in the main paper and this supplement are calculated using the API prices in Table 8, recorded on May 2, 2026. For each benchmark, the total configuration cost is the sum of input- and output-token charges over its full development split. Table 8. API prices in USD per million tokens, recorded on May 2, 2026. For reasoning backbones, output charges include all billed reasoning tokens used in evaluation. Model Input Output GPT-4o-mini 0.15 0.60 Gemini-2.5-Flash 0.30 2.50 DeepSeek-V4-Flash 0.14 0.28 GPT-5.4 2.50 15.00 Llama-3.1-8B-Instruct 0.02 0.04 Qwen3-32B 0.08 0.28 o3-mini 1.10 4.40 Qwen3-235B-A22B (Thinking) 0.23 2.30 Claude Sonnet 4.5 3.00 15.00 Appendix D Detailed Results with Confidence Intervals This section supplements the module-level results in Table 2 of the main paper. For every paradigm–backbone pair, Table 6 reports execution accuracy, its change from the matched-budget reference baseline with a corresponding 95% confidence interval, and cost per EXEX point (CPPCPP). The main-paper table provides the corresponding API costs and mean per-question input/output-token counts. Confidence intervals. For each paradigm–backbone pair, we compare per-example correctness with a matched-budget reference over the 1,534 BIRD development examples. Two-sided 95% confidence intervals for ΔEX use the normal approximation to the paired difference based on discordant outcome counts, while significance is assessed by a two-sided McNemar’s test at α=0.05α=0.05. Candidate Selection uses an N=5N=5 reference, whereas all other paradigms use N=1N=1, isolating the paradigm effect from additional candidate sampling. We report the intervals directly to preserve effect direction and magnitude rather than only binary significance decisions. *