Paper deep dive
Visible Reasoning Is Not a Universal Optimizer: Persona- and Thinking-Dependent Effects in Analytics Code Generation
Bhawani Shankar Leelar, Pawan Chorasiya, Davin Hill, Robert E. Tillman, Tamer Soliman
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/9/2026, 7:46:10 AM
Summary
This study evaluates the effectiveness of Visible Chain-of-Thought (CoT) in analytics code generation (SQL and Python/pandas), challenging the assumption that visible reasoning is a universal optimizer. Using a query-matched benchmark from BIRD-Persona, the authors test various persona phrasings, target languages, CoT formats, and internal-reasoning configurations. Results indicate that visible CoT does not provide a universal accuracy advantage, nor does matching the reasoning representation to the target language consistently improve performance. Instead, outcomes depend heavily on the specific model, persona, and internal-reasoning settings. The study highlights the importance of distinguishing between reasoning content and prompt format effects, suggesting that reasoning strategies should be selected contextually rather than adopted as defaults.
Entities (11)
Relation Signals (11)
Visible Chain-of-Thought → evaluatedon → Analytics Code Generation
confidence 95% · Visible Chain-of-Thought (CoT) is often treated as a broadly useful reasoning instruction, yet analytics code generation combines...
Gemini Flash → testedinstudy → Visible Chain-of-Thought
confidence 95% · Table 1: Six completed configurations used in the experiments... Gemini Flash-0... Gemini Flash-24k
Gemma 4 → testedinstudy → Visible Chain-of-Thought
confidence 95% · Table 1: Six completed configurations used in the experiments... Gemma4-off... Gemma4-on
GLM-4 → testedinstudy → Visible Chain-of-Thought
confidence 95% · Table 1: Six completed configurations used in the experiments... GLM4
Qwen3 → testedinstudy → Visible Chain-of-Thought
confidence 95% · Table 1: Six completed configurations used in the experiments... Qwen3
Visible Chain-of-Thought → testedwithtargetlanguages → SQL
confidence 95% · Because the same analytics request can be expressed in two distinct target languages-SQL and Python (pandas)
Visible Chain-of-Thought → testedwithtargetlanguages → Python
confidence 95% · Because the same analytics request can be expressed in two distinct target languages-SQL and Python (pandas)
Study → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Visible Chain-of-Thought (CoT) is often treated as a broadly useful reasoning instruction, yet analytics code generation combines natural-language ambiguity, schema grounding, target-language constraints, and model-specific inference behavior. Because the same analytics request can be expressed in two distinct target languages-SQL and Python (pandas)-this setting provides a natural test of a common but under-examined assumption: that visible reasoning is more effective when its representation matches the requested target, as in "think in SQL" or "think in Python." Together with generic instructions such as "think step-by-step," such recommendations remain insufficiently evaluated under controlled, execution-based comparisons. We study a query matched SQL-pandas benchmark that crosses persona phrasing, target language, visible-CoT format, control prefixes, direct generation, and internal-reasoning configurations. The results do not support either a universal accuracy advantage from visible CoT or a consistent benefit from matching the reasoning representation to the target language. Instead, the effects depend on the persona, target, model configuration, and internal-reasoning setting. The control ablations further distinguish effects of reasoning content from those of prompt format. These findings indicate that reasoning strategies should be selected jointly for the model, persona, target, and internal-reasoning configuration rather than adopted as universal defaults. More broadly, the study provides a controlled framework for identifying when visible reasoning improves executable generation, when it primarily perturbs model behavior, and when the internal-reasoning configuration is the more consequential factor.
Tags
Links
- Source: https://arxiv.org/abs/2610.10639v1
- Canonical: https://arxiv.org/abs/2610.10639v1
Trouble viewing inline? Open PDF directly →
Full Text
94,083 characters extracted from source content.
Expand or collapse full text
Visible Reasoning Is Not a Universal Optimizer: Persona- and Thinking-Dependent Effects in Analytics Code Generation Bhawani Shankar LeelarPawan ChorasiyaDavin Hill Optum AIOptum AIOptum AI bhawani.leelar@optum.com pawan chorasiya@optum.com davinhill@optum.com Robert E. TillmanTamer Soliman Optum AIOptum AI rob.tillman@optum.com tamer.soliman@optum.com Abstract Visible Chain-of-Thought (CoT) is often treated as a broadly useful reasoning instruction, yet analyt- ics code generation combines natural-language ambigu- ity, schema grounding, target-language constraints, and model-specific inference behavior. Because the same an- alytics request can be expressed in two distinct target languages-SQL and Python (pandas)-this setting pro- vides a natural test of a common but under-examined as- sumption: that visible reasoning is more effective when its representation matches the requested target, as in “think in SQL” or “think in Python.” Together with generic instructions such as “think step-by-step,” such recom- mendations remain insufficiently evaluated under con- trolled, execution-based comparisons. We study a query- matched SQL-pandas benchmark that crosses persona phrasing, target language, visible-CoT format, control prefixes, direct generation, and internal-reasoning config- urations. The results do not support either a universal accuracy advantage from visible CoT or a consistent ben- efit from matching the reasoning representation to the target language. Instead, the effects depend on the per- sona, target, model configuration, and internal-reasoning setting. The control ablations further distinguish effects of reasoning content from those of prompt format. These findings indicate that reasoning strategies should be se- lected jointly for the model, persona, target, and internal- reasoning configuration rather than adopted as universal defaults. More broadly, the study provides a controlled framework for identifying when visible reasoning improves executable generation, when it primarily perturbs model behavior, and when the internal-reasoning configuration is the more consequential factor. 1 Introduction Modern language models increasingly translate natural- language data requests into executable analytics code. The same request can be realized in two different paradigms: Structured Query Language (SQL), which specifies relational operations declaratively, and Python (pandas), which expresses the result through a sequence of DataFrame transformations. Visible CoT is now a com- mon prompting device for both, whether as a general instruction to reason step-by-step (Wei et al., 2022) or as a target-matched instruction such as think in SQL or think in Python. If the most useful visible-CoT rep- resentation is the one that matches the requested code, it should preferentially benefit procedural pandas gener- ation. A query-matched SQL-pandas benchmark makes this proposition testable: the database task, schema, ques- tion intent, and executable reference result can be fixed while the requested target and visible-CoT representation vary. Several alternative mechanisms complicate this target- matching hypothesis. A visible CoT may alter generated code without supplying a better task solution: code-like notation can prime target vocabulary, a fixed-step out- put contract can alter decoding behavior, and any addi- tional prefix changes the context presented to the model. A fluent rationale is not necessarily a faithful account of the computation leading to the final answer (Turpin et al., 2023; Lanham et al., 2023); additional inference- time computation is also not uniformly useful across tasks (Snell et al., 2025). We therefore distinguish two chan- nels. In this paper, visible CoT (or visible reasoning ) de- notes text in the externally observable prompt-response exchange: a condition-specific CoT instruction or prefix and the decoded thinking list returned with the final code. The explicit-CoT conditions request eight labeled entries; direct generation and the controls have differ- ent contracts described in Section 3. This visible text is not taken to reveal latent computation. Internal reason- ing denotes unobserved model- or service-level inference- time computation. The provider-exposed switches studied here are therefore termed internal-reasoning configura- tions; we neither observe nor interpret provider-side inter- nal traces (Zelikman et al., 2024; Hao et al., 2025). Thus, toggling an internal-reasoning configuration changes an 1 arXiv:2610.10639v1 [cs.LG] 7 Oct 2026 inference setting while we continue to measure only visi- ble output traces and executable code. We examine these channels on a query-matched SQL- pandas (Li et al., 2023; Hill et al., 2026) panel that crosses persona phrasing, target language, visible CoT format, direct generation, three control prefixes, and internal- reasoning configurations. The controls are ablations of coherent NL CoT, not candidate prompting methods: length filler removes task-relevant semantics while re- taining approximate prefix volume; scrambling disrupts word order while retaining related lexical material; and wrong content supplies a reasoning-like prefix from a dif- ferent task. Each is compared with both direct generation and coherent NL CoT. This dual comparison separates a change caused by loss of task-aligned reasoning from a change caused by adding a long, structured context. The central result is not a universal visible-CoT or target-matching advantage. Instead, outcomes vary by persona, target, model configuration, and internal- reasoning setting. In particular, the ablations show that reasoning content and prompt format need not have the same behavioral role: a scrambled prefix can match or exceed coherent NL CoT in localized cells even though it does not preserve a chronological solution plan. Visible CoT also adds token and latency overhead whose return is configuration-specific. The relevant unit of comparison is consequently a model-persona-target configuration, not a global prompting default. In summary, this work makes four contributions: • We present a query-matched evaluation that holds the database task and executable reference result fixed while varying SQL or pandas targets, persona phrasing, direct generation, visible-CoT format, and model configuration. • We test visible NL-CoT with three complementary controls: length filler, scrambled tokens, and wrong content. Each control is compared with both coher- ent NL-CoT and direct generation to separate task alignment from structured-prefix effects. • We find that neither visible CoT nor target-language matching provides a universal accuracy advantage. The observed changes instead depend on the per- sona, target, and model configuration, with a local- ized scrambled-prefix result motivating a prompt- format explanation rather than a new default. • We examine visible CoT and internal-reasoning con- figurations alongside difficulty, token use, and la- tency. These analyses show why direct generation remains an essential reference when selecting a rea- soning configuration. The next Section situates this design within prior work on visible CoT, code-generation intermediates, and inference-time computation before specifying the matched experiment. 2 Related Work Visible CoT and causal interpretation. CoT prompting established that decoded intermediate text can improve performance on multi-step tasks (Wei et al., 2022). In prior work, such text may be supplied in demon- strations, elicited as an output, or used alongside unre- ported model-side computation; it should not automat- ically be read as an internal reasoning trace. What an externally visible rationale contributes remains less set- tled. Ablations of demonstrations and noisy rationales indicate that relevance, ordering, and task fit can mat- ter separately from local plausibility (Wang et al., 2023; Zhou et al., 2024). Other analyses document settings in which an incomplete or misleading trace harms perfor- mance (Zheng et al., 2025; Su et al., 2026). Faithfulness work further cautions that a fluent explanation may not be the causal computation responsible for the final answer (Turpin et al., 2023; Lanham et al., 2023). Our length- filler, scrambled, and wrong-content conditions instanti- ate these concerns as dual-baseline ablations of a visible NL-CoT prefix and output contract for executable ana- lytics generation. Reasoning representations for code. Several code- generation approaches use plans, programs, or struc- tured intermediate representations to decompose a task. Program-aided language models delegate suitable sub- computations to an executable program (Gao et al., 2023), while work on intermediate languages and code prompting compares natural-language, pseudocode, and code-like representations (Deng et al., 2024; Sun et al., 2024; Puerto et al., 2024). Recent studies extend this line to multilingual structured traces (Jin et al., 2025b) and to broad comparisons of CoT strategies in program- ming benchmarks (Jin et al., 2025a). These contributions seek more effective generation procedures and may use reasoning representations differently from our prompt- response trace. In contrast, our objective is diagnostic: the task, reasoning-slot structure, and evaluator are held fixed while the externally visible representation, target language, and user phrasing are crossed. Text-to-SQL planning and analytics generation. Reasoning is also central to text-to-SQL systems, com- monly through schema linking, query planning, verifica- tion, or repair. SQL-of-Thought, for example, combines staged planning and correction in a multi-agent proce- dure (Chaturvedi et al., 2025); STaR-SQL uses reasoning- oriented learning and verification for text-to-SQL (He et al., 2025). Such systems change several ingredients jointly, including the number of calls, feedback, and se- lection policy. Our factorial design does not estimate the value of full agentic systems with extended planning, tool use, or open-ended multi-turn revision. Instead, it isolates whether the surface form of an otherwise matched visible prefix changes SQL or pandas execution accuracy under 2 a bounded recovery policy. Internal computation and adaptive inference. Quiet and latent reasoning approaches distinguish model- side computation from the public token stream (Zelikman et al., 2024; Hao et al., 2025), and test-time-compute work shows that additional inference effort should be consid- ered jointly with task difficulty and resource cost (Snell et al., 2025). In code generation, uncertainty-guided CoT similarly allocates extended reasoning selectively rather than unconditionally (Zhu et al., 2025). These approaches motivate our separate treatment of visible CoT and internal-reasoning configurations, as well as the difficulty- conditioned token and latency analyses. They do not li- cense an equivalence between a longer visible trace and provider-side internal reasoning; our configuration com- parisons observe behavioral consequences, not the content of internal computation. Benchmarks, schema grounding, and persona. Text-to-SQL benchmarks such as Spider and BIRD fore- ground cross-domain schema grounding and execution correctness (Yu et al., 2018; Li et al., 2023); data-science benchmarks demonstrate the additional challenges of re- alistic Python-library code (Lai et al., 2023; Ouyang et al., 2025). The matched setting used here combines database- grounded requests and executable SQL and pandas tar- gets derived from BIRD (Li et al., 2023) with Origi- nal, Novice, and Expert persona phrasings supplied by BIRD-Persona (Hill et al., 2026). The paired text-to- Python/text-to-SQL setting makes persona a particularly important moderator, because the same underlying intent can leave different amounts of procedural and schema de- tail implicit (Hu et al., 2026; Ding et al., 2026). We there- fore analyze wording as part of the experimental design rather than averaging it away. Taken together, this literature motivates a matched evaluation that separates reasoning representation, task phrasing, and inference configuration. The next Section defines that design and the execution-based estimands used to evaluate it. 3 Experimental Design This Section defines the matched factors, common output contract, and execution-based estimands used throughout the paper. It first specifies what is held constant across conditions, then describes the model configurations and the paired statistical comparisons that support the Re- sults in Section 4. 3.1 Matched Panel and Conditions From the 1,534-task BIRD-Persona benchmark (Hill et al., 2026), seeded random sampling retained at most 10 questions from each database-difficulty stratum. After the pre-inference exclusions documented in Appendix A.3, 304 database-grounded questions form the common panel for all six completed model configurations. For every ques- tion, database schema and executable gold answer are fixed while phrasing (Original, Novice, Expert), target (SQL, pandas), and reasoning condition vary. The four explicit CoT conditions share the same eight semantic slots. The representation contract, not the decomposition, is manipulated within those four conditions. NL. Complete each slot with concise English prose and avoid formal code bodies. SQL. Use clause, projection, source, join, predicate, ag- gregate, subquery, alias, and validation fragments. Python. Use literals, assignments, selectors, masks, pan- das transformations, and assertions. Pseudocode. Use an uppercase structured vocabulary including Output, Map, Tables, Columns, Com- pute, Filter, Build, Convert, and Validate. Direct. Receive no visible-CoT definition and return an empty visible thinking list. Three ablations of coherent NL-CoT alter properties of its visible prefix: Length filler. Ablate task-relevant NL CoT sentences while retaining approximately matched textual vol- ume in an eight-label unrelated prefix. Scrambled. Ablate chronological order by permuting the words of a reasoning-like eight-step template while retaining its lexical material. Wrong content. Ablate task relevance by supplying a reasoning trace associated with a different query matched by database/difficulty where possible. Thus each model configuration contains 304× 3× 2× 8 = 14,592 matched trials. Appendix A.2 specifies the prompt slots, controls, query exclusions, evaluator, and full condition matrix. Every trial uses a shared schema-grounded JSON re- sponse contract containing a visible thinking list, target- language code, and an explicit code-type field. The four explicit-CoT prompts require exactly eight labeled steps with the same semantic roles and differ only in the required representation; direct generation requires an empty visible thinking list. The controls retain the shared code and schema guardrails but replace the coherent NL- CoT prefix with the relevant ablation. Thus, the exact- eight-step statistic reported in the Results is an observed response-compliance measure, not a substitute for exe- cution accuracy or a measurement of latent reasoning. Internal-reasoning configurations are toggled separately; they change the inference setting, while the study contin- ues to measure only visible output traces and executable 3 ModelFull evaluated model and config- uration GLM4GLM-4-32B-0414; non-reasoning Qwen3Qwen3-30B-A3B-Instruct-2507; non-reasoning Gemma4-offGemma 4 26B-A4B-IT; internal reasoning disabled Gemma4-onGemma 4 26B-A4B-IT; internal reasoning enabled Gemini Flash-0Gemini2.5Flash;internal- reasoning budget 0 Gemini Flash-24kGemini2.5Flash;internal- reasoning budget 24,576 Table 1: Six completed configurations used in the exper- iments. code. Full prompt and control construction details are given in Appendix A.2. The task panel supplies a database-grounded request, a schema context, executable SQL and pandas reference programs, and three semantically matched persona phras- ings for the same analytic intent (Hill et al., 2026). Orig- inal is the benchmark wording from (Li et al., 2023); Novice uses a less technical user-facing formulation; Ex- pert makes analytic details more explicit. We retain the benchmark’s paired formulation rather than treat- ing these as different questions. Consequently, a within- query comparison isolates a change in prompt wording or reasoning condition while holding the database, reference output, and target language fixed. This is especially im- portant for SQL-pandas comparison: the same analytic result can be declarative in SQL yet require an explicit sequence of DataFrame operations in pandas. Appendix A.1 defines the benchmark-derived anno- tations used in the adjusted analyses: BIRD difficulty, an AmbiSQL-derived balanced ambiguity score (Ding et al., 2026), a SQL-grounded phrasing-specificity score, and Structural Complexity Score (SCS), an equal-weight SQL-structure count. The 304 questions are present for every completed model configuration and every factorial cell. The com- mon frame prevents a model-specific missing row from changing the query distribution used in a paired contrast; Appendix A.3 reports the pre-comparison audit that pro- duced this frame. Table 1 gives the full names and estab- lishes the compact reference names used in the remainder of the paper. The open-weight configurations were served with vLLM on an AWS g6.12xlarge instance using BF16 weights and a 32,768-token context limit. Gemini Flash was queried through the provider API from an Apple M4 Max client with 64 GB unified memory; provider-side infer- ence hardware was not observable. Full infrastructure de- tails, including completion limits, retries, timeouts, soft- ware dependencies, and the execution environment, are reported in Appendix A.4. These serving paths motivate within-configuration, rather than cross-provider, resource comparisons. 3.2 Evaluation and Inference Generated code is executed in an isolated worker and matched to a freshly executed reference result after de- terministic normalization. Parse failures, runtime fail- ures, and missing outputs are incorrect. The primary es- timand is the paired within-query accuracy change rela- tive to direct generation. Exact two-sided McNemar tests, 4,000 query-cluster bootstrap intervals, and Benjamini- Hochberg (BH) correction evaluate 252 primary model- persona-target-condition contrasts. Declared secondary families cover internal-reasoning configuration contrasts, target matching, persona interactions, and the control mechanism tests. A complementary query-clustered lo- gistic Generalized Estimating Equations (GEE) adjusts for target, BIRD difficulty, database, AmbiSQL-derived ambiguity (Ding et al., 2026), SQL-grounded phrasing specificity, and SCS (Hu et al., 2026); exact paired con- trasts remain primary. Complete procedures and all cor- rection families are reported in Appendix C. Each com- pleted cell uses one zero-temperature generation under the fixed experiment seed; the analysis therefore estimates conditional behavior of these six serving configurations, not sampling variability over repeated stochastic decodes. For a reasoning condition m and direct generation 0, the paired effect is the query-average difference b ∆ m = 1 n n X q=1 (y qm − y q0 ), reported in percentage points. Here n = 304, q indexes the retained query, m is the reasoning condition being compared with direct generation, 0 denotes direct gener- ation, and y qm ∈0, 1 is the exact-execution correctness indicator for query q under condition m. Exact McNe- mar tests use only discordant query pairs and are ap- propriate because both conditions are evaluated on the same underlying task. The 95% intervals resample com- plete query clusters rather than individual rows, preserv- ing the dependence induced by the two targets, three phrasings, and eight conditions attached to each query. BH correction controls the declared family of 252 model- persona-target-condition tests. The same-family internal- reasoning, target-language matching, persona interaction, and control-mechanism questions have separate, explic- itly reported correction families because they answer dif- ferent estimands. The complementary GEE is deliberately not a replace- ment for paired inference. It asks whether a condition association persists after including model configuration, target, persona, difficulty, database, ambiguity, phras- ing specificity, and SCS, with query-clustered robust co- variance. This model supplies a check on confounding 4 Inference familyTestsBH discoveries Mode-direct2520 Target matching360 Internal-reasoning toggle9616 Persona interactions7561 Table 2: Multiplicity-controlled inference families. Pri- mary condition tests are the first row; other rows address distinct predeclared questions. by annotated query properties; it does not turn corre- lated benchmark features into independently randomized causes. Token and latency summaries are similarly de- scriptive. They retain individual CoT formats, condition on difficulty, and compare values within a serving con- figuration rather than asserting that latency is directly comparable across infrastructure. This design preserves query-level pairing while mak- ing the scope of each comparison explicit. The next Sec- tion reports the execution accuracy, ablation, internal- reasoning, and resource findings. 4 Results This Section first establishes direct generation as the exe- cution reference, then evaluates visible-CoT formats, ab- lations, internal-reasoning configurations, and resource distributions against that reference. The paired-inference design from Section 3 is retained in every comparison. 4.1 Direct Generation as the Reference Table 3 gives the corresponding descriptive results for Original phrasing. Direct generation is an essential part of this comparison rather than a residual baseline: its ac- curacy differs substantially by target and model configu- ration. Across the six configurations, direct SQL accuracy exceeds direct Python accuracy by 8.22-16.45 percentage points, and direct generation ties the strongest direct-or- explicit-CoT value for Gemma4-off on Python. The four explicit formats do not produce a common ranking across SQL and Python. Together, these cells establish a target- and configuration-dependent reference against which the paired tests below should be read. 4.2 No Universal Visible-CoT Advantage No condition survives the 252-test primary family, and no target-matched format has a BH-corrected advantage (Table 2). The direct-reference matrix above is therefore descriptive evidence of heterogeneity, not a replacement for the corrected paired tests. Persona nevertheless matters. The paired persona- interaction family in Table 2 and the all-persona GEE both retain mode-by-persona evidence after BH correc- tion, indicating that phrasing is not a uniform shift in task difficulty. SCS does not retain a distinct adjusted associa- tion in that model, whereas ambiguity and phrasing speci- ficity do; these covariates describe the benchmark rather than causal persona mechanisms. The appendix supplies the complete persona-by-difficulty matrices, which show why the Original-focused main table must not be general- ized to every phrasing regime. It also shows that a control can be the highest cell in a row, particularly under Novice wording. This pattern motivates the dual-baseline abla- tion analysis below instead of a post-hoc selection of the highest-scoring prefix. The paired accuracy comparisons must be read with their resource distribution in view. Figures 2, 3, and the appendix latency analysis retain target, difficulty, and in- dividual CoT formats: the internal-reasoning view uses signed enabled-minus-disabled resource buckets, whereas the token view locates direct and visible-CoT accuracy in model-adaptive bins. These views show that visible CoT usually occupies larger token and latency budgets, but the associated accuracy return is small and configuration- specific rather than systematic. Harder queries shift both direct and prompted generations toward larger budgets, and enabling internal reasoning changes the resource range occupied by paired evaluations. Thus, the results do not support visible CoT as a generally cost-effective de- fault; they support configuration-specific selection against a direct-generation reference. 4.3 Controls as NL-CoT Ablations The controls test semantic relevance, order, and task alignment in a visible prefix; they are diagnostic abla- tions, not optimized prompts. Table 4 compares each con- trol with both coherent Natural Language (NL) CoT and direct generation. Length filler and wrong content show no corrected difference from either comparator. Scram- bling shows no corrected difference from coherent NL CoT, but is modestly higher than direct generation. This does not imply that scrambled reasoning is useful. Rather, it suggests that a related but non-chronological prefix can sometimes alter the model’s generation regime without preserving a valid stepwise plan. The full matrix shows that this effect is persona- and difficulty-dependent, with the strongest localized result under Novice-SQL. Figure 1 and the appendix compli- ance tests show why the effect should not be attributed to recovery of the eight-step contract: scrambled outputs have low exact-step adherence, and compliance is not as- sociated with correctness within the scrambled control. The supported interpretation is therefore a configuration- specific sensitivity to perturbed task-related context, not a general benefit of incoherent reasoning. 5 ModelTarget CoTControl DirectNLSQLPythonPseudocodemeanFillerScrambledWrong GLM4SQL23.6828.2928.2925.9929.2827.9624.0125.6627.30 GLM4Python15.13 19.7418.4217.1117.7618.26 16.4515.1315.79 Qwen3SQL29.93 33.5530.2633.5530.5931.9931.5834.5430.26 Qwen3Python21.71 22.0421.3820.3920.7221.1323.0325.3324.01 Gemma4-offSQL36.1837.17 37.8337.5036.5137.25 37.1736.8436.51 Gemma4-offPython19.7417.7619.4119.7418.0918.7518.4218.4218.75 Gemma4-onSQL37.8337.8338.1638.4939.1438.4039.1439.4741.12 Gemma4-onPython26.3227.30 27.6326.3225.6626.7324.3425.9925.99 Gemini Flash-0SQL35.8636.5137.1735.5339.8037.2539.4739.1441.12 Gemini Flash-0Python23.0325.0025.9922.0426.3224.84 26.9725.0024.34 Gemini Flash-24kSQL36.84 37.5035.8637.5036.5136.8436.1836.5133.22 Gemini Flash-24kPython26.3225.00 27.6325.9926.3226.2326.6427.6323.36 Table 3: Original-phrasing exact execution accuracy (%) along with the mean of the four explicit-CoT formats. In each row, bold marks the best direct-or-explicit-CoT value and the best control value. Ablation∆ NL ∆ direct BH outcome (NL; direct) Length filler-0.360.90 not detected; not detected Scrambled0.161.43 not detected; higher Wrong content -0.490.77 not detected; not detected Table 4: Original-phrasing control contrasts, pooled over six configurations, targets, and matched queries. ∆ NL is control minus NL-CoT accuracy; ∆ direct is control minus direct-generation accuracy. Changes are percentage points. “Not detected” denotes failure to reject after BH correction; “higher” is an observed corrected contrast. 4.4 Internal-ReasoningConfiguration and Resource-Bounded Accuracy The signed resource plots in Figure 2 provide the model- family detail behind the preceding overview. The paired points occupy different resource ranges for Gemma4 and Gemini Flash, and their accuracy changes differ across targets and difficulty levels. Thus, enabling internal rea- soning redistributes evaluations across resource budgets as well as changing accuracy; it cannot be interpreted as a common treatment across model families. The complete three-persona display is provided in the appendix. Visible CoT must also be evaluated against its re- source cost. Figure 3 keeps the four formats separate and conditions on target and difficulty. Direct and explicit- CoT points are distributed across model-adaptive token bins rather than along a shared accuracy curve: harder queries shift both target languages toward larger budgets, while visible CoT generally extends that spread. The rel- ative positions of direct and CoT points differ by model and target, and latency view (in the appendix) should be compared only within a serving configuration. These budget-stratified summaries are descriptive rather than causal dose-response plots or cross-provider latency com- m-CoTLength fillerScrambledWrong content GLM4 Gemma4-off Gemma4-on Qwen3 Gemini Flash-0 Gemini Flash-24k 99.688.59.227.1 99.595.227.571.1 94.983.015.559.8 99.322.425.432.7 99.917.35.021.2 98.740.710.220.3 Visible exact-eight-step compliance (%) 20 40 60 80 Contract compliance (%) Figure 1: Exact eight-step compliance by model. m-CoT averages the four explicit-CoT formats. parisons. These results distinguish the observed patterns from their causal interpretation. The next Section considers the mechanism hypotheses and the resulting evaluation guidance. 5 Discussion The results favor a conditional account of visible CoT. Direct generation is not uniformly dominated: Table 3 shows that it can match the strongest direct-or-explicit- CoT option, while other cells favor different explicit for- mats. The primary analysis therefore does not support either a universal accuracy benefit or a target-language- matched advantage for the tested panel. Persona is not an ornamental factor: the adjusted interaction and the complete appendix matrices show that phrasing changes 6 255--793 793--1,331 1,331--1,8701,870--2,4082,408--2,9462,946--3,4843,484--4,0234,023--4,5614,561--5,0995,099--5,637 Enabled − disabled token range 0 5 10 15 Original Accuracy change (p) Gemma4 — tokens 13--68 68--123 123--178178--234234--289289--344344--399399--454454--510510--565 Enabled − disabled latency range (s) 0 5 10 15 Gemma4 — latency 414--554554--694694--834834--974 974--1,114 1,114--1,254 1,254--1,3941,394--1,5341,534--1,6741,674--1,814 Enabled − disabled token range −5 0 5 Original Accuracy change (p) Gemini Flash — tokens 2.3--3.43.4--4.54.5--5.65.6--6.76.7--7.97.9--9.0 9--10 10--1111--12 12--13 Enabled − disabled latency range (s) −5 0 5 Gemini Flash — latency Thinking-toggle accuracy changes by signed resource bucket: Original phrasing Color encodes reasoning condition; marker shape encodes target; point size encodes difficulty. Reasoning condition DirectNL CoTSQL CoTPython CoTPseudocode CoT Target / difficulty SQLPYTHONSimpleModerateChallenging Figure 2: Original-phrasing internal-reasoning configuration effects within family-adaptive signed token and latency buckets. Each point is the mean paired enabled-minus-disabled accuracy change in its resource bucket. 801--1,108 1,108--1,4151,415--1,7211,721--2,0282,028--2,3352,335--2,6422,642--2,9492,949--3,2553,255--3,5623,562--3,869 Token bucket 0 20 40 60 80 100 Original Accuracy (%) GLM4 922--1,366 1,366--1,8101,810--2,2542,254--2,6982,698--3,1423,142--3,5853,585--4,0294,029--4,4734,473--4,9174,917--5,361 Token bucket 0 20 40 60 80 100 Gemma4-off 924--2,650 2,650--4,3754,375--6,1006,100--7,8267,826--9,552 9,552--11,277 11,277--13,002 13,002--14,72814,728--16,45416,454--18,179 Token bucket 0 20 40 60 80 100 Gemma4-on 799--1,561 1,561--2,3242,324--3,0863,086--3,849 3,849--4,6114,611--5,373 5,373--6,1366,136--6,8986,898--7,6617,661--8,423 Token bucket 0 20 40 60 80 100 Original Accuracy (%) Qwen3 0--504 504--1,008 1,008--1,5121,512--2,0162,016--2,5202,520--3,0233,023--3,5273,527--4,0314,031--4,5354,535--5,039 Token bucket 0 20 40 60 80 100 Gemini Flash-0 0--2,684 2,684--5,3695,369--8,054 8,054--10,738 10,738--13,42213,422--16,10716,107--18,79218,792--21,47621,476--24,16024,160--26,845 Token bucket 0 20 40 60 80 100 Gemini Flash-24k Accuracy within model-adaptive tokens bins: Original phrasing Color encodes reasoning condition; marker shape encodes target; point size encodes difficulty. CoT condition DirectNL CoTSQL CoTPython CoTPseudocode CoT Target / difficulty SQLPYTHONSimpleModerateChallenging Figure 3: Original-phrasing accuracy within model-adaptive total-token bins. Each point is the mean accuracy in a bin. the operating regime in which a prompt is evaluated. The ablations sharpen the interpretation. Their results do not support the claim that a longer prefix alone ex- plains accuracy movements, nor do they show that mod- els reliably follow an injected wrong plan. Instead, the surprising scrambled outcome is consistent with an in- teraction between task-related tokens and a visible out- put structure. This is a falsifiable mechanism hypothesis: a frozen-trace experiment can independently manipulate token content, sequence, and step format to determine which component carries the effect. For deployment, the relevant comparison is an accuracy-resource frontier, not a single accuracy-per- token ratio. Figures 2, 3, and the appendix latency analysis show that visible CoT usually occupies larger token and latency budgets, while its accuracy gain is small and configuration-specific. Difficulty also shifts both direct and prompted generations toward larger bud- gets, and internal-reasoning settings change the resource range of paired evaluations. These bucket views support configuration-specific selection against direct generation, not a global recommendation to add visible CoT or enable 7 internal reasoning. The next Section consolidates these conditional find- ings and their implications for evaluating reasoning prompts in analytics-code generation. 6 Conclusion In matched analytics-code generation trials, direct gen- eration remains a competitive, configuration-dependent reference rather than a uniformly weaker alternative to visible CoT. Visible CoT is neither a universal opti- mizer nor a reliably target-language-matched one; its ef- fects vary with persona, target, model configuration, and internal-reasoning configuration. The dual-baseline ab- lations indicate that prompt-format and lexical effects can perturb generation independently of coherent visi- ble CoT, while the difficulty-conditioned resource anal- yses show why configuration-specific choices are neces- sary. Together, these findings provide a controlled basis for future work on when visible CoT helps code genera- tion, when it changes behavior without improving it, and when an internal-reasoning configuration is the more con- sequential lever. 7 Limitations The controls are diagnostic ablations of coherent NL- CoT, not counterfactual edits of one frozen rationale. They preserve different properties of an injected prefix and therefore cannot separately identify the causal roles of semantic content, sequence, length, and output struc- ture. The wrong-content and scrambled prefixes provide reasoning-like contexts, not interventions on latent com- putation; the scrambled-control findings should be read as mechanism hypotheses about sensitivity to perturbed task-related context, not as evidence that incoherent rea- soning is generally beneficial. Each matched cell contributes one temperature-zero evaluation under a fixed bounded-recovery policy. Be- cause the runner permits limited recovery after tran- sient call failures, JSON-parse failures, and execution fail- ures, the outcome reflects a controlled single-task evalu- ation loop rather than either a pure first-attempt sam- ple or an unrestricted agentic system. Repeated decoding seeds, service replicas, and independently sampled wrong- content traces would be needed to quantify stochastic and operational variance. Persona phrasing, ambiguity, phrasing specificity, difficulty, and SCS are correlated benchmark descriptors, so the adjusted GEE associations should not be read as independent causal effects. The six configurations provide structured coverage, not a randomized model sample. Same-family internal- reasoning contrasts are informative, but architecture, training data, weights, access path, serving stack, and provider implementation remain confounded across fam- ilies. Token fields may be provider-reported or locally es- timated, and latency is client-observed wall-clock time rather than a hardware-normalized service-time metric; resource analyses therefore support within-configuration comparisons against direct generation, not cross-provider cost benchmarks. Finally, the task scope is executable SQL and pandas analytics generation under a shared JSON response con- tract. The conclusions may not transfer to open-ended software engineering, interactive tool-using agents, alter- native execution engines, multi-turn repair systems, or substantially different reasoning interfaces. A Factorial and Artifact Audit The common analysis frame contains 87,552 unique fac- torial cells: six model configurations, 304 query IDs, three personas, two targets, and eight reasoning condi- tions. Every configuration contributes 14,592 rows. There are no duplicated setting-query-persona-target-condition keys, no missing final-correctness labels, and no miss- ing SCS values in the evaluated frame. The internal- reasoning-enabled configurations use the same cached na- tive gold executions and common question set as their paired disabled or budget-zero configurations. Table 6 summarizes the completed common-frame inventory. This appendix records the construction details needed to audit the annotations, exclusions, controls, serving set- tings, and analyses. A.1 Annotation score definitions The adjusted analyses use three benchmark-derived an- notations: phrasing specificity, ambiguity, and SQL struc- tural complexity. They are not additional experimental treatments. They are fixed query-level or phrasing-level descriptors used to audit whether a reasoning-condition comparison is confounded with properties inherited from the BIRD-Persona benchmark panel. Phrasing specificity. Phrasing specificity is the benchmark-provided wording annotation used in the ad- justed analyses to reflect how explicitly a question word- ing states schema-grounded details relative to its paired SQL task (Hill et al., 2026). We treat it only as a fixed inherited benchmark descriptor of the displayed wording, not as an experimental treatment or a stand-alone mea- sure of semantic equivalence. Ambiguity. Ambiguity is derived from an AmbiSQL- style ambiguity detector and taxonomy (Ding et al., 2026). Let L x be the multiset of level-2 ambiguity labels detected for wording x, and let K be the number of level- 2 taxonomy labels (K = 7 in the detector used here). The coverage and volume components are A cov (x) = |unique(L x )| K , A vol (x) = |L x | |L x | + K . 8 The balanced ambiguity score used as the GEE covariate is A bal (x) = λA cov (x) + (1− λ)A vol (x), with λ = 0.5. Thus the score lies in [0, 1], increases when more ambiguity categories are detected, and also gives diminishing credit for multiple detected ambiguity items. It is a study-specific metric derived from AmbiSQL out- puts, not a claim about AmbiSQL’s original evaluation protocol. Structural Complexity Score. SCS is an equal- weight count of SQL structural features computed from the executable reference SQL after parsing it in the SQLite dialect. Following the matched text-to- Python/text-to-SQL benchmark’s use of SCS (Hu et al., 2026) and related text-to-SQL workload profiling prac- tice (Ma et al., 2025), let c col q , c tab q , c where q , c join q , c agg q , c scalar q , c cte q , and c subq q denote the counts of distinct ref- erenced columns, physical tables, atomic WHERE predi- cates, join pairs, aggregate calls, nonaggregate scalar- function calls, common-table-expression definitions, and nested subqueries for query q. Then SCS q = c col q + c tab q + c where q + c join q +c agg q + c scalar q + c cte q + c subq q . The score characterizes the reference SQL structure. A generated SQL or pandas program can be correct without reproducing the same structure. The following subsections distinguish the instructions imposed on a model from response properties measured after generation. A.2 Prompt construction and compli- ance protocol The main paper specifies the shared three-field JSON re- sponse and eight semantic slots. At implementation level, SQL must be one self-contained SQLite query. Pandas code executes in a supplied DataFrame environment, can- not import modules or access files, networks, or processes, and must assign the answer to result. Schema identi- fiers must come from the displayed context. These target- specific guardrails are identical across reasoning condi- tions. The three controls replace the coherent NL-CoT prefix while retaining the same schema, target-code, and JSON contracts. Length filler is a seeded, unrelated eight-step trace assembled from an everyday-task phrase bank. Its text is extended until its count under the experiment’s shared token counter approximately matches a determin- istic eight-step reasoning-like template for the current query and target. Scrambling applies a seeded permuta- tion to the whitespace-delimited tokens of that template, preserving its token multiset but removing sequence and its visible step labels. Wrong content selects a different query with the same database identifier and difficulty when available, using a seed determined by the query identifier. It injects that source query’s pre-generated NL- CoT trace from a fixed bank; if no bank entry is avail- able, it uses a deterministic non-gold eight-step trace de- rived only from the source question, database identifier, and target. Neither path exposes reference SQL, reference pandas code, or execution evidence to the target trial. For the retained study bank, source-query NL-CoT traces were generated offline with gpt-5-mini (bank prompt version v4) and stored only as thinking-step text plus source-query metadata; evidence fields, gold SQL, and gold pandas code were excluded from the bank entries. We distinguish instruction compliance from correct- ness. For explicit CoT, the eight-string requirement is part of the prompt. For controls, the injected prefix pre- serves, destroys, or replaces visible step labels but does not add the explicit CoT conditions’ exact-eight-output instruction. Exact eight-step output is therefore an ob- served behavioral measure rather than a uniform control requirement. We parse each response against the shared JSON contract and record the number of generated vis- ible thinking steps. Exact step compliance is the indica- tor that the parsed thinking list contains eight strings. The main-paper heatmap aggregates this indicator by model and reports explicit-CoT as its mean over the four formats; the full format-level matrix is reported in Fig- ure 8. Execution correctness remains the primary out- come throughout. A.3 Reproducibility protocol The seeded database-difficulty sampling described in the main paper produced 310 candidates. Before inference, five were excluded because native SQL gold answers were not executable and one because its required function was unsupported by the SQLite evaluator. Exclusions were fixed before condition comparisons and did not depend on model output. Intersecting completed settings then yielded the audited 304-query common frame in Table 6. For each query, the database identifier, difficulty la- bel, schema context, question phrasing, target language, reasoning prefix, generated answer, execution status, nor- malized output, correctness label, token counts, and la- tency were retained. The gold answer was executed once per database and target under the same deterministic evaluator used for model outputs. A model output was marked correct only when its executed result matched the normalized gold result; parse failures, runtime fail- ures, and missing outputs were marked incorrect rather than silently discarded. No semantic judge was used for the primary label. Query order was fixed by the common manifest, and all summaries followed key and missingness au- dits. Reported or locally estimated token fields were re- tained: provider/serving-stack usage metadata was pre- ferred when present, with a common local counter used as 9 fallback. Latency was measured locally with a monotonic, high-resolution clock around the client request/response path and accumulated with the timed trial phases; it is not supplied by a provider. These measurements support within-configuration runtime descriptions rather than a hardware-normalized comparison across serving paths. Generation used a fixed experiment seed (20260630) and temperature zero for the completed runs. Each matched cell contributed one recorded generation; the analysis does not average repeated decoding seeds. Internal-reasoning configurations are identified in Table 6 and the main model table. They are request or service configurations that may alter model-side inference, but no provider-side thought tokens or internal-reasoning traces were requested, retained, or analyzed; every configura- tion is evaluated through the same visible response con- tract and executable code. Model-specific token budgets and provider-side internal-reasoning controls are there- fore treated as serving configurations rather than as archi- tecture isolations. Authentication, transport retries, and provider response metadata were handled before the final row-level audit; no failed request was silently replaced by a second sampled question. No reasoning format, correction family, or resource-bin width was selected by maximizing accuracy. The model panel is descriptive coverage of completed configurations, not a hyperparameter search. A.4 Serving and runtime configuration Table 5 records the execution environment for the six completed configurations. The open-weight configura- tions were served with vLLM on an AWS g6.12xlarge in- stance provisioned with 48 vCPUs, 192 GiB memory, and four NVIDIA L4 GPUs (96 GB aggregate GPU memory). Inference used BF16 weights and a 32,768-token maxi- mum context length. The internal-reasoning switch for Gemma4 was passed at request time, so Gemma4-off and Gemma4-on use the same checkpoint while differing in the enabled inference channel. The Gemini Flash configura- tions were queried through the provider API from a Mac with an M4 Max processor and 64 GB unified memory. This computer was an API client and experiment orches- trator; it was not the provider’s inference host. Accord- ingly, resource totals compare conditions within a serv- ing configuration and should not be read as a hardware- normalized benchmark between the two serving paths. The experiment configuration fixes temperature at zero, the experiment seed at 20260630, a request time- out of 600 seconds, at most two runner-level recov- ery attempts after the initial generation (shared across transient-call, JSON-parse, and execution repair), and a 30-second execution timeout. Non-thinking vLLM gener- ations use a 4,096-token completion limit. Gemma4-on starts at 8,192 tokens and, only when the completion reaches that limit, retries the same prompt at 12,288 and then 16,384 tokens subject to the 32,768-token con- Evaluation pathRuntime configuration GLM4; Qwen3vLLM on AWS g6.12xlarge (4 L4 GPUs, 48 vCPUs, 192 GiB); BF16; 32,768-token context; no internal-reasoning setting. Gemma4-off/onSame AWS/vLLM host and checkpoint; BF16; 32,768-token context; request-level internal reasoning disabled/enabled. Gemini Flash-0/24kProvider API, issued from Mac M4 Max with 64 GB unified memory; provider-side serving hardware not observed. All configurationsTemperature 0; fixed experi- ment seed; bounded repair and execution; deterministic refer- ence normalization. Table 5: Serving and execution configuration. Full model identifiers and the compact reference names used else- where are listed in the main model table. ModelRowsQueriesCorrect (%) GLM414,59230437.38 Qwen314,59230441.18 Gemma4-off14,59230443.54 Gemma4-on14,59230446.57 Gemini Flash-014,59230446.15 Gemini Flash-24k14,59230445.96 Table 6: Completed common-frame inventory. text bound. Gemini Flash uses provider-side internal- reasoning budgets of 0 and 24,576; no thought summaries are requested, and the provider’s final-output limit is not overridden. The analysis analysis environment used Python 3.12 with pandas, SQLGlot, matplotlib, seaborn, statsmodels, and vLLM 0.25.1. The original row-level records do not preserve a complete operating-system image, CUDA/driver image, or provider-side hardware manifest. Each latency is client-observed wall-clock time: it includes the instrumented request/response call and other timed trial phases, rather than an internal provider service-time field. It is therefore a within-configuration outcome. Open-model servers used a bounded context window to retain key-value-cache headroom during batched serv- ing. Generated programs were evaluated in an isolated worker with deterministic normalization before compari- son against a fresh native reference execution. Per-request timeout, transport retry, parsing, execution status, token fields, and locally measured end-to-end latency were re- tained in the row-level artifact. A failed generation con- tributes an incorrect outcome rather than causing a query to be resampled. 10 With the shared panel and runtime inventory fixed, the next section expands the main paper’s Original-only dis- play to every persona and target. As throughout the pa- per, failure to reject a paired difference is not evidence of equivalence because no equivalence margin was prede- clared. B Detailed Persona-Wise Accu- racy Table 7 complements the Original-phrasing matrix in the main paper with every persona, target, and visible-CoT condition. It preserves direct generation as an explicit comparator and shows the phrasing-dependent cells that are necessarily compressed in the main narrative. The matrix supplies the descriptive cells for the paired contrasts below; it does not itself establish a condition effect. In particular, direct generation remains compet- itive in multiple rows, so a highlighted control or CoT cell is not evidence of a universal prompting advantage. The next section specifies the tests, confidence intervals, multiplicity families, and adjusted covariates. C Paired Statistical Procedures C.1 Notation and analysis unit Let q ∈ 1,...,Q index retained queries, with Q = 304 in the common frame. Let s index the six model config- urations, p the persona phrasing, t the target language, and r the reasoning condition. The binary outcome is y qsptr = ( 1, correct, 0, otherwise. Here correct means that the generated program executes and matches the gold result after deterministic normal- ization. All paired contrasts hold q, s, p, and t fixed and vary only the reasoning condition. For the primary fam- ily, the comparator a is direct generation. For the control- mechanism family, a can be direct generation, NL CoT, another control, or the explicit-CoT mean, depending on the stated hypothesis. C.2 McNemar Test For a reasoning condition r and comparator a, let b = X q I(y qsptr = 1, y qspta = 0), c = X q I(y qsptr = 0, y qspta = 1), where I(·) is the indicator function. Thus b counts matched queries on which condition r is correct and com- parator a is wrong, while c counts the opposite discor- dance. Under the paired null, the b + c discordant out- comes are exchangeable with probability one half (McNe- mar, 1947). Let B ∼ Binomial(b + c, 0.5), P − = Pr[B ≤ min(b,c)], and P + = Pr[B ≥ max(b,c)]. The exact two- sided McNemar/binomial value is p McN = min1, 2 min(P − ,P + ). Concordant pairs do not provide evidence of a difference. C.3 Effect Sizes and Confidence Intervals The primary effect is b ∆ sptr,a = 100 Q Q X q=1 (y qsptr − y qspta ), reported in percentage points. Here a is the comparator condition and all other indices are held fixed. We also report a continuity-corrected paired odds ratio d OR paired = b + 0.5 c + 0.5 , where b and c are the discordant counts defined above. Confidence intervals resample the Q = 304 whole-query cluster means with replacement 4,000 times using seed 20260630 and take the 2.5th and 97.5th bootstrap per- centiles. Resampling individual rows would understate uncertainty because each query appears in many condi- tions. C.4 BH False-Discovery Control If M raw values are ordered p (1) ≤ · ≤ p (M) , the Benjamini-Hochberg procedure (Benjamini and Hochberg, 1995) computes the adjusted value p BH (i) = min j≥i min 1, M j p (j) . It controls the expected fraction of false discoveries among rejected hypotheses, not the probability of any false pos- itive. The 252 mode-versus-direct cells form the primary family. Internal-reasoning configuration contrasts, tar- get matching, best-CoT comparisons, persona modera- tion, and control-mechanism screens are separate scien- tific questions and therefore separate declared families. C.5 Token Comparisons Prompt, completion, and total-token differences are paired within query-persona-target, then averaged to one difference per query within each setting. For a token metric z ∈ prompt, completion, total, the query-level paired token difference is d z qsr,a = 1 |P ×T| X p,t (z qsptr − z qspta ), 11 whereP andT are the evaluated persona and target sets. Two-sided Wilcoxon signed-rank tests (Wilcoxon, 1945) operate on those 304 query-level differences, with BH ap- plied separately to each token metric. Query-cluster boot- strap intervals target the mean token difference; the rank test targets a distributional location shift, so the two sum- maries need not agree for a heavy-tailed cell. Token use cannot establish whether a prefix was attended to or se- mantically ignored. C.6 Clustered Logistic GEE The complementary GEE contains 87,552 Bernoulli out- comes and 304 query clusters. GEE estimates population- averaged associations under correlated responses (Liang and Zeger, 1986). For observation i, let q(i) denote its query cluster and let μ i = E[y i ]. We fit a logistic marginal mean model logit(μ i ) = β 0 + β ⊤ X i , where X i contains model configuration, reasoning condi- tion, persona, target, BIRD difficulty (Li et al., 2023), database, balanced ambiguity score, phrasing-specificity score, SCS, and the declared interaction terms. The esti- mating equation is Q X q=1 D ⊤ q V −1 q (Y q − μ q ) = 0, where Y q is the vector of binary outcomes for query q, μ q is the corresponding fitted mean vector, D q = ∂μ q /∂β, and V q is the working covariance matrix. The reported fit uses an independence working correlation and query-clustered robust sandwich covariance. The exchangeable working- correlation fit was numerically unstable and failed to converge; it was discarded. For an omnibus term repre- sented by the contrast matrix R, the null hypothesis is H 0 : Rβ = 0 and the robust Wald statistic is W = (R b β) ⊤ h R d Var( b β)R ⊤ i −1 (R b β), where b β is the fitted coefficient vector and d Var( b β) is its query-clustered sandwich covariance; W is compared with a chi-square distribution having rank(R) degrees of free- dom. Table 8 lists the BH-adjusted omnibus Wald fami- lies. C.7 Structural Complexity and Other Query Covariates SCS is retained as a standardized query-level structural- complexity annotation in the adjusted GEE so that a reasoning-condition comparison is not mistaken for a dif- ference in the structural demands of the query. Its BH- adjusted omnibus value in Table 8 is 0.362, so it does not provide a distinct adjusted association with correctness in this panel. Difficulty is likewise non-significant after ad- justment, whereas ambiguity and phrasing specificity re- main associated with correctness. These variables are cor- related features of the benchmark and persona rewrites, not independently manipulated causal treatments; the paired condition tests remain the primary evidence. These procedures distinguish confirmatory paired com- parisons from descriptive or focused analyses. The next section reports the Original- and Novice-specific families that were used to probe localized directions without re- placing the primary family. D Original and Novice Focused Families Expert phrasing intentionally exposes more task struc- ture, so we also analyzed Original and Novice phrasings as separate focused families. These analyses do not re- place the full 252-test primary family. They ask whether less-explicit phrasing contains a clearer condition effect after removing expert prompts from the family under test. Table 9 gives the leading pooled focused contrasts. Original phrasing contains positive raw movement, es- pecially for SQL-side visible CoT, but no corrected dis- covery. Across the 48 Original-only explicit-CoT per- model cells, the smallest adjusted value is 0.481; across the 14 pooled Original all-condition tests, the smallest adjusted value is 0.196. Within the Original-only pooled contrasts, the largest positive SQL-side movements occur for pseudocode CoT, scrambled context, and NL CoT. Novice phrasing gives the only focused corrected re- sult, and it is not an explicit-CoT language: scrambled context improves SQL accuracy by 5.15 points after cor- rection across the 14 Novice all-condition pooled tests. Explicit CoT remains non-significant for Novice phras- ing; the smallest adjusted value in the pooled explicit- CoT family is 0.652. The narrowed GEE over Original and Novice rows converges and again does not make the omnibus mode family significant (p BH = 0.270), while the mode-by-phrasing interaction is suggestive but not corrected (p BH = 0.109). An additional exploratory test averages the four ex- plicit CoT assignments within query before compar- ing them with direct generation. Original-SQL is +1.56 points (95% query-bootstrap interval [0.23, 2.93], raw p = 0.0239), but does not cross the BH threshold across the four focused persona-target tests (p BH = 0.0954). GLM4 has the clearest model-wise Original direction: +4.28 points for SQL (raw p = 0.0154, p BH = 0.0921 across the six models in that persona-target family) and +3.13 points for Python (raw p = 0.0355). No aggre- gate explicit-CoT cell is a corrected discovery. Seventeen of 24 focused model-persona-target aggregate-CoT deltas are positive; Original-SQL has five positive configurations and one zero-delta configuration. This is a descriptive fixed-panel direction, not an independent-systems signifi- 12 cance test. These analyses were motivated after inspection and are labeled post hoc; they do not change the primary family. D.1 Focused Control Mechanism Con- trasts The focused control analysis keeps Original and Novice phrasing separate and uses two matched comparators for every ablation. The NL-CoT comparison asks whether re- moving coherent task-relevant content, disrupting order, or introducing wrong-task content changes the visible- reasoning condition. The direct-generation comparison asks whether the same prefix changes accuracy at all rela- tive to a prompt without visible CoT. Table 10 reports the corrected contrasts that localize the Novice-SQL scram- bled result. The Novice-SQL result is not a generic length effect: no corrected length-filler difference from direct is detected, while scrambled exceeds both filler and wrong content. Nor does it establish a benefit from incoherent reason- ing. The interventions do not share one frozen trace, and the supplied prefix changes the model’s own generated reasoning behavior. The most specific supported claim is that the Novice-SQL configuration is sensitive to a scram- bled reasoning-like context. At the model level, scrambled Novice-SQL gains are +7.89 points for Gemini Flash-0, +7.24 for GLM4, and +6.58 for Qwen3. These three survive correction over the 18 controls inside the Novice-SQL screen, but none sur- vives the wider 72-cell focused-control correction. The pooled fixed-panel result therefore captures a repeated directional effect within the completed panel. The difficulty split is shown separately in Table 11. Novice scrambled contrasts are positive across all three difficulty levels, with the strongest corrected evidence on simple and challenging queries and a smaller direct- generation contrast on moderate queries. The correspond- ing Original contrasts do not survive their correction family. This localization is consistent with a phrasing- dependent sensitivity, not a difficulty-invariant property of token permutation. The focused control results motivate a separate look at internal-reasoning configuration, which is the next chan- nel of variation examined below. E Internal-Reasoning Configura- tion Detail The 96-contrast family contains two model families, three personas, two targets, and eight reasoning conditions. Six- teen cells survive BH. Gemma4 accounts for 15: SQL-CoT under Expert-Python; five Novice-Python visible/control cells; all eight Original-Python conditions, including di- rect; and wrong-content Original-SQL. Gemini Flash con- tributes one corrected cell: wrong-content Original-SQL, where Flash-24k is 7.89 points below Flash-0. This distri- bution explains why a single pooled “internal reasoning helps” coefficient would be misleading. Figure 4 shows the complete configuration-contrast matrix. The contrast matrix establishes that changing the internal-reasoning configuration is not a single global factor. It does not identify the contents or amount of model-side internal reasoning: the analysis observes only accuracy, resources, and the externally visible response trace under each configuration. The next section evalu- ates whether target-matched explicit CoT or a selected best format provides a corresponding global rule. F Target Matching, Controls, and Best Formats No target-matched comparison survives BH across 36 cells, and no selected best CoT is significantly supe- rior to every alternative in its corrected family. This is not evidence that representations are interchangeable; in- teraction tests show that their effects vary by persona and model. It is evidence against a single global target- matching rule. Six cells survive the narrower seven-tests-within-cell correction, but none survives the 252-test primary family. Three are scrambled Novice-SQL gains (GLM4, Qwen3, and Gemini Flash-0); two are GLM4 Expert-Python losses for NL/SQL CoT; one is a Gemma4-on Expert- Python SQL-CoT gain. These are localized within-cell results, not corrected discoveries in the primary family. Because a representation can change resource use even when it has no global accuracy advantage, the next sec- tion characterizes the associated token and latency trade- offs. G Efficiency Detail The efficiency analysis treats each matched model- persona-target-condition cell as an accuracy-resource point. We report direct generation and each of the four explicit CoTs separately; controls remain in the abla- tion and step-compliance analyses but are not assigned a cost/latency comparison here. This makes the marginal cost of each visible CoT explicit without pretending that a single accuracy-per-token ratio is comparable across providers. Table 12 reports the individual formats, while Table 13 gives their compact mean and Table 14 separates task difficulty from prefix overhead. The bucket tables use model-adaptive bins. The ini- tial width targets roughly 10% of a model’s observed re- source span, is rounded to a readable standard width, and is widened when necessary to retain at most ten plotted bucket positions in a model panel. Every explicit CoT format remains separate. Figures 5 and 6 use persona- faceted scatter plots with the six model configurations 13 expert-pythonexpert-sqlnovice-pythonnovice-sqloriginal-pythonoriginal-sql Persona / target Gemini 2.5 Flash-cot_nl Gemini 2.5 Flash-cot_pseudocode Gemini 2.5 Flash-cot_python Gemini 2.5 Flash-cot_sql Gemini 2.5 Flash-length_filler Gemini 2.5 Flash-none Gemini 2.5 Flash-scrambled Gemini 2.5 Flash-wrong Gemma 4-cot_nl Gemma 4-cot_pseudocode Gemma 4-cot_python Gemma 4-cot_sql Gemma 4-length_filler Gemma 4-none Gemma 4-scrambled Gemma 4-wrong Family / visible condition 1.60.3-0.3-1.30.01.0 0.70.0-1.3-2.30.0-3.3 4.6-2.01.0-0.33.92.0 2.0-1.61.0-0.71.6-1.3 -1.0-0.3-0.31.0-0.3-3.3 -1.6-0.31.00.73.31.0 2.3-1.01.3-4.32.6-2.6 -1.0-1.00.3-1.6-1.0-7.9 1.6-0.35.61.39.50.7 2.00.75.31.37.62.6 2.30.34.90.36.61.0 5.60.74.62.08.20.3 2.02.05.62.35.92.0 1.0-1.04.91.06.61.6 1.6-0.35.60.07.62.6 3.3-0.34.30.37.24.6 −6 −4 −2 0 2 4 6 8 Figure 4: Complete same-family internal-reasoning effects (enabled minus disabled, or budget 24,576 minus 0). arranged in a 2-by-3 layout inside each Original, Novice, and Expert block. Color encodes the reasoning condi- tion, marker shape identifies the target language, and point size encodes difficulty; each point is the mean ac- curacy within the corresponding model-adaptive bucket. The y-axis is tightened separately within each persona row block so that the central mass of points is easier to read. Bucket widths target roughly 10% of the observed span, are rounded to readable standard widths, and are widened for high-range panels when needed to avoid more than ten plotted positions. Harder tasks move both direct and CoT conditions toward larger resource budgets, while each explicit CoT adds further overhead whose accuracy return is model- and target-dependent. Figure 7 applies the same signed-bucket construction to enabled-minus- disabled thinking changes for each model family. Table 15 lists representative accuracy, token, and la- tency points used to interpret cost trade-offs. Every control increases both completion and total to- kens relative to direct generation in all six configura- tions after BH correction. Compared with the mean of the four explicit-CoT conditions, control completions are shorter in 14 of 18 model-by-control cells, and all 14 dif- ferences are corrected. Shorter output does not identify noise rejection: controls still generate 8.6-10.2 thinking steps on average. Exact adherence to the intended eight- step count is 58.5% for length filler, 15.6% for scrambled, and 39.2% for wrong content, compared with 99.5-99.9% for explicit CoT. The control signature is therefore shorter but more fragmented generated reasoning plus a posi- tive token overhead relative to direct generation. Figure 8 shows the exact-step compliance pattern behind this in- terpretation. Table 16 then tests whether exact-step com- pliance is associated with correctness within each control after adjustment for model configuration, persona, target, and difficulty. Compliance is positively associated with correctness for length filler and wrong content, but not for scrambled context (Table 16). This difference supports two behav- ioral axes: adherence to the visible output structure for the labeled controls, and a separate lexical/context sen- sitivity after token permutation. Because compliance is observed rather than randomized, these adjusted associa- tions do not establish that producing exactly eight steps causes a correct answer. The bucket figures should be interpreted as budget- stratified descriptive comparisons, not causal dose- response curves. Formal equivalence or non-inferiority 14 tests with predeclared margins are needed before recom- mending a cheaper configuration as a substitute. References Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing.Journal of the Royal Statisti- cal Society: Series B (Methodological), 57(1):289-300, 1995. doi: https://doi.org/10.1111/j.2517-6161.1995. tb02031.x.URL https://rss.onlinelibrary.wiley.com/ doi/abs/10.1111/j.2517-6161.1995.tb02031.x. Saumya Chaturvedi, Aman Chadha, and Laurent Bind- schaedler. SQL-of-thought: Multi-agentic text-to-SQL with guided error correction. In NeurIPS 2025 Fourth Workshop on Deep Learning for Code, 2025. URL https://openreview.net/forum?id=NBDa9012jE. Xun Deng, Sicheng Zhong, Honghua Dong, Jingyu Hu, Sidi Mohamed Beillahi, Xujie Si, and Fan Long. As- sessing code generation with intermediate languages. arXiv preprint arXiv:2407.05411, 2024. URL https: //arxiv.org/abs/2407.05411. Zhongjun Ding, Yin Lin, Tianjing Zeng, Rong Zhu, Bolin Ding, and Jingren Zhou. Ambisql: Interactive ambigu- ity detection and resolution for text-to-sql. In Compan- ion of the International Conference on Management of Data, SIGMOD Companion ’26, page 26–29, New York, NY, USA, 2026. Association for Computing Machinery. ISBN 9798400724503. doi: 10.1145/3788853.3801581. URL https://doi.org/10.1145/3788853.3801581. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. In Pro- ceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10764-10799, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/gao23f.html. Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason E Weston, and Yuandong Tian. Training large language models to reason in a contin- uous latent space. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum? id=Itxz7S4Ip3. Mingqian He, Yongliang Shen, Wenqi Zhang, Qiuying Peng, Jun Wang, and Weiming Lu. STaR-SQL: Self- taught reasoner for text-to-SQL. In Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 24365-24375, Vienna, Austria, July 2025. ISBN 979- 8-89176-251-0.doi: 10.18653/v1/2025.acl-long.1187. URL https://aclanthology.org/2025.acl-long.1187/. Davin Hill, Pawan Chorasiya, Bhawani Shankar Leelar, Oren Avram, Robert E. Tillman, and Tamer Soliman. Bird-persona: A persona-aware benchmark for sql and pandas analytics code generation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP 2026): Industry Track, Bu- dapest, Hungary, October 2026. Association for Com- putational Linguistics. Hangle Hu, Chenyu Hou, Bin Cao, and Ruizhe Li. Bench- marking text-to-python against text-to-sql: The im- pact of explicit logic and ambiguity. arXiv preprint arXiv:2601.15728, 2026. URL https://arxiv.org/abs/ 2601.15728. Naizhu Jin, Zhong Li, Guang Yang, Tian Zhang, and Qingkai Zeng.Understanding chain-of-thought ef- fectiveness in code generation: An empirical and information-theoretic analysis.2025a.URL https: //arxiv.org/abs/2512.09679. Naizhu Jin, Zhong Li, Tian Zhang, and Qingkai Zeng. MSCoT: Structured chain-of-thought generation for multiple programming languages. In 2025 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2025b. doi: 10.1109/IJCNN64981.2025.11228673. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pages 18319-18345, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/lai23b. html. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, et al. Measuring faith- fulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023. URL https://arxiv.org/abs/ 2307.13702. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chang, Fei Huang, Reynold Cheng, and Yongbin Li. Can LLM already serve as a database interface? a BIg bench for large-scale database grounded text-to-SQLs.In Advances in Neural Information Processing Systems, volume 36, pages 42330-42357, 2023. doi: 10.52202/075280-1835. URL https://proceedings.neurips.c/paper files/paper/ 2023/file/83fc8fab1710363050bbd1d4b8c0021-Paper- Datasets andBenchmarks.pdf. Kung-Yee Liang and Scott L. Zeger. Longitudinal data analysis using generalized linear models. Biometrika, 73(1):13-22, 1986. ISSN 00063444. URL http://w. jstor.org/stable/2336267. 15 Limin Ma, Ken Pu, Ying Zhu, and Wesley Taylor. Com- paring large language models for generating complex queries. Journal of Computer and Communications, 13 (2):236-249, 2025. Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percent- ages. Psychometrika, 12(2):153-157, 1947. doi: 10.1007/ BF02295996. Shuyin Ouyang, Dong Huang, Jingwen Guo, Zeyu Sun, Qihao Zhu, and Jie M. Zhang. DSCodebench: A realis- tic benchmark for data science code generation. 2025. URL https://arxiv.org/abs/2505.15621. Haritz Puerto, Martin Tutek, Somak Aditya, Xiaodan Zhu, and Iryna Gurevych.Code prompting elicits conditional reasoning abilities in Text+Code LLMs. In Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 11234-11258, Miami, Florida, USA, November 2024. doi: 10.18653/v1/2024.emnlp-main.629. URL https: //aclanthology.org/2024.emnlp-main.629/. Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learn- ing Representations, 2025. URL https://openreview. net/forum?id=4FWAwZtd2n. Ian Su, Gaurav Purushothaman, Jey Narayan, Ruhika Goel, Kevin Zhu, Sunishchal Dev, Yash More, and Ma- heep Chaudhary. Broken chains: The cost of incomplete reasoning in llms. arXiv preprint arXiv:2602.14444, 2026. URL https://arxiv.org/abs/2602.14444. Tao Sun, Linzheng Chai, Jian Yang, Yuwei Yin, Hongcheng Guo, Jiaheng Liu, Bing Wang, Liqun Yang, and Zhoujun Li. UniCoder: Scaling code large language model via universal code. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1812-1824, Bangkok, Thailand, August 2024. doi: 10.18653/v1/ 2024.acl-long.100. URL https://aclanthology.org/2024. acl-long.100/. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman.Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, volume 36, pages 74952-74965, 2023. doi: 10.52202/075280-3275. URL https://proceedings.neurips.c/paper files/paper/ 2023/file/ed3fea9033a80fea1376299fa7863f4a-Paper- Conference.pdf. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empir- ical study of what matters. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2717-2739, Toronto, Canada, July 2023. doi: 10.18653/v1/2023. acl-long.153. URL https://aclanthology.org/2023.acl- long.153/. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837, 2022. doi: 10.52202/068431-1800. URLhttps://proceedings.neurips.c/paper files/ paper/2022/file/9d5609613524ecf4f15af0f7b31abca4- Paper-Conference.pdf. Frank Wilcoxon.Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. doi: 10. 2307/3001968. URL https://doi.org/10.2307/3001968. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 3911-3921, Brussels, Belgium, October- November 2018. doi: 10.18653/v1/D18-1425. URL https://aclanthology.org/D18-1425/. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet- STaR: Language models can teach themselves to think before speaking.arXiv preprint arXiv:2403.09629, 2024. URL https://arxiv.org/abs/2403.09629. Tianshi Zheng, Yixiang Chen, Chengxi Li, Chunyang Li, Qing Zong, Haochen Shi, Baixuan Xu, Yangqiu Song, Ginny Wong, and Simon See. The curse of CoT: On the limitations of chain-of-thought in in-context learn- ing. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https://openreview.net/forum? id=7SIrvcYNYj. Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of-thought prompt- ing with noisy rationales?In Advances in Neural Information Processing Systems, volume 37, pages 123846-123910, 2024.doi: 10.52202/079017-3936. URLhttps://proceedings.neurips.c/paper files/ paper/2024/file/dfaa29ed28dfa175bcc5e2a54a199f8- Paper-Conference.pdf. Yuqi Zhu, Ge Li, Xue Jiang, Jia Li, Hong Mei, Zhi Jin, and Yihong Dong.Uncertainty-guided chain- of-thought for code generation with llms.CoRR, 16 abs/2503.15341, 2025.doi: 10.48550/ARXIV.2503. 15341.URL https://doi.org/10.48550/arXiv.2503. 15341. 17 ModelPersonaTargetDirect NL CoT SQL CoT Python CoT Pseudo code CoT Length fillerScrambled Wrong content GLM4expertSQL88.1687.8386.1887.8388.8290.7986.8487.83 GLM4expertPYTHON60.2054.6153.9555.5957.8960.2060.8655.92 GLM4noviceSQL19.0821.3818.4221.0518.4220.7226.3220.72 GLM4novicePYTHON14.1414.8013.4915.1315.1314.1416.4513.16 GLM4originalSQL23.6828.2928.2925.9929.2824.0125.6627.30 GLM4originalPYTHON15.13 19.7418.4217.1117.7616.4515.1315.79 Gemini Flash-0expertSQL92.4391.45 93.4293.4290.7992.1192.1192.11 Gemini Flash-0expertPYTHON68.4265.1367.1164.1467.7668.4266.4566.78 Gemini Flash-0noviceSQL30.9232.5733.5530.9233.2230.9238.8234.21 Gemini Flash-0novicePYTHON20.3922.7023.3620.7222.7021.3824.0119.41 Gemini Flash-0originalSQL35.8636.5137.1735.5339.8039.4739.1441.12 Gemini Flash-0originalPYTHON23.0325.0025.9922.0426.3226.9725.0024.34 Gemini Flash-24kexpertSQL92.1191.7891.7891.4590.7991.7891.1291.12 Gemini Flash-24kexpertPYTHON66.7866.78 69.0868.7568.4267.4368.7565.79 Gemini Flash-24knoviceSQL31.5831.2532.8930.5930.9231.9134.5432.57 Gemini Flash-24knovicePYTHON21.3822.3724.3421.7121.3821.0525.3319.74 Gemini Flash-24koriginalSQL36.84 37.5035.8637.5036.5136.1836.5133.22 Gemini Flash-24koriginalPYTHON26.3225.00 27.6325.9926.3226.6427.6323.36 Gemma4-offexpertSQL93.7593.09 93.7592.4393.4292.4393.7593.75 Gemma4-offexpertPYTHON64.8064.8064.4764.8066.1264.1464.8064.47 Gemma4-offnoviceSQL29.6130.2628.9530.5929.9327.6333.2230.92 Gemma4-offnovicePYTHON18.0917.1117.1117.4316.7815.7918.4217.43 Gemma4-offoriginalSQL36.1837.17 37.8337.5036.5137.1736.8436.51 Gemma4-offoriginalPYTHON19.7417.7619.4119.7418.0918.4218.4218.75 Gemma4-onexpertSQL92.7692.76 94.4192.7694.0894.4193.4293.42 Gemma4-onexpertPYTHON65.7966.45 70.0767.1168.0966.1266.4567.76 Gemma4-onnoviceSQL30.5931.5830.9230.9231.2529.9333.2231.25 Gemma4-onnovicePYTHON23.0322.7021.7122.3722.0421.3824.0121.71 Gemma4-onoriginalSQL37.8337.8338.1638.4939.1439.1439.4741.12 Gemma4-onoriginalPYTHON26.32 27.30 27.6326.3225.6624.3425.9925.99 Qwen3expertSQL91.1289.8090.7989.4790.7991.7892.4391.12 Qwen3expertPYTHON58.8858.8859.8754.6158.2261.8462.8362.17 Qwen3noviceSQL23.0325.9923.0323.6825.3324.3429.6125.33 Qwen3novicePYTHON16.1219.0816.1213.8217.4318.0918.4219.74 Qwen3originalSQL29.9333.5530.2633.5530.5931.5834.5430.26 Qwen3originalPYTHON21.7122.0421.3820.3920.7223.0325.3324.01 Table 7: Full persona-wise exact execution accuracy (%). The best condition in each model-persona-target row is bold. 18 801--1,108 1,108--1,4151,415--1,7211,721--2,0282,028--2,3352,335--2,6422,642--2,9492,949--3,2553,255--3,5623,562--3,869 Token bucket 0 20 40 60 80 100 Original Accuracy (%) GLM4 922--1,366 1,366--1,8101,810--2,2542,254--2,6982,698--3,1423,142--3,5853,585--4,0294,029--4,4734,473--4,9174,917--5,361 Token bucket 0 20 40 60 80 100 Gemma4-off 924--2,650 2,650--4,3754,375--6,1006,100--7,8267,826--9,552 9,552--11,277 11,277--13,002 13,002--14,72814,728--16,45416,454--18,179 Token bucket 0 20 40 60 80 100 Gemma4-on 799--1,561 1,561--2,3242,324--3,0863,086--3,849 3,849--4,6114,611--5,373 5,373--6,1366,136--6,8986,898--7,6617,661--8,423 Token bucket 0 20 40 60 80 100 Original Accuracy (%) Qwen3 0--504 504--1,008 1,008--1,5121,512--2,0162,016--2,5202,520--3,0233,023--3,5273,527--4,0314,031--4,5354,535--5,039 Token bucket 0 20 40 60 80 100 Gemini Flash-0 0--2,684 2,684--5,3695,369--8,054 8,054--10,738 10,738--13,42213,422--16,10716,107--18,79218,792--21,47621,476--24,16024,160--26,845 Token bucket 0 20 40 60 80 100 Gemini Flash-24k 803--1,125 1,125--1,4471,447--1,7691,769--2,0912,091--2,4122,412--2,7342,734--3,0563,056--3,3783,378--3,7003,700--4,022 Token bucket 0 20 40 60 80 100 Novice Accuracy (%) GLM4 916--1,521 1,521--2,1262,126--2,7302,730--3,3353,335--3,9403,940--4,5454,545--5,1505,150--5,7545,754--6,3596,359--6,964 Token bucket 0 10 20 30 40 50 60 70 Gemma4-off 923--2,808 2,808--4,6924,692--6,5766,576--8,461 8,461--10,346 10,346--12,230 12,230--14,11414,114--15,999 15,999--17,88417,884--19,768 Token bucket 0 20 40 60 80 100 Gemma4-on 787--1,543 1,543--2,2982,298--3,0543,054--3,8103,810--4,5664,566--5,3215,321--6,0776,077--6,8336,833--7,5887,588--8,344 Token bucket 0 10 20 30 40 50 60 70 Novice Accuracy (%) Qwen3 0--520 520--1,039 1,039--1,5591,559--2,0782,078--2,598 2,598--3,1183,118--3,637 3,637--4,1574,157--4,6764,676--5,196 Token bucket 0 20 40 60 80 100 Gemini Flash-0 0--3,300 3,300--6,6006,600--9,900 9,900--13,200 13,200--16,50016,500--19,79919,799--23,09923,099--26,39926,399--29,69929,699--32,999 Token bucket 0 20 40 60 80 100 Gemini Flash-24k 853--1,191 1,191--1,5281,528--1,8661,866--2,2032,203--2,5412,541--2,8792,879--3,2163,216--3,5543,554--3,8913,891--4,229 Token bucket 0 20 40 60 80 100 Expert Accuracy (%) GLM4 965--1,631 1,631--2,2972,297--2,9642,964--3,6303,630--4,2964,296--4,9624,962--5,6285,628--6,2956,295--6,9616,961--7,627 Token bucket 0 20 40 60 80 100 Gemma4-off 975--2,836 2,836--4,6964,696--6,5566,556--8,417 8,417--10,278 10,278--12,13812,138--13,99813,998--15,85915,859--17,72017,720--19,580 Token bucket 0 20 40 60 80 100 Gemma4-on 873--1,608 1,608--2,3422,342--3,076 3,076--3,8113,811--4,546 4,546--5,2805,280--6,0146,014--6,7496,749--7,4847,484--8,218 Token bucket 0 20 40 60 80 100 Expert Accuracy (%) Qwen3 0--459 459--917 917--1,376 1,376--1,8351,835--2,2942,294--2,752 2,752--3,2113,211--3,670 3,670--4,1284,128--4,587 Token bucket 0 20 40 60 80 100 Gemini Flash-0 0--1,487 1,487--2,9742,974--4,4614,461--5,9485,948--7,4347,434--8,921 8,921--10,408 10,408--11,89511,895--13,382 13,382--14,869 Token bucket 0 20 40 60 80 100 Gemini Flash-24k Accuracy within model-adaptive tokens bins: Original, Novice, Expert phrasing Color encodes reasoning condition; marker shape encodes target; point size encodes difficulty. CoT condition DirectNL CoTSQL CoTPython CoTPseudocode CoT Target / difficulty SQLPYTHONSimpleModerateChallenging Figure 5: Accuracy within model-adaptive token bins, stratified by model, persona, target, difficulty, and reasoning condition. Original, Novice, and Expert are shown; point size encodes difficulty and the y-axis is tightened within each persona row block. Bucket widths target roughly 10% of the span and widen when necessary to keep high-range panels readable. 19 8--140 140--271271--403403--534534--666666--797797--929 929--1060 1060--11921192--1323 Latency bucket (s) 0 10 20 30 40 Original Accuracy (%) GLM4 7--76 76--144 144--213213--282282--350350--419419--488488--556556--625625--694 Latency bucket (s) 0 20 40 60 80 100 Gemma4-off 6--387 387--767 767--1148 1148--1529 1529--19091909--22902290--26702670--30513051--34313431--3812 Latency bucket (s) 0 20 40 60 80 100 Gemma4-on 6--62 62--118 118--174 174--230230--285285--341341--397397--453453--509509--565 Latency bucket (s) 0 20 40 60 80 100 Original Accuracy (%) Qwen3 2.5--8.6 9--15 15--2121--2727--3333--3939--4545--5151--5757--64 Latency bucket (s) 0 20 40 60 80 100 Gemini Flash-0 0--54 54--108 108--162162--216216--271271--325325--379379--433433--487487--541 Latency bucket (s) 0 20 40 60 80 100 Gemini Flash-24k 8--440 440--872 872--1303 1303--17351735--21672167--25992599--30303030--34623462--38943894--4325 Latency bucket (s) 0 5 10 15 20 25 30 Novice Accuracy (%) GLM4 6--434 434--862 862--1290 1290--17171717--21452145--25732573--30013001--34283428--38563856--4284 Latency bucket (s) 0 20 40 60 80 100 Gemma4-off 7--437 437--867 867--1298 1298--17281728--21592159--25892589--30203020--34503450--3880 3880--4311 Latency bucket (s) 0 10 20 30 40 50 Gemma4-on 6--59 59--113 113--166 166--219219--272272--326326--379379--432432--486486--539 Latency bucket (s) 0 10 20 30 40 50 Novice Accuracy (%) Qwen3 0--313 313--625625--938 938--1251 1251--15631563--18761876--21892189--25022502--28142814--3127 Latency bucket (s) 0 5 10 15 20 25 30 35 40 Gemini Flash-0 0--54 54--108 108--162162--216216--270270--325325--379379--433433--487487--541 Latency bucket (s) 0 20 40 60 80 100 Gemini Flash-24k 8--140 140--272272--403403--535535--667667--799799--930 930--1062 1062--11941194--1326 Latency bucket (s) 0 20 40 60 80 100 Expert Accuracy (%) GLM4 7--79 79--151 151--223223--295295--367367--439 439--511511--583 583--655655--727 Latency bucket (s) 0 20 40 60 80 100 Gemma4-off 6--359 359--711 711--1063 1063--14151415--17681768--21202120--24722472--28252825--31773177--3529 Latency bucket (s) 0 20 40 60 80 100 Gemma4-on 6--46 46--86 86--126 126--166166--206206--246246--286286--326326--366366--405 Latency bucket (s) 0 20 40 60 80 100 Expert Accuracy (%) Qwen3 0.0--3.63.6--7.1 7--11 11--14 14--1818--2121--2525--2828--3232--36 Latency bucket (s) 0 20 40 60 80 100 Gemini Flash-0 0--54 54--108 108--162162--216216--270270--325325--379379--433433--487487--541 Latency bucket (s) 0 20 40 60 80 100 Gemini Flash-24k Accuracy within model-adaptive latency bins: Original, Novice, Expert phrasing Color encodes reasoning condition; marker shape encodes target; point size encodes difficulty. CoT condition DirectNL CoTSQL CoTPython CoTPseudocode CoT Target / difficulty SQLPYTHONSimpleModerateChallenging Figure 6: Accuracy within model-adaptive latency bins, stratified by model, persona, target, difficulty, and reasoning condition. Original, Novice, and Expert are shown; point size encodes difficulty and the y-axis is tightened within each persona row block. Bucket widths are chosen from roughly 10% of the observed span and rounded to the nearest standard latency bin. 20 255--793 793--1,331 1,331--1,8701,870--2,4082,408--2,9462,946--3,4843,484--4,0234,023--4,5614,561--5,0995,099--5,637 Enabled − disabled token range 0 5 10 15 Original Accuracy change (p) Gemma4 — tokens 13--68 68--123 123--178178--234234--289289--344344--399399--454454--510510--565 Enabled − disabled latency range (s) 0 5 10 15 Gemma4 — latency 414--554554--694694--834834--974 974--1,114 1,114--1,254 1,254--1,3941,394--1,5341,534--1,6741,674--1,814 Enabled − disabled token range −5.0 −2.5 0.0 2.5 5.0 7.5 Original Accuracy change (p) Gemini Flash — tokens 2.3--3.43.4--4.54.5--5.65.6--6.76.7--7.97.9--9.0 9--10 10--1111--12 12--13 Enabled − disabled latency range (s) −5.0 −2.5 0.0 2.5 5.0 7.5 Gemini Flash — latency 1,102--1,5171,517--1,9331,933--2,3492,349--2,7642,764--3,1803,180--3,595 3,595--4,0114,011--4,427 4,427--4,8424,842--5,258 Enabled − disabled token range −2.5 0.0 2.5 5.0 7.5 10.0 12.5 Novice Accuracy change (p) Gemma4 — tokens 80--125 125--169169--214214--258258--303303--347347--392392--436436--481481--526 Enabled − disabled latency range (s) −2.5 0.0 2.5 5.0 7.5 10.0 12.5 Gemma4 — latency 705--868 868--1,031 1,031--1,1941,194--1,3581,358--1,5211,521--1,6841,684--1,847 1,847--2,0112,011--2,174 2,174--2,337 Enabled − disabled token range −6 −4 −2 0 2 4 Novice Accuracy change (p) Gemini Flash — tokens -22---19-19---15 -15---11 -11---8 -7.6---4.0-4.0---0.4 -0.4--3.3 3.3--6.9 7--11 11--14 Enabled − disabled latency range (s) −6 −4 −2 0 2 4 Gemini Flash — latency 70--469 469--869 869--1,268 1,268--1,6671,667--2,0672,067--2,4662,466--2,8662,866--3,2653,265--3,6653,665--4,064 Enabled − disabled token range −10 −5 0 5 10 Expert Accuracy change (p) Gemma4 — tokens 0--27 27--5454--81 81--109 109--136136--163163--190190--217217--244244--271 Enabled − disabled latency range (s) −10 −5 0 5 10 Gemma4 — latency 240--402402--563563--725725--886 886--1,048 1,048--1,2101,210--1,3711,371--1,5331,533--1,6941,694--1,856 Enabled − disabled token range −6 −4 −2 0 2 4 6 Expert Accuracy change (p) Gemini Flash — tokens 1.6--3.03.0--4.44.4--5.85.8--7.37.3--8.7 9--10 10--1212--1313--1414--16 Enabled − disabled latency range (s) −6 −4 −2 0 2 4 6 Gemini Flash — latency Thinking-toggle accuracy changes by signed resource bucket: Original, Novice, Expert phrasing Color encodes reasoning condition; marker shape encodes target; point size encodes difficulty. Reasoning condition DirectNL CoTSQL CoTPython CoTPseudocode CoT Target / difficulty SQLPYTHONSimpleModerateChallenging Figure 7: Internal-reasoning configuration effects in signed token and latency buckets for Original, Novice, and Expert phrasing. Color encodes the reasoning condition, marker shape target, and marker size difficulty. 21 NL CoT SQL CoT Python CoT Pseudocode CoT Length fi ller Scrambled Wrong content GLM4 Gemma4-off Gemma4-on Qwen3 Gemini Flash-0 Gemini Flash-24k 99.599.999.599.488.59.227.1 99.299.899.699.395.227.571.1 95.794.994.694.483.015.559.8 99.999.897.499.922.425.432.7 99.999.999.699.917.35.021.2 98.798.898.798.740.710.220.3 Visible exact-eight-step compliance (%) 20 40 60 80 Contract compliance (%) Figure 8: Exact eight-step contract compliance for explicit-CoT and control conditions by model. Low compliance, especially under scrambled context, argues against interpreting lower completion length as evidence that the control was simply ignored. 22 Term familyp BH Persona< 10 −6 Target< 10 −26 Model configuration< 10 −11 Mode0.620 Mode × persona0.00033 Mode × target0.626 Model configuration × mode0.0030 Difficulty0.179 SCS0.362 Ambiguity0.0016 Phrasing specificity0.0017 Table 8: Complementary robust-GEE omnibus tests. Paired exact contrasts remain the primary inference. 23 PhrasingTargetConditionDirectCondition∆ p BH OriginalSQLPseudocode CoT33.3935.311.920.168 OriginalSQLNL CoT33.3935.141.750.168 OriginalSQLScrambled33.3935.361.970.196 NoviceSQLScrambled27.4732.625.150.00052 NovicePythonScrambled18.8621.112.250.111 NoviceSQLNL CoT27.4728.841.370.439 Table 9: Top pooled condition-versus-direct results in the focused Original and Novice families. Pooled contrasts average paired deltas across the six configurations within each query before testing. PhrasingTargetContrast∆ p95% CIRaw p BH NoviceSQLScrambled − direct5.15[2.80, 7.62]3.72× 10 −5 0.00045 NoviceSQLScrambled − length filler5.04[3.07, 7.07]9.14× 10 −7 0.000011 NoviceSQLScrambled − wrong content3.45[1.54, 5.48]0.000630.0038 NoviceSQLScrambled − explicit-CoT mean4.39[2.43, 6.41]1.61× 10 −5 0.00019 NovicePythonScrambled − length filler2.47[0.77, 4.22]0.00460.0184 NovicePythonScrambled − wrong content2.58[0.77, 4.50]0.00830.0250 Table 10: Corrected focused control contrasts. BH is applied within three separate 12-test families: controls versus direct, between controls, and controls versus the four-format explicit-CoT mean. PersonaDifficulty∆ vs NL CoT q BH ∆ vs direct q BH OriginalSimple0.001.00001.850.1646 OriginalModerate-0.620.80760.230.9469 OriginalChallenging1.310.45392.340.1646 NoviceSimple2.850.01834.010.0026 NoviceModerate1.170.45393.040.0183 NoviceChallenging3.840.00774.120.0034 Table 11: Scrambled-minus-comparator accuracy contrasts by persona and difficulty, pooled across the six configu- rations and both targets. Deltas are percentage points; q BH values use the focused difficulty family. Reasoning conditionAccuracy (%)Mean tokensMean latency (s)∆ ppExtra tokensExtra latency (s) Direct42.942,15337.10.0000.0 NL CoT43.413,09569.30.4894232.2 SQL CoT43.523,07775.40.5892438.4 Python CoT42.823,06976.0-0.1291638.9 Pseudocode CoT43.513,19481.30.581,04144.2 Table 12: Pooled cost and accuracy for each explicit CoT format versus direct generation over the six configurations, three personas, and two targets. Bold marks the highest accuracy and the smallest total-token and latency values. Condition familyAccuracy (%)TokensLatency (s)∆ p Extra tokens Extra latency (s) Dominated (T/L, %) Direct42.942,15337.10.0000.00.0 / 0.0 Explicit CoT mean43.323,10975.50.3895638.441.7 / 38.9 Table 13: Pooled descriptive efficiency summary over six configurations, three personas, and two targets. Direct- dominated percentages count matched cells where direct generation is at least as accurate and no more costly, with one strict advantage. 24 DifficultyCondition familyAccuracy (%)Mean tokensMean latency (s)∆ p vs direct SimpleDirect47.251,94331.90.00 SimpleExplicit CoT mean47.692,83461.80.44 ModerateDirect42.522,12834.20.00 ModerateExplicit CoT mean43.123,07973.10.60 ChallengingDirect38.202,43946.90.00 ChallengingExplicit CoT mean38.253,47895.00.05 Table 14: Difficulty-stratified efficiency means pooled across model, persona, and target cells. Difficulty raises both baseline and prompted resource use; the table should not be interpreted as a causal difficulty effect. From simple to challenging, direct generation adds approximately 496 tokens and 15.0 seconds, while explicit CoT adds 644 tokens and 33.2 seconds. 25 Model / CoTAccuracyTokensLatency (s) GLM4 direct36.731,53720.5 GLM4 NL CoT37.772,22263.8 Qwen3 direct40.131,54913.6 Gemma4-off direct43.701,87328.8 Gemma4-on direct46.053,743144.7 Gemma4-on SQL CoT47.156,056318.3 Gemini Flash-0 direct45.181,7425.4 Gemini Flash-24k direct45.832,4759.5 Table 15: Selected pooled efficiency points. Accuracy is percent; latency is locally measured client-observed wall- clock time and is not hardware-normalized. ControlCorrectness ORq BH Length filler1.3250.00048 Scrambled0.9950.94542 Wrong content1.2630.00264 Table 16: Adjusted association between exact eight-step compliance and correctness within each control. Odds ra- tios are from query-clustered logistic GEE; BH correction is across the three control-specific tests. 26