Paper deep dive

SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

Mohsen Larni, Sobhan Ebrahimi Azar, Pouyan Nahed, Kazem Taghva

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 94

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Large language models are increasingly used where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through isolated probes and aggregate accuracy. We introduce SyntaxBench, a diagnostic benchmark and statistical evaluation framework for character-level reasoning. It contains five core tasks, character counting, letter containment, palindrome detection, edit distance, and longest-string selection, plus index_to_span, a harder substring-extraction stress test. The five core tasks use paired English and character-length-matched random-string inputs. index_to_span documents share a 200-500 word band and are not character-length matched. All six tasks use zero-, one-, and four-shot prompts. We evaluate eight open-weight models from 2B to 32B parameters across 11 reasoning-mode configurations. The framework reports exact-match and relaxed accuracy, Cohen's kappa, paired McNemar tests with odds ratios, bootstrap confidence intervals, Kendall's tau, class-conditional metrics, tokenization analysis, and multiple-comparison-corrected tests. Three findings stand out. First, tokenization shapes accuracy: random strings are more character-visible than English strings (1.892 vs. 3.169 characters per token), and character-counting accuracy falls as English words occupy more tokens. Second, reasoning mode is not uniformly helpful: Gemma4-31B is nearly unchanged across modes on the near-saturated tasks, while Qwen3.6-27B is worse with thinking on palindrome detection (0.952 non-thinking vs. 0.886 thinking at four-shot). Third, index_to_span remains largely unsolved; the best four-shot exact-match accuracy is 6.75%. Character-level evaluation needs controlled inputs, paired tests, and analyses of tokenization and reasoning mode rather than aggregate accuracy alone.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

90,977 characters extracted from source content.

Expand or collapse full text

SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models Mohsen Larni ∗ Department of Computer Science University of Nevada, Las Vegas Sobhan Ebrahimi Azar ∗ Department of Computer Science University of Nevada, Las Vegas Pouyan Nahed Department of Computer Science University of Nevada, Las Vegas Kazem Taghva Department of Computer Science University of Nevada, Las Vegas Abstract Large language models are increasingly used in settings where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through iso- lated probes and aggregate accuracy. We introduce SyntaxBench, a diagnostic benchmark and statistical evaluation framework for character-level reasoning. Syn- taxBench contains five core deterministic tasks, character counting, letter con- tainment, palindrome detection, edit distance, and longest-string selection, plus index_to_span, a harder substring-extraction stress test. The five core tasks are instantiated on paired English and character-length-matched random-string inputs. index_to_spandocuments are drawn from the same 200–500 word band and are not character-length matched. All six tasks use zero-, one-, and four-shot prompts. We evaluate eight open-weight models spanning 2B–32B parameters across 11 available reasoning-mode configurations. The framework reports exact-match and relaxed accuracy together with Cohen’sκ, paired McNemar tests with odds-ratio effect sizes, bootstrap confidence intervals, Kendall’sτover shot count, class- conditional metrics, tokenization analysis, and multiple-comparison-corrected condition tests. Three findings stand out. First, tokenization strongly shapes accuracy: random strings are more character-visible than English strings (1.892 vs. 3.169 chars/token), andcount_charsaccuracy falls sharply as English words occupy more tokens. Second, reasoning mode is not uniformly helpful: Gemma4- 31B is essentially indistinguishable across modes on the near-saturated tasks, while Qwen3.6-27B is worse with thinking enabled onpalindrome_test(0.952 non- thinking vs. 0.886 thinking at four-shot). Third,index_to_spanremains largely unsolved; the best four-shot exact-match accuracy is 6.75%. SyntaxBench shows that character-level evaluation needs controlled input condi- tions, paired statistical tests, and analyses of tokenization and reasoning mode rather than aggregate accuracy alone. The code will be released soon. Keywords: Large Language Models, Character-Level Reasoning, Tokenization, Chain-of-Thought, Diagnostic Benchmark, Statistical Evaluation ∗ These authors contributed equally. Preprint. arXiv:2610.03329v1 [cs.CL] 2 Oct 2026 1 Introduction Large language models (LLMs) are now used as everyday tools for writing, programming, tutoring, and analysis. In these settings, users often expect exact answers to small syntactic questions: How many characters are in a string? Does a letter appear? What is the edit distance between two strings? These operations are simple for deterministic programs, but they remain brittle for LLMs. The familiar “strawberry” counting failure [28,1] is a visible instance of a broader problem: fluent text generation does not imply reliable character-level reasoning. Character-level tasks are useful because they have deterministic ground truth. They also expose failure modes that semantic benchmarks can hide. A model can sound confident while being off by one character, returning the wrong binary label, or extracting the wrong substring from a long context. Standard benchmarks [32,31,29,15] measure broad language understanding, but they rarely isolate these low-level operations. When character-level probes are used, results are often summarized with aggregate accuracy alone, without paired tests, effect sizes, class-conditional behavior, or controls for tokenization and input distribution. We introduce SyntaxBench, a diagnostic benchmark and evaluation framework for this setting. The benchmark has six deterministic tasks. Five are core character-level tasks:count_chars, letter_container,palindrome_test,edit_distance, andlongest_string_finder. The sixth,index_to_span, is a harder stress test that asks models to recover a substring from explicit start and end indices in a long text. The five core tasks are instantiated on paired English and length- matched random-string inputs, allowing us to separate linguistic familiarity from character-level processing. The stress-test documents are drawn from the same 200–500 word band and are not character-length matched. We evaluate zero-, one-, and four-shot prompting, and compare thinking and non-thinking modes where both are available. SyntaxBench is organized around four research questions. RQ1 (mechanism): is the English- vs-random accuracy difference driven by tokenizer granularity (characters per token) rather than linguistic familiarity? RQ2 (reasoning mode): treated as a deployment switch, when does enabling thinking help, hurt, or leave character-level accuracy unchanged? RQ3 (short-to-long transfer): does competence on short-input character operations transfer to character-offset extraction over long documents? RQ4 (methodology): what statistical machinery is needed to make character-level benchmark claims falsifiable rather than anecdotal? The framework is designed for benchmark interpretation, not just score reporting. In addition to exact-match and relaxed accuracy, it reports bootstrap confidence intervals, Cohen’sκ, paired McNemar tests with odds-ratio effect sizes, Kendall’sτover shot count, class-conditional binary metrics, tokenization-stratified accuracy, integer and string error summaries, and multiple-comparison- corrected English-vs-random tests. These diagnostics make each headline claim falsifiable: they distinguish model differences from sample noise, input-condition effects from tokenization artifacts, and accuracy gains from validity or class-bias effects. We evaluate eight open-weight models spanning 2B–32B parameters: the Qwen3.5 family at 2B, 9B, and 27B; Qwen3.6-27B; Gemma4-31B; GPT-OSS-20B; Nemotron-Cascade-2; and DeepSeek-R1-32B. The resulting 11 available model-mode configurations show that character- level reasoning is not a single capability. Several models saturateletter_containerand per- form well onpalindrome_test, whileedit_distanceremains strongly model-dependent and index_to_spanis near the floor for every model. At four-shot in thinking mode, GPT-OSS-20B has the best composite score (macro EMA 0.818), but even it reaches only 6.75% exact match on index_to_span. Three empirical patterns motivate the benchmark. First, tokenization mediates English-vs-random behavior. Across tokenizer-covered models, random strings are more character-visible than English strings: 0.542 versus 0.344 tokens per character, or 1.892 versus 3.169 characters per token (Cohen’s d = 2.165on tokens per character). Oncount_chars, accuracy declines as English examples occupy more tokens, consistent with errors inside multi-character subwords. Second, reasoning mode is not uniformly helpful. DeepSeek-R1-32B benefits from thinking on several tasks, but Qwen3.6- 27B is worse with thinking enabled onpalindrome_test(0.952 non-thinking vs. 0.886 thinking at four-shot). Third, scale helps but does not solve the benchmark. Within Qwen3.5, four-shot count_charsimproves from 0.698 to 0.880 to 0.958 across 2B, 9B, and 27B, yet the complete 9B 2 and 27Bindex_to_spancells remain at0.5%, and the 2B 4-shot cell has no random-condition responses (0/113 English). Contributions. We make three contributions. (1) SyntaxBench: a six-task diagnostic benchmark for character-level reasoning with paired English and random inputs. The five core tasks are character- length matched. The stress test matches word count, not character length. (2) A reproducible evaluation framework: a pipeline that reports accuracy, validity, agreement, effect sizes, monotonicity, tokenization effects, class-conditional behavior, and multiple-comparison-corrected statistical tests. (3) An empirical study of open-weight LLMs: an analysis showing tokenization-mediated input effects, non-uniform reasoning-mode behavior, within-family scaling, and a persistent failure mode on positional substring extraction. The code will be released soon. 2 Related Work General-purpose LLM benchmarks. GLUE [32], SuperGLUE [31], BIG-Bench [29], MMLU [12], and HELM [15] established standardized evaluation suites for natural language un- derstanding and broad reasoning. These benchmarks emphasize semantic capabilities and probe character-level operations only incidentally. SyntaxBench addresses this gap with deterministic ground truth, balanced positive/negative cases per task, and an evaluation framework that pairs accu- racy with statistical machinery (paired McNemar, Cohen’s κ, Kendall’s τ , tokenizer stratification). Character-level reasoning, tokenization, and the “strawberry” problem.Subword tokenization decouples a model’s input representation from individual characters, with measurable consequences for orthographic tasks. Tokenization-free architectures such as ByT5 [35] and CANINE [5] demon- strate improved robustness on character-level tasks but are not the dominant deployment paradigm. Recent work [28,1] has shown that even frontier models miscount characters in common words. Our results in §4.2 mechanistically link this failure mode to per-token chars-per-token packing rather than to lexical familiarity, showing that for tasks requiring per-character enumeration the gap is in the direction of tokenizer-induced opacity on linguistic input, not unfamiliarity with random characters. In-context learning and shot-mode effects.Brown et al. [3] introduced few-shot in-context learn- ing; chain-of-thought prompting [34] extended this to multi-step reasoning. Subsequent work has documented that few-shot performance is non-monotonic on certain tasks [17,4] and that demonstra- tion selection [16] substantially affects outcomes. SyntaxBench operationalizes monotonicity testing via Kendall’sτover shot index per (model, mode, task, condition) and demonstrates that the same model on the same task can have sign-flipped trajectories between thinking and non-thinking modes (§4.4). Reasoning mode and overthinking.The reasoning-mode toggle has become standard in mid-sized open-weight LLMs (Qwen3.6, Gemma4, GPT-OSS, DeepSeek-R1). The overthinking literature [23] has documented that chain-of-thought traces can degrade performance on tasks where the underlying decision is direct. Our results in §4.3 extend this to character-level operations and provide a paired- McNemar test on the same model, showing the regime is not uniform: thinking can help, hurt, or be a no-op depending on (model, task, shot). Algorithmic and symbolic reasoning evaluations.Scratchpads [23], MATH [13], and GSM8K [6] target multi-step numeric reasoning rather than primitive string operations. SyntaxBench targets the lower stratum: deterministic operations whose ground truth is computable in a single pass, with partial- correctness metrics that decouple “approximately right” from “exactly right”. Theindex_to_span stress test (§4.5) provides a long-horizon counter task that the core suite intentionally avoids. Statistical methodology in LLM benchmarking. A recurring methodological concern is the reliance on point-estimate accuracy without effect sizes, confidence intervals, or correction for multiple comparisons [30,20]. Established tools, McNemar’s test [19,9] for paired classifier comparisons, Cohen’sκ[7] for class-imbalance-aware agreement, Benjamini-Hochberg FDR [2] for power-aware multiple correction, and lm-evaluation-harness [10] for standardized protocols, are rarely combined in a single benchmark release. SyntaxBench couples its dataset with a reproducible statistical framework that applies these tools together to surface findings invisible in accuracy alone. 3 Table 1: SyntaxBench tasks. The Inputs column reports per-condition sample counts (English / random). TaskDescriptionOutput Inputs (En / Rand) Intra-word core count_charslength of a string sinteger200 / 200 letter_containerdoes s contain character c?yes/no400 / 400 palindrome_testdoes s read the same forward and backward?yes/no400 / 400 Inter-word core edit_distanceLevenshtein distance between (s 1 ,s 2 )integer200 / 200 longest_string_finder longest of≥ 3 stringsstring200 / 200 Stress test index_to_spansubstring d[i : j] in long textstring200 / 200 Linguistic priors and generalization.Several studies show LMs exploit distributional properties of natural language at the expense of robust symbolic reasoning [18,21]. The SyntaxBench length- matched random-string corpus for the five core tasks tests the “LLMs degrade on non-natural inputs” hypothesis directly: if degradation were universal, we would expect significant English-vs-random gaps across all tasks. Our chi-square analysis (§4.2) refines this picture: of the 25 Bonferroni- significant cells, 23 go the other direction (random easier) and 2 do not, and the mechanism on the majority is tokenizer granularity, not familiarity. 3 Methodology 3.1 Benchmark Overview SyntaxBench evaluates whether LLMs can perform deterministic syntactic operations under strict output constraints. The benchmark consists of six tasks (Table 1): a five-task core suite with interpretable outputs and well-defined partial-credit metrics, plus one stress test (index_to_span) that locates a target substring at specified character offsets within a 200–500 word document. Each task has deterministic ground truth and is instantiated under three shot modes (zero-, one-, four-shot) and, where supported, two thinking modes (thinking, non-thinking), giving up to six (shot, thinking- mode) configurations per (model, task). Per-condition sample counts and prompt templates are in Appendix A.2. 3.2 Data Sources and Random-String Construction The English vocabulary is drawn from the NLTK corpus (236,736 entries).palindrome_test additionally uses a curated 229-entry palindrome word list from Wiktionary, andindex_to_span uses Wikipedia article excerpts. For the five core tasks, the length-matched random-string corpus matches the empirical character-length distribution of the NLTK vocabulary: for each English word of lengthℓ, a random string of the same length is drawn uniformly from[a-z]. This controls for the trivial confound that random strings might differ in length from English words, contaminating the English-vs-random comparison via length effects.index_to_spanis the exception at the character level. Both conditions are targeted to 200–500 words (observed means 321 English and 336 random). English documents are Wikipedia excerpts. Random documents are assembled by sampling that many strings from the random vocabulary, not by replacing each English word in an excerpt. Those strings follow the NLTK word-list lengths rather than the short function words of Wikipedia prose, so mean document length is 3,593 characters on the random condition and 2,128 on the English condition. 3.3 Models and Thinking Modes We evaluate eight open-weight LLMs spanning a 16×parameter range across eleven (model, thinking-mode) configurations: Qwen3.5-2B / 9B / 27B [26] (non-thinking only); Qwen3.6- 27B [27] (both modes, vLLM at FP8); Gemma4-31B-it [11] (both modes, vLLM at BF16, TP=2); GPT-OSS-20B [24] (thinking, Ollama at MXFP4, the only stable backend at evaluation time); Nemotron-Cascade-2-31B [22] (thinking,llama.cppwith split GGUF); and DeepSeek-R1-32B [8] 4 (thinking-by-design; its “non-thinking” configuration disables reasoning-trace accounting rather than reasoning itself, and is reported separately). Two models (GPT-OSS-20B and Nemotron-Cascade- 2) are therefore evaluated in thinking mode only. The Qwen3.5 family at three scales gives a within-family scaling curve at fixed architecture; Qwen3.6-27B and Gemma4-31B in both modes each provide a within-model reasoning-mode ablation at fixed scale (the two models are distinct architectures). Per-(model, mode) backend tuning ofseq_lenandnum_seqis recorded in the run manifest and reproduced in Appendix A.5. 3.4 Evaluation Protocol Each (model, thinking-mode) is evaluated under 0-, 1-, and 4-shot prompting with deterministic decoding (temperature 0). The system prompt is task-specific; demonstrations (when present) are alternating user–assistant message pairs preceding the test query. Each test instance is an independent inference request reusing the system prompt and demonstrations but no prior model outputs. Models return a JSON object with a singleresultfield; the response parser handles whitespace and JSON- formatting variability, including a Markdown-bold (**...**) fallback for one model whose chat template wraps responses in bold markers. Reasoning traces (when emitted in a separate field) are captured for diagnostics but excluded from scoring. 3.5 Scoring and Metrics Primary metrics per (model, task, shot, thinking-mode, condition) cell: Exact-Match Accuracy (EMA, parse errors counted as incorrect); Relaxed Accuracy (RA,±1tolerance, integer tasks); Parse Error Rate (PER); Cohen’s κ [7] (unweighted for binary tasks, linearly-weighted for integer tasks). Statistical inference. (i) bootstrap 95% CIs on per-task EMA via 10,000 resamples of samples; the composite macro-EMA interval instead resamples the six task-level EMAs (Appendix A.6); (i) paired McNemar [19] for between-model and thinking-vs-non-thinking comparisons, with continuity correction, discordant counts(b 01 ,b 10 ), and odds ratio as effect size; (i) chi-square tests of indepen- dence [25] for English-vs-random per (model, task, shot, mode), with Cramér’sV; (iv) Kendall’s τ[14] between shot index and EMA. Multiple-comparison correction uses Bonferroni for headline tables and Benjamini–Hochberg FDR [2] for the supplementary q-values, both at α = 0.05. Tokenizer-stratified analysis. For each model with a public HuggingFace tokenizer, we tokenize the raw input content and bin samples by token count, reporting mean chars-per-token by condition, Cohen’sdbetween English and random tokens-per-character distributions, and per-bucket accuracy. Method details and per-model tokenizer assignments are in Appendix A.7. 4 Experiments and Results 4.1 Setup and Main Results We evaluate the eight models from §3.3 on all six SyntaxBench tasks under 0-, 1-, and 4-shot prompt- ing. The 11 (model, thinking-mode) configurations are those in §3.3. Inference uses deterministic decoding (temperature 0); per-(model, mode) backend tuning is in Appendix A.5. Figure 1 reports per-shot EMA on combined English+random inputs across all 11 configura- tions. Three patterns dominate the surface view: (i) oncount_chars,letter_container, longest_string_finder, andpalindrome_test, 25 of 32 4-shot cells for 27B+ configura- tions are at least 90% EMA; the exceptions include Nemotron-Cascade-2 at 69.0% combined on longest_string_finderand DeepSeek-R1-32B non-thinking at 43.25% combined on that task; (i)edit_distancediscriminates strongly, with GPT-OSS-20B at89.0%versus0.0–43.0%for the rest (the top of that range is Qwen3.5-27B non-thinking; DeepSeek-R1-32B non-thinking is 0.0%); (i)index_to_spanis unsolved, with max 4-shot EMA of6.75%(GPT-OSS-20B). Table 2 summarizes the 4-shot composite leaderboard; the per-task 4-shot table with 95% bootstrap CIs and Bonferroni-corrected significance is in Appendix B, Table 4. The wide CIs are diagnostic: macro EMA hides the floor effect onindex_to_span, the discrimina- tion gap onedit_distance, and the saturated near-ceiling behavior on the other four tasks. The next 5 counteditspancontainslongestpalin Task DeepSeek32 [NT] DeepSeek32 [T] GPT-OSS20 [T] Gemma31 [NT] Gemma31 [T] Nemotron [T] Qwen3.5-2 [NT] Qwen3.5-27 [NT] Qwen3.5-9 [NT] Qwen3.6-27 [NT] Qwen3.6-27 [T] Model 0.970.070.000.990.960.94 0.980.400.010.991.000.95 0.960.850.070.991.000.99 0.880.290.010.990.960.84 0.880.290.010.990.960.84 0.960.230.001.000.570.82 0.480.120.000.700.480.79 0.950.420.010.980.940.88 0.820.320.000.880.780.83 0.950.380.000.960.950.92 0.990.370.001.001.000.95 0-shot counteditspancontainslongestpalin Task 0.910.140.000.990.670.94 0.940.340.000.990.990.94 0.970.890.070.991.000.99 0.920.300.010.990.980.93 0.920.290.010.990.980.93 0.960.160.000.990.730.87 0.640.180.000.780.490.75 0.960.430.010.980.930.89 0.880.310.000.940.840.84 0.970.380.000.970.960.90 1.000.360.001.000.990.82 1-shot count edit span contains longest palin Task 0.800.000.000.980.430.88 0.960.380.000.991.000.85 0.970.890.071.000.980.99 0.960.380.010.990.980.93 0.960.400.010.990.980.93 0.970.340.001.000.690.88 0.700.260.830.480.83 0.960.430.010.990.960.93 0.880.400.010.960.820.94 0.980.330.000.980.970.95 1.000.300.000.990.960.89 4-shot 0.0 0.2 0.4 0.6 0.8 1.0 EMA_all EMA_all main heatmap (condition=all) Figure 1: Main results. Per-(model, mode) EMA on combined English+random inputs at 0-, 1-, and 4- shot. At 4-shot, 25 of 32 cells oncount_chars,letter_container,longest_string_finder, andpalindrome_testfor 27B+ configurations reach 90% EMA. Nemotron-Cascade-2 is at 69.0% onlongest_string_finder, and DeepSeek-R1-32B non-thinking is at 43.25% on that task. edit_distancediscriminates models, andindex_to_spanremains near the floor. Empty cells are configurations that were not run, or a combined score that is not estimable because one input condition is missing. Table 2: Composite leaderboard at 4-shot (thinking, condition=all). Macro EMA averages over the six tasks. The interval resamples those six task-level EMAs; it is not the sample bootstrap used for per-task EMA. VRR = valid-response rate; PER = parse-error rate. Qwen3.5 family entries (non-thinking-only) appear in Appendix B.1, Table 18. RankModelMacro EMA [95% CI]VRRPER 1GPT-OSS-20B0.818 [0.512, 0.989]0.9640.036 2Gemma4-31B0.712 [0.388, 0.971]1.0000.000 3DeepSeek-R1-32B0.697 [0.385, 0.966]0.9730.027 4Qwen3.6-27B0.690 [0.358, 0.972]1.0000.000 5Nemotron-Cascade-2-31B0.648 [0.339, 0.919]0.9920.008 four subsections break the surface view apart along the three abstract findings (tokenization, reasoning mode, and theindex_to_spanfloor) and a fourth subsection that demonstrates the diagnostic value of the framework itself. 4.2 Headline 1: Tokenization Mediates the English-vs-Random Gap The natural reading of LLM degradation on non-linguistic input is lexical familiarity: a model has not seenxqfbbut has seencat. We test this directly via chi-square independence tests on each (model, task, shot, thinking-mode) cell. After Bonferroni correction across the 197 comparisons in which both conditions were evaluated (11 configurations×6 tasks×3 shots, excluding the incomplete Qwen3.5-2Bindex_to_span4-shot cell), 25 (12.7%) show a significant English-vs-random gap. Random inputs are easier in 23 of those cells. English is easier in the other two: DeepSeek-R1-32B on longest_string_finderat 4-shot non-thinking, and Gemma4-31B onedit_distanceat 0-shot thinking. The mechanism on the random-easier majority is tokenization, not familiarity. Applying each model’s HuggingFace tokenizer to raw input strings on the five core tasks, and excluding the single-character letter query, English packs3.169chars/token while random packs 1.892chars/token, a1.67×ratio. On tokens per character the same samples give0.344versus0.542, with Cohen’sd = 2.165. Stratifying 0-shot accuracy by input token count (Figure 2),count_chars English accuracy falls from97.7%to46.2%across the 1-token to 6+ bins in non-thinking mode, and from100%to75%in thinking mode. Random inputs remain higher than English in every shared bin. Per-example failure analysis confirms the mechanism: nearly every off-by-one English failure on 6 Figure 2: 0-shotcount_charsaccuracy by tokenizer-token count. The figure shows this task only. English accuracy falls from97.7%to46.2%across the 1-token to 6+ bins in non-thinking mode, and from100%to75%in thinking mode. Random inputs remain higher than English in every shared bin. Table 3: Qwen3.6-27B onpalindrome_test, EMA and Cohen’sκon combined inputs. Non- thinking outperforms thinking at 1-shot and 4-shot, with the gap concentrated in the positive class (at 4-shot, TPR drops from 0.9075 to 0.7725 and TNR goes from 0.9975 to 1.0000). ShotNon-thinkingThinkingNT− T EMA κEMA κ∆EMA∆κ 0-shot0.9190.8380.9550.910−0.036−0.072 1-shot0.8990.7980.8190.638+0.080+0.160 4-shot0.9520.9050.8860.773+0.066+0.132 count_charslocalizes to multi-character subwords. Gemma4-31B miscountsundeclared(true 10) as 11 with tokenizationunde|cl|ared;osphresiology(true 13) as 14 withosph|res|iology; verisimilarly(true 13) as 14 withver|isim|ilarly. The non-thinking 1-token English bin is correct on 43 of 44 items (97.7%). getter and terminal are examples from that bin. The implication for benchmark design is sharp: the standard “LLMs fail on non-linguistic input” framing conflates lexical familiarity and tokenizer granularity. Tokenizer-stratified evaluation disen- tangles them, and on character-level tasks the gap is in the direction of tokenizer-induced opacity on linguistic input, not unfamiliarity with random characters. The tokenizer-stratified method is in Appendix A.7. Per-bucket accuracies and per-tokenizer statistics are in the released tokenization CSVs, not tabulated in the appendix. 4.3 Headline 2: Reasoning Mode Helps Unevenly and Sometimes Hurts For Qwen3.6-27B and Gemma4-31B we evaluate both thinking and non-thinking configurations of the same model, and DeepSeek-R1-32B is thinking-by-design. The naive prediction is that thinking dominates non-thinking. Our data refute this for character-level tasks. Three concrete patterns emerge. (1) Thinking can hurt at fixed scale. Table 3 shows Qwen3.6-27B’s non-thinking variant outperforming thinking at 1-shot (0.899vs.0.819) and 4-shot (0.952vs.0.886); McNemar’s test on the 4-shot combined cell is significant after Bonferroni correction (odds ratio 27.5). Theκdrop (0.905→ 0.773) is consistent with the EMA drop and rules out a parsing artifact (PER = 0). (2) Thinking can be a no-op. Figure 3 shows Gemma4-31B’s thinking and non-thinking 7 0.40.30.20.10.0 accuracy (non_thinking thinking) DeepSeek32 count DeepSeek32 edit DeepSeek32 span DeepSeek32 contains DeepSeek32 longest DeepSeek32 palin Gemma31 count Gemma31 edit Gemma31 span Gemma31 contains Gemma31 longest Gemma31 palin Qwen3.6-27 count Qwen3.6-27 edit Qwen3.6-27 span Qwen3.6-27 contains Qwen3.6-27 longest Qwen3.6-27 palin 0-shot (paired samples) 0.30.20.10.00.1 accuracy (non_thinking thinking) DeepSeek32 count DeepSeek32 edit DeepSeek32 span DeepSeek32 contains DeepSeek32 longest DeepSeek32 palin Gemma31 count Gemma31 edit Gemma31 span Gemma31 contains Gemma31 longest Gemma31 palin Qwen3.6-27 count Qwen3.6-27 edit Qwen3.6-27 span Qwen3.6-27 contains Qwen3.6-27 longest Qwen3.6-27 palin 1-shot (paired samples) 0.60.50.40.30.20.10.00.1 accuracy (non_thinking thinking) DeepSeek32 count DeepSeek32 edit DeepSeek32 span DeepSeek32 contains DeepSeek32 longest DeepSeek32 palin Gemma31 count Gemma31 edit Gemma31 span Gemma31 contains Gemma31 longest Gemma31 palin Qwen3.6-27 count Qwen3.6-27 edit Qwen3.6-27 span Qwen3.6-27 contains Qwen3.6-27 longest Qwen3.6-27 palin 4-shot (paired samples) Thinking vs non-thinking: paired accuracy delta with bootstrap 95% CI Figure 3: Thinking-mode accuracy delta (EMA non-think − EMA think , per task, at 0-, 1-, and 4-shot) with 95% bootstrap CIs. Bars to the right of zero indicate thinking hurts; bars to the left indicate it helps. Qwen3.6-27B is hurt onpalindrome_test; Gemma4-31B is essentially indistinguishable across modes for character-level tasks at this scale. results are essentially identical across the four near-saturated tasks at this scale, suggesting that for those character-level operations the model’s reasoning trace adds no decision-relevant information. (3) Thinking can help when reasoning is explicitly trained for. DeepSeek-R1-32B (thinking-by-design) achieves99.8%onlongest_string_finderand99.4%onletter_containerat 4-shot. The pattern is consistent across English and random conditions, ruling out linguistic-prior explanations. We interpret it as reasoning-mode interference: chain-of-thought traces inject surface-format variance that on already-saturated tasks distorts more than it helps. Token-cost and latency penalties of thinking mode are not always offset by accuracy gains for character-level operations; the per-task thinking-cost table (mean tokens, EMA per token) is in Appendix B, Table 15. 4.4 Diagnostic Depth: What Aggregate Accuracy Misses The framework reports class-conditional metrics, monotonicity over shot count, and effect sizes alongside accuracy. We illustrate two cases where these surface findings that aggregate accuracy hides. Class-conditional asymmetry on binary tasks. Qwen3.5-2B on 0-shotpalindrome_test(En- glish) achieves0.430EMA on the positive class (true palindromes) and0.965on the negative class. The model is systematically biased toward “no”, and aggregate0.6975EMA reads like a noisy classifier. The same disaggregation flags Gemma4-31B’s 0-shotpalindrome_test(English) at TPR = 0.685vs.TNR = 1.000, also a non-trivial under-prediction of the positive class. The examples above are the class-conditional cases reported in the text. A full TPR/TNR/MCC table is not included in the appendix; those per-cell metrics are in the released evaluation CSVs. Non-monotone response to demonstrations.Conventional intuition predictsmonotone_up: more examples should help. Kendall’sτon shot index versus EMA per (model, task, mode, condition) shows this fails systematically for already-saturated tasks. Qwen3.6-27B onpalindrome_test isvalley_1shotin both modes (Table 3): non-thinking0.919 → 0.899 → 0.952and thinking 0.955 → 0.819 → 0.886. Kendall’sτon those combined trajectories is+0.33and−0.33. The 1-shot dip is shared; thinking is the mode that stays below its 0-shot accuracy. We propose that monotonicity testing across0, 1, 4shots becomes a standard diagnostic. The Kendall-τsparkline grid for all (model, mode, task, condition) cells is in Appendix B, Figure 7. 4.5 Floor: Positional Substring Extraction The five core tasks span saturated to challenging-but-tractable. To probe what current architectures cannot do, we include a sixth task that requires extracting the substring located at a specified character offset within a document of 200–500 words. English items are Wikipedia excerpts (mean 2,128 characters). Random items are documents sampled from the random vocabulary (mean 8 3,593 characters): about as many words, but longer words. Across all 11 (model, thinking-mode) configurations and three shot modes, max 4-shot EMA is6.75%(GPT-OSS-20B). Gemma4-31B and DeepSeek-R1-32B reach0.5%and0.25%. Qwen3.5-9B and Qwen3.5-27B reach0.5%. The Qwen3.5-2B 4-shot cell is incomplete: the observed English subset is 0/113, and no random- condition estimate is available. The remaining complete configurations are at exact0%. Scale alone is insufficient. The failure is computational: maintaining a character-position counter across a long document while extracting span contents is beyond every model evaluated. We release this task as a long-horizon discriminator that future architectures can target as the core tasks saturate. 4.6 Pairwise Model Comparisons McNemar’s test with continuity correction is computed separately within each thinking mode on the 18 combined-condition cells (6 tasks×3 shots). Tables 8 and 9 report, for each pair, how many of those cells remain significant after Bonferroni correction across the pair’s 18 cells. These tables are not pooled into one ordering: Qwen3.5 was evaluated only without thinking, while GPT-OSS-20B and Nemotron-Cascade-2 were evaluated only with thinking, so several cross-family pairs have no shared-mode test. In thinking mode, GPT-OSS-20B wins 9 and loses 0 of 18 cells against DeepSeek-R1-32B, wins 12 and loses 0 against Gemma4-31B, wins 12 and loses 0 against Nemotron-Cascade-2, and wins 9 and loses 1 against Qwen3.6-27B. Qwen3.6-27B wins 5, loses 3, and is nonsignificant on 10 cells against Gemma4-31B; it wins 8 and loses 0 against Nemotron. DeepSeek-R1-32B and Qwen3.6-27B each win 2 of their 4 significant cells. In non-thinking mode, Qwen3.5-27B wins 12 and loses 0 against Qwen3.5-9B, and wins 15 and loses 0 against Qwen3.5-2B; Qwen3.5-9B wins 14 and loses 0 against Qwen3.5-2B. Qwen3.5-27B and Qwen3.6-27B are significant on only 3 cells (2 wins for Qwen3.5-27B, 1 for Qwen3.6-27B, 15 nonsignificant). The non-thinking DeepSeek-R1-32B run wins 2 and loses 7 against Qwen3.5-27B, wins 1 and loses 8 against Qwen3.6-27B, and wins 2 and loses 7 against Gemma4-31B. Its comparison with Qwen3.5-9B is mixed, 6 wins each. There is no GPT-OSS-20B versus Qwen3.5 test and no Nemotron versus Qwen3.5 test. 5 Discussion SyntaxBench surfaces three findings whose interpretation depends on the statistical machinery applied: tokenization mediates the English-vs-random behavior usually attributed to lexical familiarity (§4.2), reasoning mode helps unevenly and sometimes hurts on character-level tasks (§4.3), and positional substring extraction (index_to_span) is unsolved across all 11 (model, mode) configurations (§4.5). Tokenization is a confounder, not a covariate. The standard “LLMs fail on non-natural inputs” framing is descriptive: it observes the gap but does not localize the cause. Our chi-square analysis (§4.2) shows 25 of 197 complete cells exceed the Bonferroni threshold. Random inputs are easier in 23 of those cells and English is easier in 2. The mechanism on the random-easier majority is tokenizer-mediated: random strings pack1.892chars/token, English3.169, andcount_chars accuracy declines as English token count rises, while random remains higher in every shared bin. Any character-level evaluation that does not stratify by tokenization risks attributing tokenizer artifacts to model capability. Reasoning mode is a falsifiable axis.Paired McNemar across thinking and non-thinking variants of Qwen3.6-27B and Gemma4-31B (§4.3) shows three regimes coexist: thinking can hurt at fixed scale (Qwen3.6palindrome_test,∆EMA =−0.066at 4-shot, odds ratio 27.5), be a no-op (Gemma4 across the four near-saturated tasks), or unambiguously help when reasoning is explicitly trained for (DeepSeek-R1). Reporting paired thinking-vs-non-thinking results with token-cost and latency should be standard for any reasoning-toggleable model. Composite leaderboards are shape-blind.Macro EMA at 4-shot ranges from 0.65 to 0.82 across the five thinking models. The composite intervals resample the six task-level EMAs, so their width is between-task spread rather than a sample bootstrap: GPT-OSS-20B is0.818[0.512, 0.989]. Averaging over a saturated task (> 99%), a discrimination task (0–89%), and a floor task (< 7%) hides exactly the structure that distinguishes models. Per-task and per-(mode, shot) reporting is necessary even when a composite is requested. 9 Recommendations for character-level benchmark design. (1) Tokenizer stratification by de- fault; the chars-per-token ratio between conditions disentangles familiarity from granularity. (2) Paired statistical tests across modes and shots (McNemar with continuity correction; Kendall’sτ over shot index). (3) Class-conditional reporting on binary tasks (TPR/TNR/MCC/κ) to surface positive- or negative-class under-prediction. (4) Both Bonferroni and BH-FDR corrections reported alongside; the conservative cutoff anchors headline claims while FDR retains exploratory signal. The remaining headroom onedit_distanceand the floor onindex_to_spanare the two operations most resistant to current mid-sized open-weight transformers; both involve maintaining a counter (over edit operations or character offsets) across a string-valued state and should be implemented with deterministic library calls until architectures designed for long-horizon symbolic state become standard. 6 Limitations Several limitations bound the conclusions of this work. Model and tokenizer coverage. We evaluate eight open-weight LLMs (2B–32B); frontier-scale models (100B+) and closed-weight models are not included. The tokenizer-stratified analysis (§4.2) requires public tokenizer access, so closed-tokenizer models are absent from per-bucket results. Closed prompt and language scope. Each task uses a fixed task-specific system prompt; we do not vary instruction phrasing systematically [33]. SyntaxBench evaluates English and synthetic random strings only; findings may differ for non-Latin scripts, agglutinative morphology, or scripts with qualitatively different tokenizer behavior. Inference-backend differences. Models are served via the backend best supported per architecture (vLLM,llama.cpp, Ollama). With deterministic decoding throughout, backend differences in tokenization handling and reasoning-trace stripping may introduce small per-model variations beyond the model itself. Task scope. The six tasks cover intra-word, inter-word, and positional-substring operations but exclude document-level constrained generation, encoding/decoding, and multi-step algorithmic reasoning, as we deliberately scope to operations with single-pass ground truth and well-defined partial-credit metrics. Statistical and intervention caveats. We report both Bonferroni and BH-FDR corrections; bootstrap CIs assume sample independence within each cell, which our protocol enforces. The thinking-vs- non-thinking comparison conflates differences in the chat template, decoding length, and (for some models) training data; we treat thinking mode as the deployment-level switch a user has access to, not as a controlled intervention isolating the reasoning trace alone. 7 Conclusion We presented SyntaxBench, a diagnostic benchmark and statistical evaluation framework for character-level reasoning in LLMs. The benchmark contains six deterministic tasks, and the code will be released soon. The five core tasks use paired English and length-matched random-string inputs; the index_to_spandocuments share a 200–500 word band and are not character-length matched. The pipeline reports bootstrap CIs, Cohen’sκ, paired McNemar with odds-ratio effect sizes, Kendall’s τover shot index, class-conditional binary metrics, tokenizer-stratified accuracy, integer and string error structure, and Bonferroni- and FDR-corrected significance. Applying this framework to eight open-weight LLMs (2B–32B) across 11 (model, thinking-mode) configurations surfaces three findings that aggregate accuracy hides: (i) tokenization mediates the English-vs-random gap, with random strings easier than English in 23 of the 25 Bonferroni-significant cells, contradicting the standard lexical-familiarity framing; (i) reasoning mode is not uniformly beneficial and sometimes hurts at fixed scale (Qwen3.6-27B onpalindrome_testdrops 6.6 EMA points at 4-shot in thinking mode); (i) positional substring extraction (index_to_span) is unsolved by every model evaluated (max 4-shot EMA 6.75%), differentiating future architectures as the core tasks saturate. 10 The methodological contribution generalizes beyond the specific benchmark: character-level LLM evaluation should stratify by tokenization, report paired tests across modes and shots, and treat reasoning-mode and class-conditional behavior as falsifiable hypotheses with explicit effect sizes. The code will be released soon. References [1] Tairi Ballesteros, Pedro Reviriego, José Alberto Hernández, and Fabrizio Lombardi. Why do large language models (LLMs) struggle to count letters? arXiv preprint arXiv:2412.18626, 2024. [2] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300, 1995. [3] Tom B. Brown et al. Language models are few-shot learners. In NeurIPS, 2020. [4]Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 155–167, 2023. [5] Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. CANINE: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 2022. [6]Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [7]Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46, 1960. [8]DeepSeek-AI. DeepSeek-R1: Open-weight reasoning model, 2025. Open-weight reasoning model; we evaluate the 32B distilled variant. [9]Thomas G. Dietterich. Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1923, 1998. [10]Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2023. [11] Gemma Team, Google DeepMind. Gemma 4 technical report, 2026. Open-weight LLM family; we evaluate the 31B instruction-tuned variant. [12]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021. [13]Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurIPS Datasets and Benchmarks, 2021. [14] M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81–93, 1938. [15]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, et al. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. [16]Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In DeeLIO Workshop, ACL, 2022. 11 [17]Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2022. [18]R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In ACL, 2019. [19] Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947. [20]Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024. [21] Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments. In ACL, 2019. [22] NVIDIA. Nemotron-cascade-2: A hybrid mamba-attention reasoning model, 2026. Open- weight LLM with hybrid Mamba-attention architecture; we evaluate the 31B (3.5B active) variant. [23] Maxwell Nye et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. [24]OpenAI. GPT-OSS: Open-weight reasoning model from OpenAI, 2025. Open-weight LLM with chain-of-thought reasoning; we evaluate the 20B variant. [25] Karl Pearson. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine Series 5, 50(302):157–175, 1900. [26]Qwen Team. Qwen3.5 technical report, 2025. Open-weight LLM family; we evaluate the 2B, 9B, and 27B variants via vLLM at BF16 precision. [27]Qwen Team. Qwen3.6 technical report, 2026. Open-weight LLM family; we evaluate the 27B variant via vLLM at FP8 precision. [28]Xiang Shin and Mamoru Kaneko. Counting ability of large language models and impact of tokenization. arXiv preprint arXiv:2410.19730, 2024. [29] Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. arXiv preprint arXiv:2206.04615, 2022. [30]Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob Van Der Goot, Christian Hardmeier, and Barbara Plank. Experimental standards for deep learning in natural language processing research. In Findings of the association for computational linguistics: EMNLP 2022, pages 2673–2692, 2022. [31] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, and Samuel R. Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019. [32]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [33]Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In NAACL, 2022. [34]Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. [35]Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 2022. 12 Appendix Contents A SyntaxBench Details14 A.1 Dataset Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 A.2 Task Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .14 A.3 Prompt Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 A.4 Evaluation Protocol Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 A.5 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 A.6 Statistical Methods Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.7 Tokenizer-Stratified Analysis Methods . . . . . . . . . . . . . . . . . . . . . . . .17 A.8 Datasheet Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 B Supplementary Tables and Figures18 B.1 Aggregate Performance Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 B.2 Supplementary Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 13 A SyntaxBench Details A.1 Dataset Construction A.1.1 Text Data Collection We construct SyntaxBench inputs from three complementary sources. The English vocabulary is drawn from the Natural Language Toolkit (NLTK) corpus (236,736 words). Forpalindrome_test, we additionally use a curated palindrome word list of 229 entries sourced from Wiktionary 2 . For index_to_span, we use English Wikipedia excerpts and random-vocabulary documents targeted to 200–500 words. Mean length is 2,128 characters on the English condition and 3,593 characters on the random condition, because the random tokens are longer than Wikipedia’s short function words. A.1.2 Length-Matched Random-String Corpus We generate a synthetic random-string corpus designed to match the empirical character-length distribution of the NLTK English vocabulary. For each English word of lengthℓ, we sample a random string of the same length uniformly from the alphabet[a-z]. This pairing controls for the trivial confound that random strings might differ in length from English words, which would otherwise contaminate any English-vs-random comparison via length effects rather than character-distribution effects. Forindex_to_span, both conditions are targeted to 200–500 words (observed means 321 English and 336 random). Random documents are assembled by sampling that many strings from the random vocabulary, not by replacing each English word in an excerpt. Those strings follow the NLTK word-list lengths rather than Wikipedia’s short function words, so mean document length is 3,593 characters versus 2,128. The matched-length design for the word-level corpus has been checked against the chi-square analysis: across 197 complete (model, task, shot, mode) comparisons, 25 (12.7%) show a significant English- vs-random gap after Bonferroni correction. Random inputs are easier in 23 of those cells and English is easier in 2 (§4.2). One Qwen3.5-2Bindex_to_spancell is incomplete and is excluded from the family. A.1.3 Per-Task Sample Counts • count_chars: 200 English + 200 random = 400 samples • letter_container: 400 English + 400 random = 800 samples (balanced across yes/no ground truth) • palindrome_test : 400 English + 400 random = 800 samples (balanced across yes/no ground truth) • edit_distance: 200 English pairs + 200 random pairs = 400 samples • longest_string_finder: 200 English lists + 200 random lists = 400 samples • index_to_span: 200 English documents + 200 random documents = 400 samples Total: 3,200 samples per (shot, thinking-mode) configuration. Across the three shot modes and (where applicable) two thinking modes, this gives up to 19,200 inference requests per model. A.2 Task Definitions We define six syntactic tasks organized along an intra-word vs. inter-word structural axis (the five core tasks) plus one stress-test task (index_to_span). Each task includes (1) a task-specific system prompt, (2) demonstration examples enabling 0-, 1-, and 4-shot evaluation, and (3) test samples with deterministic ground-truth labels. A.2.1 Intra-Word Core Tasks Count Characters (count_chars). Given a strings, return the integer character length|s|. Models must accurately enumerate all characters. The dataset contains 400 samples (200 English words, 200 length-matched random strings). Output type: integer. 2 https://en.wiktionary.org/wiki/Appendix:English_palindromes 14 Letter Container (letter_container). Given a stringsand a single characterc, return “yes” if cappears ins(case-insensitive), else “no”. The dataset contains 800 samples balanced across four cells of 200 each: English-yes, English-no, random-yes, random-no. Output type: yes/no. Palindrome Test (palindrome_test). Given a strings, return “yes” ifsreads identically forward and backward (case-insensitive), else “no”. The dataset contains 800 samples balanced across English palindromes, English non-palindromes, random palindromes, and random non-palindromes (200 each). Output type: yes/no. A.2.2 Inter-Word Core Tasks Edit Distance (edit_distance). Given a pair of strings(s 1 ,s 2 ), return the integer Levenshtein edit distance, the minimum number of single-character insertions, deletions, or substitutions required to transform s 1 into s 2 . The dataset contains 400 samples (200 English word pairs, 200 random pairs). Output type: integer. Longest String Finder (longest_string_finder). Given a list of≥ 3strings with one strictly longest element, return the longest string verbatim (case-insensitive match). The dataset contains 400 samples (200 English-word lists, 200 random-string lists). Output type: string. A.2.3 Stress-Test Task Index to Span (index_to_span). Given a long text documentdand two character offsets(i,j), return the substringd[i : j]verbatim. The model must maintain a character-position counter across the document while extracting span contents. The dataset contains 400 samples (200 English Wikipedia excerpts, 200 documents sampled from the random vocabulary), both in the 200–500 word band; mean document length is 2,128 characters on the English condition and 3,593 characters on the random condition. Output type: string. A.3 Prompt Engineering We designed prompts to minimize ambiguity and enforce machine-checkable JSON output. Each task uses a fixed task-specific system prompt with multiple natural-language phrasings as variants. All models are instructed to return a JSON object with a single fieldresultholding the answer; the response parser handles whitespace and JSON-formatting variability (§3.5). Initial prompt candidates were generated and manually refined through pilot testing to ensure consistent formatting across model families. The full prompt templates and demonstration pools are released alongside the dataset. A.4 Evaluation Protocol Details For each task, models receive the task-specific system prompt followed by 0, 1, or 4 demonstration message pairs (one user turn + one assistant turn per demonstration), then the test query. Demon- strations are drawn from a held-out pool that does not overlap with the test set. Each test instance is executed as an independent inference request that reuses the system prompt and demonstrations but does not include outputs from prior test queries. Requests are dispatched concurrently to the inference backend. For models with both reasoning configurations (Qwen3.6-27B, Gemma4-31B), each (shot) cell is evaluated twice: once with the chat template’s thinking flag enabled (enable_thinking=True), once disabled. Reasoning content (when emitted in a separate field by reasoning-capable models) is captured for diagnostic purposes (mean thinking-token counts, see Table 15) but excluded from the parsed answer used for scoring. The response parser extracts the value of theresultkey after stripping whitespace and normalizing JSON. We additionally apply a Markdown-bold (**...**) recovery rule for one model whose chat template wraps responses in bold markers; this rule is applied uniformly across all models for fairness. A.5 Experimental Setup A.5.1 Hardware Configuration • GPUs: 2× NVIDIA L40 (48GB VRAM each) 15 • Memory: 96GB GPU VRAM total, 256GB system RAM • Compute Environment: Ubuntu 24.04 LTS with CUDA 12.4 A.5.2 Models, Modes, and Inference Backends We evaluate eight open-weight LLMs across 11 (model, thinking-mode) configurations. Per-(model, mode) backend, quantization, and tensor-parallel-size are summarized below. • Qwen3.5-2B / 9B / 27B [26]: served via vLLM at BF16, single-GPU (2B/9B) or TP=2 (27B). Non-thinking only. •Qwen3.6-27B [27]: served via vLLM at FP8, single-GPU. Both thinking and non-thinking modes via the chat template. •Gemma4-31B-it [11]: served via vLLM at BF16 with tensor-parallel-size 2 (split across both GPUs). Both modes. •GPT-OSS-20B [24]: served via Ollama. Vendor-native MXFP4 quantization. Thinking- only (Ollama was the only backend with stable support for this model’s architecture on Ada-Lovelace GPUs at the time of evaluation). •Nemotron-Cascade-2-31B [22]: served viallama.cppwith split GGUF shards. Thinking- only. • DeepSeek-R1-32B [8]: served via vLLM at BF16 with TP=2. Thinking-by-design. Its non-thinking configuration disables reasoning-trace accounting rather than reasoning itself, and is reported separately. Together with the entries above, these are the 11 configurations. All models are instruction-tuned, support system prompts, and produce JSON-formatted output reliably enough to make per-response parsing the primary signal-extraction mechanism rather than format-recovery heuristics. A.5.3 Inference Hyperparameters All models were evaluated with the following standardized configuration: • Temperature: 0.0 (greedy decoding for reproducibility) • Top-p: 1.0 (nucleus sampling disabled) •Max Tokens: 2048 for non-thinking, 8192 for thinking (sufficient for reasoning traces plus JSON response on all tasks) • Repetition Penalty: 1.0 (disabled) • Stop Sequences: None (rely on EOS token) A.5.4 Reproducibility • Random Seeds: Fixed seed (42) for all stochastic operations in the evaluation pipeline (bootstrap, demonstration sampling). •Dataset Splits: No train/test split. Each task is evaluated at 0-, 1-, and 4-shot on a fixed test set. •Prompt Templates: Standardized system prompts and demonstration formats, released alongside the dataset (see §A.2 and supplementary materials). •Deterministic Decoding: Temperature 0.0 ensures reproducible outputs across runs for a given backend version. •Run Manifest: The code release will include a per-run JSON manifest recording GPU model, backend version, quantization, wall-clock time, and peak VRAM. The evaluation pipeline and the analysis scripts that produce every figure and table in this paper will be released soon. Raw model outputs are stored in JSONL format. A.6 Statistical Methods Reference This section provides quick reference for the statistical procedures used in the main text. Cohen’sκ[7]. For binary tasks, we compute unweightedκon the contingency table of (predicted, ground-truth) pairs:κ = (p o − p e )/(1− p e ), wherep o is observed agreement andp e is chance agreement. For integer tasks (count_chars,edit_distance), we compute linearly-weightedκ w 16 where the disagreement penalty scales with|y pred − y true |. Interpretation thresholds (Landis-Koch): κ < 0.20slight,0.20–0.40fair,0.40–0.60moderate,0.60–0.80substantial,≥ 0.80almost perfect. Negative κ indicates systematic anti-agreement. McNemar’s test [19]. For a paired-classifier comparison onnsamples, letb 01 be the count of samples where model A is correct and model B incorrect, andb 10 the reverse. The test statistic with continuity correction isχ 2 = (|b 01 − b 10 |− 1) 2 /(b 01 + b 10 ), distributed asχ 2 1 under the null. The odds ratiob 01 /b 10 is reported as effect size; cells withb 01 = b 10 = 0are flagged as ceiling effects. The test is also applied per-model in a paired thinking-vs-non-thinking comparison on the same (task, shot) cell. Bootstrap confidence intervals. 95% CIs on per-task EMA are computed by resampling samples (not models) with replacement 10,000 times, using the percentile method. The composite macro-EMA interval instead resamples the six task-level EMAs, so it measures between-task spread. The random seed is fixed for reproducibility. Chi-square test of independence [25]. For the English-vs-random comparison on each (model, task, shot, mode) cell, we construct a 2×2 contingency table (condition×correctness) and apply Pearson’s chi-square test with Yates’ continuity correction when any cell count is below 5. Cramér’s Vis reported as effect size. We use chi-square rather than McNemar because English and random are independent samples, not paired observations. Kendall’sτ[14] on shot mode. For each (model, task, mode, condition), we computeτbetween shot index0, 1, 4and EMA. Withn = 3points,τtakes values in−1,−0.33, 0.33, 1, and we additionally classify the trajectory pattern by examining the actual values:monotone_up(τ = 1), monotone_down(τ =−1),peak_1shot(max at shot 1),valley_1shot(min at shot 1), orflat (no variation). Multiple-comparison correction. We apply both Bonferroni and Benjamini-Hochberg FDR [2] corrections atα = 0.05over the relevant comparison family. Headline tables in the main text use the more conservative Bonferroni cutoff; the FDR-corrected q-values are released in the supplementary CSVs (evaluation/mcnemar_*.csv,evaluation/chi2_eng_vs_rand.csv). The McNemar matrix is corrected over all model pairs in each (task, shot, mode, condition) cell; the chi-square English-vs-random tests are corrected over the 197 comparisons that contain both conditions. The one incomplete cell is excluded, and Benjamini–Hochberg FDR is applied to the same 197-comparison family. A.7 Tokenizer-Stratified Analysis Methods For each model with a publicly-available HuggingFace tokenizer, we tokenize the raw input content (the text inside the<string>,<list>, etc., XML-style markers used by the system prompts) and report (i) per-(model, task, condition) mean tokens, mean chars-per-token, and per-bin sample counts, and (i) per-(model, task, condition, token-bin) accuracy at 0-shot. The tokenizer used per model is recorded inanalysis/tokenization/tokenization_full.csv; for closed-tokenizer models we omit the model from the tokenizer-stratified analysis but retain it in the chi-square English-vs- random analysis (which does not depend on tokenizer access). Cohen’sdis computed on per-sample tokens per character, pooled across the five core tasks and all public tokenizers. The single-characterletterquery tag is excluded, as isindex_to_span. Withn = 8,000samples in each condition, the means are0.542(random) and0.344(En- glish) tokens per character, andd = ( ̄x random − ̄x English )/s pooled = 2.165. The corresponding mean characters per token are1.892and3.169. The inputs and inclusion rule are recorded in analysis/tokenization/tokenization_headline.csv. A.8 Datasheet Summary Motivation. SyntaxBench was created to evaluate LLM character-level reasoning under deterministic ground truth, with paired English and random inputs and statistical machinery sufficient to surface tokenization, reasoning-mode, and class-bias effects. Length matching applies to the five core tasks. Composition. Six tasks, 3,200 samples per complete (shot, thinking-mode) configuration, with En- glish and random conditions. The Qwen3.5-2Bindex_to_span4-shot cell is incomplete (§4.5). In- 17 Table 4: Main results (thinking mode where supported): Exact Match Accuracy (EMA, %) at 4-shot with 95% bootstrap CI. Significance vs. best model: ∗ p<0.05, ∗ p<0.01, ∗ p<0.001(Bonferroni- corrected McNemar’s test). count_charsedit_distanceindex_to_spanletter_containerlongest_string_finderpalindrome_test englishrandomenglishrandomenglishrandomenglishrandomenglishrandomenglishrandom deepseek_r1_32b92.5 88.5,96.0 ∗ 100.0 100.0,100.0 34.5 28.0,41.5 ∗40.5 33.5,47.5 ∗0.0 0.0,0.0 ∗0.5 0.0,1.5 ∗98.8 97.5,99.8 100.0 100.0,100.0 99.5 98.5,100.0 100.0 100.0,100.0 83.2 79.5,86.8 ∗86.8 83.2,90.0 ∗ gemma4_31b94.5 91.0,97.5 ∗98.0 96.0,99.5 44.5 37.5,51.5 ∗34.5 28.0,41.0 ∗0.5 0.0,1.5 ∗0.5 0.0,1.5 ∗98.8 97.5,99.8 99.8 99.2,100.0 97.0 94.5,99.0 100.0 100.0,100.0 92.0 89.2,94.5 ∗94.5 92.2,96.5 ∗ gpt-oss_20b94.5 91.0,97.5 ∗ 100.0 100.0,100.0 86.0 81.0,90.5 92.0 88.0,95.5 8.0 4.5,12.0 5.5 2.5,9.0 99.5 98.8,100.0 100.0 100.0,100.0 97.5 95.0,99.5 99.5 98.5,100.0 99.8 99.2,100.0 99.0 98.0,99.8 nemotron-cascade-296.5 94.0,99.0 98.0 96.0,99.5 26.0 20.0,32.0 ∗42.5 35.5,49.5 ∗0.0 0.0,0.0 ∗0.0 0.0,0.0 ∗ 100.0 100.0,100.0 99.8 99.2,100.0 66.5 60.0,73.0 ∗71.5 65.0,77.5 ∗83.5 79.8,87.0 ∗93.0 90.5,95.5 ∗ qwen3.6_27b 100.0 100.0,100.0 100.0 100.0,100.0 31.5 25.0,38.0 ∗28.0 22.0,34.5 ∗0.0 0.0,0.0 ∗0.0 0.0,0.0 ∗98.0 96.5,99.2 100.0 100.0,100.0 95.0 92.0,98.0 98.0 96.0,99.5 88.0 84.8,91.0 ∗89.2 86.2,92.2 ∗ Table 5: Main results (non-thinking mode): Exact Match Accuracy (EMA, %) at 4-shot with 95% bootstrap CI. Significance vs. best model: ∗ p<0.05, ∗ p<0.01, ∗ p<0.001(Bonferroni-corrected McNemar’s test). “–” means that condition was not evaluated. For qwen3.5_2b on index_to_span at 4-shot, the English cell is the observed 113-item subset and the random condition is missing; the combined score is therefore not estimated. count_charsedit_distanceindex_to_spanletter_containerlongest_string_finderpalindrome_test englishrandomenglishrandomenglishrandomenglishrandomenglishrandomenglishrandom deepseek_r1_32b76.0 70.0,82.0 ∗84.5 79.0,89.5 ∗0.0 0.0,0.0 ∗0.0 0.0,0.0 ∗0.0 0.0,0.0 0.0 0.0,0.0 97.8 96.2,99.0 98.2 97.0,99.5 53.0 46.0,60.0 ∗33.5 27.0,40.0 ∗87.8 84.5,90.8 ∗87.8 84.5,90.8 ∗ gemma4_31b94.5 91.0,97.5 98.0 96.0,99.5 39.0 32.0,45.5 36.0 29.5,42.5 ∗0.5 0.0,1.5 0.5 0.0,1.5 98.8 97.5,99.8 99.8 99.2,100.0 97.0 94.5,99.0 100.0 100.0,100.0 92.2 89.5,94.8 94.2 91.8,96.5 qwen3.5_27b91.5 87.5,95.0 100.0 100.0,100.0 36.5 30.0,43.0 49.5 42.5,56.5 1.0 0.0,2.5 0.0 0.0,0.0 98.5 97.2,99.5 99.2 98.2,100.0 94.0 90.5,97.0 98.5 96.5,100.0 91.2 88.5,93.8 95.2 93.0,97.2 qwen3.5_2b75.5 69.5,81.5 ∗64.0 57.5,70.5 ∗23.0 17.0,29.0 ∗29.0 23.0,35.5 ∗0.0 0.0,0.0 –77.5 73.2,81.5 ∗88.0 84.8,91.2 ∗48.5 41.5,55.5 ∗47.5 40.5,54.5 ∗72.0 67.5,76.2 ∗94.8 92.5,96.8 qwen3.5_9b79.0 73.0,84.5 ∗97.0 94.5,99.0 34.0 27.5,40.5 45.5 38.5,52.5 0.5 0.0,1.5 0.5 0.0,1.5 93.5 91.0,95.8 ∗98.8 97.5,99.8 74.5 68.0,80.5 ∗89.0 84.5,93.0 ∗90.5 87.5,93.2 96.8 95.0,98.2 qwen3.6_27b 96.5 94.0,99.0 99.5 98.5,100.0 32.0 25.5,38.5 33.5 27.0,40.0 ∗0.0 0.0,0.0 0.0 0.0,0.0 96.8 95.0,98.5 99.5 98.8,100.0 96.5 93.5,99.0 98.5 96.5,100.0 94.2 91.8,96.2 96.2 94.2,98.0 puts are non-personal lexical items (words, character lists, public Wikipedia excerpts). No personally- identifiable information is included. Collection process. English vocabulary is the NLTK corpus; palindromes are sourced from Wik- tionary; Wikipedia excerpts are used under C BY-SA. Random strings are programmatically generated (seeded). The construction scripts will be released soon. Preprocessing. For the five core tasks, length-matching pairs each English word of lengthℓwith a uniformly-random string of lengthℓ.index_to_spanrandom documents are sampled from this vocabulary into the same 200–500 word band and are not matched to the English character lengths. Demonstrations are sampled from a held-out pool with fixed seed. Uses. The dataset is intended for diagnostic evaluation of character-level reasoning, comparison of tokenizer-mediated effects, and analysis of reasoning-mode behavior. It is not intended for safety, fairness, or capability claims about LLMs as a whole. Distribution. The dataset underdata/is released under C BY-SA 4.0, the same share-alike terms required by the Wikipedia excerpts. The code will be released soon. The Wiktionary-derived palindrome word list retains the attribution and share-alike terms of its C BY-SA 3.0 source. A machine-readable Croissant metadata record, croissant.json, accompanies the dataset. Maintenance. The code will be released soon. Updates will use semantic-version bumps. The release manifest records all evaluated model versions and inference backends. B Supplementary Tables and Figures This appendix contains supplementary tables and figures referenced from the main text. All tables are generated by the analysis code, which will be released soon. Readers can reproduce them from evaluation/metrics_summary.csvand the per-test CSVs. Per-sample model outputs are stored in JSONL format, one file per (model, task, shot, mode) cell. B.1 Aggregate Performance Tables This subsection collects the full aggregate tables: per-task EMA in thinking mode (Table 4) and non-thinking mode (Table 5), the shot ablation (Table 6), Cohen’sκ(Table 7), the pairwise McNemar significance matrices for both modes (Tables 8 and 9), Kendall’sτover shot count (Table 10), the English-vs-random tests (Table 11), parse-error rates (Table 12), delta distributions for the integer tasks (Tables 13 and 14), thinking-cost accounting (Table 15), effect sizes (Table 16), and the full composite leaderboards (Tables 17 and 18). 18 Table 6: Shot-mode ablation (thinking mode where supported): EMA (%) across 0-, 1-, and 4-shot settings (condition=all). Bold = best shot per model× task. count_charsedit_distanceindex_to_spanletter_containerlongest_string_finderpalindrome_test 0-shot1-shot4-shot0-shot1-shot4-shot0-shot1-shot4-shot0-shot1-shot4-shot0-shot1-shot4-shot0-shot1-shot4-shot deepseek_r1_32b97.894.596.239.834.237.50.50.00.298.998.699.499.899.099.895.094.285.0 gemma4_31b87.892.096.229.028.739.50.50.50.599.599.199.295.897.898.584.192.993.2 gpt-oss_20b96.597.597.285.288.889.07.46.56.899.599.299.899.899.898.598.898.999.4 nemotron-cascade-296.296.297.223.016.034.20.00.00.099.899.599.957.572.869.081.986.888.2 qwen3.6_27b99.299.8100.037.235.829.80.00.00.099.999.699.0100.099.396.595.581.988.6 Table 7: Cohen’sκ(thinking mode where supported) (4-shot, condition=all). Annotation uses headline κ (linear weighted for integer tasks). Modelcount_charsedit_distanceletter_containerlongest_string_finderpalindrome_test deepseek_r1_32b0.984 almost_perfect0.556 moderate0.988 almost_perfect0.997 almost_perfect0.7 substantial gemma4_31b0.988 almost_perfect0.743 substantial0.985 almost_perfect0.985 almost_perfect0.865 almost_perfect gpt-oss_20b0.991 almost_perfect0.941 almost_perfect0.995 almost_perfect0.985 almost_perfect0.988 almost_perfect nemotron-cascade-20.988 almost_perfect0.623 substantial0.998 almost_perfect0.689 substantial0.765 substantial qwen3.6_27b1.0 almost_perfect0.563 moderate0.98 almost_perfect0.965 almost_perfect0.772 substantial Table 8: McNemar pairwise comparisons (thinking mode where supported): fraction of Bonferroni- significant task×shot cells (numerator/denominator). Blue tint: row model tends to win net; red: column model; gray: mixed. deepseek_r1_32bgemma4_31bgpt-oss_20bnemotron-cascade-2qwen3.6_27b deepseek_r1_32b—5/189/187/184/18 gemma4_31b5/18—12/187/188/18 gpt-oss_20b9/1812/18—12/1810/18 nemotron-cascade-27/187/1812/18—8/18 qwen3.6_27b4/188/1810/188/18— Table 9: McNemar pairwise comparisons (non-thinking mode): fraction of Bonferroni-significant task×shot cells (numerator/denominator). Blue tint: row model tends to win net; red: column model; gray: mixed. deepseek_r1_32bgemma4_31bqwen3.5_27bqwen3.5_2bqwen3.5_9bqwen3.6_27b deepseek_r1_32b—9/189/1811/1812/189/18 gemma4_31b9/18—6/1815/188/185/18 qwen3.5_27b9/186/18—15/1812/183/18 qwen3.5_2b11/1815/1815/18—14/1814/18 qwen3.5_9b12/188/1812/1814/18—9/18 qwen3.6_27b9/185/183/1814/189/18— Table 10: Kendall’sτ(thinking mode where supported) between shot count0, 1, 4and EMA (condition=all). Low power at n=3 points — interpret patterns qualitatively. Modelcount_charsedit_distanceindex_to_spanletter_containerlongest_string_finderpalindrome_test deepseek_r1_32b τ =− 0.333 valley_1shot τ =− 0.333 valley_1shot τ =− 0.333 valley_1shot τ =0.333 valley_1shotτ =0.0 valley_1shotτ =− 1.0 monotone_down gemma4_31bτ =1.0 monotone_upτ =0.333 valley_1shot–τ =− 0.333 valley_1shotτ =1.0 monotone_upτ =1.0 monotone_up gpt-oss_20bτ =0.333 peak_1shotτ =1.0 monotone_up τ =− 0.333 valley_1shot τ =0.333 valley_1shot τ =− 0.816 monotone_down τ =1.0 monotone_up nemotron-cascade-2 τ =0.816 monotone_up τ =0.333 valley_1shot–τ =0.333 valley_1shotτ =0.333 peak_1shotτ =1.0 monotone_up qwen3.6_27bτ =1.0 monotone_up τ =− 1.0 monotone_down–τ =− 1.0 monotone_down τ =− 1.0 monotone_down τ =− 0.333 valley_1shot Table 11: English vs. random gap (thinking mode where supported) (∆EMA = EMA eng −EMA rand , %) at 4-shot.Blue= English advantage;Red= random advantage. Significance (Bonferroni- corrected): ∗ p<0.05, ∗ p<0.01, ∗ p<0.001. count_charsedit_distanceindex_to_spanletter_containerlongest_string_finderpalindrome_test deepseek_r1_32b−7.5∗−6.0−0.5−1.2−0.5−3.5 gemma4_31b−3.5+10.00.0−1.0−3.0−2.5 gpt-oss_20b−5.5−6.0+2.5−0.5−2.0+0.8 nemotron-cascade-2−1.5−16.50.0+0.2−5.0−9.5∗ qwen3.6_27b0.0+3.50.0−2.0−3.0−1.2 Table 12: Parse error rate (PER, %) across models, tasks, and shot settings (thinking mode where supported) (condition=all). Format: PER (blank; inf.err). count_charsedit_distanceindex_to_spanletter_containerlongest_string_finderpalindrome_test 0-shot1-shot4-shot0-shot1-shot4-shot0-shot1-shot4-shot0-shot1-shot4-shot0-shot1-shot4-shot0-shot1-shot4-shot deepseek_r1_32b0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.5 (0.0; 0.5)0.2 (0.0; 0.2)0.5 (0.0; 0.5)0.2 (0.0; 0.2)0.2 (0.0; 0.2)0.5 (0.0; 0.5)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.2 (0.0; 0.2)0.0 (0.0; 0.0)0.2 (0.0; 0.2)3.5 (0.0; 3.5)4.5 (0.0; 4.5)14.8 (0.0; 14.8) gemma4_31b0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0) gpt-oss_20b0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)2.1 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)3.7 (0.0; 3.7)17.2 (0.0; 17.2)21.2 (0.0; 21.2)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.2 (0.0; 0.0)0.5 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0) nemotron-cascade-20.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)2.0 (0.0; 0.0)0.0 (0.0; 0.0)0.8 (0.0; 0.0)0.3 (0.0; 0.3)2.2 (0.0; 2.2)2.2 (0.0; 2.2)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)15.2 (0.0; 0.0)0.0 (0.0; 0.0)1.2 (0.0; 0.0)4.4 (0.0; 0.0)0.0 (0.0; 0.0)0.4 (0.0; 0.0) qwen3.6_27b0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0)0.0 (0.0; 0.0) 19 Table 13: Signed delta distribution (ˆy− y) forcount_chars(thinking mode where supported) at 4-shot (valid responses only). ModelCond.Mean± StdMedianP5P95 n deepseek_r1_32b english0.03± 0.390.000.000.05200 random0.00± 0.000.000.000.00200 all0.01± 0.270.000.000.00400 gemma4_31b english0.06± 0.230.000.001.00200 random0.01± 0.140.000.000.00200 all0.03± 0.190.000.000.00400 gpt-oss_20b english −0.05± 0.260.00-0.050.00200 random0.00± 0.000.000.000.00200 all −0.03± 0.190.000.000.00400 nemotron-cascade-2 english −0.03± 0.310.000.000.00200 random0.00± 0.140.000.000.00200 all −0.01± 0.240.000.000.00400 qwen3.6_27b english0.00± 0.000.000.000.00200 random0.00± 0.000.000.000.00200 all0.00± 0.000.000.000.00400 Table 14: Signed delta distribution (ˆy− y) foredit_distance(thinking mode where supported) at 4-shot (valid responses only). ModelCond.Mean± StdMedianP5P95 n deepseek_r1_32b english0.00± 1.590.00-2.003.00199 random −0.07± 2.150.00-3.003.10199 all −0.04± 1.890.00-2.153.00398 gemma4_31b english0.01± 0.920.00-1.001.05200 random −0.66± 0.73-1.00-2.000.00200 all −0.33± 0.900.00-2.001.00400 gpt-oss_20b english0.07± 0.740.00-1.001.00200 random0.00± 0.400.000.000.00200 all0.03± 0.600.00-1.001.00400 nemotron-cascade-2 english −0.06± 1.330.00-2.002.00197 random −0.40± 1.060.00-2.001.00200 all −0.23± 1.210.00-2.002.00397 qwen3.6_27b english −0.69± 1.29-1.00-3.001.00200 random −1.01± 1.14-1.00-3.001.00200 all −0.85± 1.23-1.00-3.001.00400 20 Table 15: Thinking-cost efficiency at 4-shot (thinking mode, condition=all). Top-3 per task by EMA / mean thinking tokens (higher is better). NaN tokens (no paired runs) shown as “–”. TaskModelEMAMean tokensEMA/token count_charsdeepseek_r1_32b96.2%2740.00351 count_charsgemma4_31b96.2%0– count_charsgpt-oss_20b97.2%– edit_distancedeepseek_r1_32b37.5%15550.00024 edit_distancegemma4_31b39.5%0– edit_distancegpt-oss_20b89.0%0– index_to_spandeepseek_r1_32b0.2%3811e− 05 index_to_spangemma4_31b0.5%0– index_to_spangpt-oss_20b6.8%0– letter_containerdeepseek_r1_32b99.4%2020.00493 letter_containergemma4_31b99.2%0– letter_containergpt-oss_20b99.8%– longest_string_finderdeepseek_r1_32b99.8%3280.00304 longest_string_findergemma4_31b98.5%0– longest_string_findergpt-oss_20b98.5%0– palindrome_testdeepseek_r1_32b85.0%2280.00373 palindrome_testgemma4_31b93.2%0– palindrome_testgpt-oss_20b99.4%0– Table 16: Top-12 model-pair effect sizes at 4-shot (thinking, condition=all), sorted by|h|. CIs are 95% bootstrap percentile. TaskModel AModel Bh [95% CI]Risk diff. p q edit_distancegpt-oss_20bqwen3.6_27b1.31 [1.18, 1.45]0.5930.00.0 edit_distancegpt-oss_20bnemotron-cascade-21.22 [1.09, 1.36]0.5470.00.0 edit_distancedeepseek_r1_32bgpt-oss_20b−1.15 [-1.29, -1.01] −0.5150.00.0 edit_distancegemma4_31bgpt-oss_20b−1.11 [-1.25, -0.97] −0.4950.00.0 longest_string_finderdeepseek_r1_32bnemotron-cascade-21.08 [0.95, 1.26]0.3080.00.0 longest_string_findergemma4_31bnemotron-cascade-20.94 [0.8, 1.09]0.2950.00.0 longest_string_findergpt-oss_20bnemotron-cascade-20.94 [0.8, 1.09]0.2950.00.0 longest_string_findernemotron-cascade-2qwen3.6_27b −0.8 [-0.94, -0.67] −0.2750.00.0 palindrome_testdeepseek_r1_32bgpt-oss_20b−0.64 [-0.74, -0.55] −0.1440.00.0 palindrome_testgpt-oss_20bnemotron-cascade-20.54 [0.45, 0.65]0.1110.00.0 palindrome_testgpt-oss_20bqwen3.6_27b0.53 [0.43, 0.64]0.1080.00.0 index_to_spangpt-oss_20bnemotron-cascade-20.53 [0.42, 0.62]0.0680.00.0 Table 17: Composite leaderboard at 4-shot (thinking, condition=all). Macro EMA averages over complete task-level EMAs and omits a task when either input condition is missing; CI is 95% bootstrap percentile of those task-level EMAs. RankModelMacro EMA [95% CI]VRRPER 1gpt-oss_20b0.818 [0.512, 0.989]0.9640.036 2gemma4_31b0.712 [0.388, 0.971]1.00.0 3deepseek_r1_32b0.697 [0.385, 0.966]0.9730.027 4qwen3.6_27b0.69 [0.358, 0.972]1.00.0 5nemotron-cascade-20.648 [0.339, 0.919]0.9920.008 Table 18: Composite leaderboard at 4-shot (non-thinking mode). Macro EMA averages over complete task-level EMAs and omits a task when either input condition is missing; CI is 95% bootstrap percentile of those task-level EMAs. RankModelMacro EMA [95% CI]VRRPER 1qwen3.5_27b0.713 [0.394, 0.966]1.00.0 2gemma4_31b0.709 [0.385, 0.971]1.00.0 3qwen3.6_27b0.703 [0.376, 0.975]1.00.0 4qwen3.5_9b0.666 [0.361, 0.915]0.9990.001 5qwen3.5_2b0.62 [0.419, 0.804]1.00.0 6deepseek_r1_32b0.515 [0.163, 0.83]0.6230.377 21 B.2 Supplementary Figures The supplementary figures visualize the English-vs-random gap (Figure 4), shot-mode learning curves (Figure 5), Cohen’sκ(Figure 6), Kendall-τtrajectories (Figure 7), the McNemar pairwise summary (Figure 8), parse-error rates (Figure 9), point-biserial condition effects (Figure 10), tolerance behavior on integer tasks (Figures 11 and 14), per-model capability profiles (Figure 12), thinking-cost trade-offs (Figure 13), and the composite Pareto frontier and mode-specific dominance graphs (Figures 15, 16, and 17). count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all DeepSeek32 [NT] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all DeepSeek32 [T] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all GPT-OSS20 [T] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all Gemma31 [NT] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all Gemma31 [T] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all Nemotron [T] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all Qwen3.5-2 [NT] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all Qwen3.5-27 [NT] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all Qwen3.5-9 [NT] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all Qwen3.6-27 [NT] English Random count edit span contains longest palin 0.00 0.25 0.50 0.75 1.00 EMA_all Qwen3.6-27 [T] English Random English vs random (4-shot) with bootstrap CIs Figure 4: English vs. random EMA gap per (model, mode, task) at 4-shot, with 95% bootstrap CIs. After Bonferroni correction over 197 complete comparisons, 25 cells (12.7%) show a significant gap; random inputs are easier in 23 of them (§4.2). 22 014 Shot 0.0 0.2 0.4 0.6 0.8 1.0 EMA_all count DeepSeek32 [NT] DeepSeek32 [T] GPT-OSS20 [T] Gemma31 [NT] Gemma31 [T] Nemotron [T] Qwen3.5-2 [NT] Qwen3.5-27 [NT] Qwen3.5-9 [NT] Qwen3.6-27 [NT] Qwen3.6-27 [T] 014 Shot 0.0 0.2 0.4 0.6 0.8 1.0 EMA_all edit 014 Shot 0.0 0.2 0.4 0.6 0.8 1.0 EMA_all span 014 Shot 0.0 0.2 0.4 0.6 0.8 1.0 EMA_all contains 014 Shot 0.0 0.2 0.4 0.6 0.8 1.0 EMA_all longest 014 Shot 0.0 0.2 0.4 0.6 0.8 1.0 EMA_all palin Shot-mode learning curves (condition=all) Figure 5: Shot-mode learning curves per (model, mode, task, condition). Reading horizontally shows the per-(model, mode) trajectory across 0/1/4-shot; descending curves correspond to the few-shot interference effect on saturated tasks (§4.4). 23 counteditspancontainslongestpalin DeepSeek32 [NT] all DeepSeek32 [NT] english DeepSeek32 [NT] random DeepSeek32 [T] all DeepSeek32 [T] english DeepSeek32 [T] random GPT-OSS20 [T] all GPT-OSS20 [T] english GPT-OSS20 [T] random Gemma31 [NT] all Gemma31 [NT] english Gemma31 [NT] random Gemma31 [T] all Gemma31 [T] english Gemma31 [T] random Nemotron [T] all Nemotron [T] english Nemotron [T] random Qwen3.5-2 [NT] all Qwen3.5-2 [NT] english Qwen3.5-2 [NT] random Qwen3.5-27 [NT] all Qwen3.5-27 [NT] english Qwen3.5-27 [NT] random Qwen3.5-9 [NT] all Qwen3.5-9 [NT] english Qwen3.5-9 [NT] random Qwen3.6-27 [NT] all Qwen3.6-27 [NT] english Qwen3.6-27 [NT] random Qwen3.6-27 [T] all Qwen3.6-27 [T] english Qwen3.6-27 [T] random modslttopmodsub modslttopmodsub modslttopfairsub topmodtoptopsub topmodtoptopsub topmodtoptopsub toptoptoptoptop toptoptoptoptop toptoptoptoptop topsubtoptoptop topsubtoptoptop topsubtoptoptop topsubtoptoptop topsubtoptoptop topsubtoptoptop topsubtopsubsub topmodtopsubsub topsubtopsubtop topmodsubmodsub topfairmodmodmod topmodsubmodtop topsubtoptoptop topsubtoptoptop topsubtoptoptop topsubtoptoptop topmodtopsubtop topsubtoptoptop topmodtoptoptop topmodtoptoptop topsubtoptoptop topmodtoptopsub topmodtoptopsub topmodtoptopsub Cohen's (shot=4); rev/<ch = negative agreement bands 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 (TwoSlopeNorm: 0 = neutral) Figure 6: Cohen’sκheatmap per (model, mode, task, shot). Cells withκ < 0indicate systematic anti-agreement (rare in the post-parser-fix data). The thinking-mode degradation on Qwen3.6 palindrome_test(§4.3) is visible as the lowerκin the thinking row vs. the non-thinking row at 1- and 4-shot. 24 DeepSeek32 [NT] =-1.000 monotone_down count =-0.333 peak_1shot edit =nan flat span =-1.000 monotone_down contains =-1.000 monotone_down longest =-0.816 monotone_down palin DeepSeek32 [T] =-0.333 valley_1shot =-0.333 valley_1shot =-0.333 valley_1shot =0.333 valley_1shot =0.000 valley_1shot =-1.000 monotone_down GPT-OSS20 [T] =0.333 peak_1shot =1.000 monotone_up =-0.333 valley_1shot =0.333 valley_1shot =-0.816 monotone_down =1.000 monotone_up Gemma31 [NT] =1.000 monotone_up =1.000 monotone_up =nan flat =-0.816 monotone_down =1.000 monotone_up =1.000 monotone_up Gemma31 [T] =1.000 monotone_up =0.333 valley_1shot =nan flat =-0.333 valley_1shot =1.000 monotone_up =1.000 monotone_up Nemotron [T] =0.816 monotone_up =0.333 valley_1shot =nan flat =0.333 valley_1shot =0.333 peak_1shot =1.000 monotone_up Qwen3.5-2 [NT] =1.000 monotone_up =1.000 monotone_up =nan flat =1.000 monotone_up =0.000 peak_1shot =0.333 valley_1shot Qwen3.5-27 [NT] =0.816 monotone_up =0.333 peak_1shot =nan flat =0.333 valley_1shot =0.333 valley_1shot =1.000 monotone_up Qwen3.5-9 [NT] =1.000 monotone_up =0.333 valley_1shot =0.816 monotone_up =1.000 monotone_up =0.333 peak_1shot =1.000 monotone_up Qwen3.6-27 [NT] =1.000 monotone_up =-1.000 monotone_down =-0.816 monotone_down =1.000 monotone_up =1.000 monotone_up =0.333 valley_1shot Qwen3.6-27 [T] =1.000 monotone_up =-1.000 monotone_down =nan flat =-1.000 monotone_down =-1.000 monotone_down =-0.333 valley_1shot Kendall + sparklines (EMA vs shot, condition=all) Figure 7: Kendall-τsparkline grid over shot mode per (model, mode, task, condition). Trajectory classification (monotone_up,monotone_down,peak_1shot,valley_1shot,flat) is shown by the cell color and the embeddedτvalue. Sign-flips between thinking and non-thinking variants of the same model on the same task are the main signal underlying §4.4. 25 DeepSeek32 [NT] DeepSeek32 [T] GPT-OSS20 [T] Gemma31 [NT] Gemma31 [T] Nemotron [T] Qwen3.5-2 [NT] Qwen3.5-27 [NT] Qwen3.5-9 [NT] Qwen3.6-27 [NT] Qwen3.6-27 [T] DeepSeek32 [NT] DeepSeek32 [T] GPT-OSS20 [T] Gemma31 [NT] Gemma31 [T] Nemotron [T] Qwen3.5-2 [NT] Qwen3.5-27 [NT] Qwen3.5-9 [NT] Qwen3.6-27 [NT] Qwen3.6-27 [T] 9/1811/189/1812/189/18 9/185/187/184/18 12/1810/18 15/186/188/185/18 12/187/188/18 8/18 14/1814/18 15/1812/183/18 9/18 McNemar: fraction of significant task×shot comparisons (Bonferroni) 0.0 0.2 0.4 0.6 0.8 1.0 Figure 8: McNemar pairwise significance summary. For each ordered pair of models, the proportion of combined-condition task×shot cells where the row model significantly outperforms the column model after Bonferroni correction within that pair. Thinking and non-thinking pairs are summarized separately in §4.6. count edit span contains longest palin DeepSeek32 [NT] DeepSeek32 [T] GPT-OSS20 [T] Gemma31 [NT] Gemma31 [T] Nemotron [T] Qwen3.5-2 [NT] Qwen3.5-27 [NT] Qwen3.5-9 [NT] Qwen3.6-27 [NT] Qwen3.6-27 [T] Model 0.790.220.040.04 0.04 0.020.04 0.020.150.04 0-shot count edit span contains longest palin 0.040.610.280.330.05 0.04 0.17 0.02 1-shot count edit span contains longest palin 0.171.000.390.020.570.12 0.15 0.21 0.020.01 1.00 4-shot 0.0 0.2 0.4 0.6 0.8 1.0 Parse-error rate Parse-error rate by model × task (B = blank rate) Figure 9: Parse error rates by (model, mode, task, shot). Most models have PER≤ 0.04at 4-shot in any mode; cells with elevated PER appear in the appendix table 12. 26 count edit span contains longest palin DeepSeek32 [NT] DeepSeek32 [T] GPT-OSS20 [T] Gemma31 [NT] Gemma31 [T] Nemotron [T] Qwen3.5-2 [NT] Qwen3.5-27 [NT] Qwen3.5-9 [NT] Qwen3.6-27 [NT] Qwen3.6-27 [T] * ** * * * *** * *** ** 0-shot count edit span contains longest palin DeepSeek32 [NT] DeepSeek32 [T] GPT-OSS20 [T] Gemma31 [NT] Gemma31 [T] Nemotron [T] Qwen3.5-2 [NT] Qwen3.5-27 [NT] Qwen3.5-9 [NT] Qwen3.6-27 [NT] Qwen3.6-27 [T] * * * * * *** ** **** * 1-shot count edit span contains longest palin DeepSeek32 [NT] DeepSeek32 [T] GPT-OSS20 [T] Gemma31 [NT] Gemma31 [T] Nemotron [T] Qwen3.5-2 [NT] Qwen3.5-27 [NT] Qwen3.5-9 [NT] Qwen3.6-27 [NT] Qwen3.6-27 [T] * * * ** ** * **** 4-shot 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 Point-biserial r Figure 10: Point-biserial correlation between input condition and correctness. Cells indicate the strength and direction of the English-vs-random effect per (model, mode, task, shot). Magnitudes are uniformly small (|r| < 0.30). On the cells that survive Bonferroni correction, random is easier in 23 of 25 (§4.2). 0.00.51.01.52.02.53.0 Allowed || tolerance 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of samples count 02468101214 Allowed || tolerance 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of samples edit DeepSeek32 [NT] (english) DeepSeek32 [NT] (random) DeepSeek32 [T] (english) DeepSeek32 [T] (random) GPT-OSS20 [T] (english) GPT-OSS20 [T] (random) Gemma31 [NT] (english) Gemma31 [NT] (random) Gemma31 [T] (english) Gemma31 [T] (random) Nemotron [T] (english) Nemotron [T] (random) Qwen3.5-2 [NT] (english) Qwen3.5-2 [NT] (random) Qwen3.5-27 [NT] (english) Qwen3.5-27 [NT] (random) Qwen3.5-9 [NT] (english) Qwen3.5-9 [NT] (random) Qwen3.6-27 [NT] (english) Qwen3.6-27 [NT] (random) Qwen3.6-27 [T] (english) Qwen3.6-27 [T] (random) Figure 11: Cumulative accuracy as a function of allowed delta tolerance on integer-output tasks at 4-shot. The gap betweenx = 0(exact match) andx = 1(within±1) is the off-by-one error rate. For edit_distance, this gap is large for all models except GPT-OSS-20B, indicating models reason near the correct edit distance but rarely at it. 27 count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 DeepSeek32 [NT] 0-shot 1-shot 4-shot count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 DeepSeek32 [T] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 GPT-OSS20 [T] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 Gemma31 [NT] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 Gemma31 [T] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 Nemotron [T] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 Qwen3.5-2 [NT] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 Qwen3.5-27 [NT] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 Qwen3.5-9 [NT] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 Qwen3.6-27 [NT] count editspan contains longest palin 0.2 0.4 0.6 0.8 1.0 Qwen3.6-27 [T] Per-model capability (EMA_all) Figure 12: Per-(model, mode) capability profile. Radar chart showing 4-shot EMA on each task. The asymmetry between the four near-saturated tasks andedit_distance/index_to_span(the discrimination and floor tasks) is visible as the radial spread. 28 Mean thinking tokens (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 Exact-match accuracy count DeepSeek32 Mean thinking tokens (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 Exact-match accuracy edit Mean thinking tokens (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 Exact-match accuracy span Mean thinking tokens (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 Exact-match accuracy contains Mean thinking tokens (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 Exact-match accuracy longest Mean thinking tokens (log scale) 0.0 0.2 0.4 0.6 0.8 1.0 Exact-match accuracy palin Thinking-cost vs accuracy (4-shot, condition=all) Figure 13: Thinking-cost Pareto frontier. EMA versus mean thinking-token count per (model, task) at 4-shot in thinking mode. Models above-and-left of the Pareto curve are accuracy-efficient; the implications for deployment cost trade-offs are discussed in §4.3. 0.02.55.07.510.012.515.017.520.0 Allowed || tolerance k 0.0 0.2 0.4 0.6 0.8 1.0 P(| | k) count DeepSeek32 Gemma31 GPT-OSS20 Nemotron Qwen3.6-27 0.02.55.07.510.012.515.017.520.0 Allowed || tolerance k 0.0 0.2 0.4 0.6 0.8 1.0 P(| | k) edit Figure 14: Tolerance curves for integer tasks. Reading fromx = 0(exact match) tox = k(within ±k) shows the cumulative accuracy as the tolerance widens. The shape distinguishes models whose errors are off-by-one from those whose errors are arbitrary. 29 0.600.650.700.750.800.85 macro EMA 0.94 0.96 0.98 1.00 1.02 VRR (1 PER) Nemotron Qwen3.6-27 DeepSeek32 Gemma31 GPT-OSS20 Composite leaderboard (size mean thinking tokens) Figure 15: Composite leaderboard Pareto frontier. Macro EMA versus VRR (valid-response rate) per (model, mode) at 4-shot. The composite leaderboard in §4.1, Table 2, ranks models on Macro EMA; this figure visualizes the alternative trade-off where parse-error-tolerant models can be preferred under a deployment budget. 30 DeepSeek32Gemma31GPT-OSS20NemotronQwen3.6-27 DeepSeek32 Gemma31 GPT-OSS20 Nemotron Qwen3.6-27 0.220.000.390.11 0.060.000.330.17 0.500.670.670.50 0.000.060.000.00 0.110.280.060.44 thinking: fraction of condition=all cells where row beats column 0.0 0.2 0.4 0.6 0.8 1.0 dominance score Figure 16: Thinking-mode pairwise dominance. Each cell is the fraction of combined-condition task×shot cells on which the row model significantly beats the column model under the per-cell Bonferroni correction. Thinking and non-thinking comparisons are not pooled, and this matrix does not define a cross-mode ordering. 31 DeepSeek32 Gemma31 Qwen3.5-27 Qwen3.5-2Qwen3.5-9 Qwen3.6-27 DeepSeek32 Gemma31 Qwen3.5-27 Qwen3.5-2 Qwen3.5-9 Qwen3.6-27 0.110.110.560.330.11 0.390.110.830.440.11 0.390.220.830.670.11 0.060.000.000.000.00 0.330.000.000.830.00 0.440.170.060.780.56 non-thinking: fraction of condition=all cells where row beats column 0.0 0.2 0.4 0.6 0.8 1.0 dominance score Figure 17: Non-thinking pairwise dominance. Same definition as Figure 16, restricted to non- thinking runs. Qwen3.5 appears only in this figure; GPT-OSS-20B and Nemotron-Cascade-2 appear only in the thinking-mode figure. 32

Instant research discovery