Paper deep dive
Can Small Models Reason About Legal Documents? A Comparative Study
Snehit Vaddi
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/31/2026, 1:35:56 AM
Summary
This study evaluates the performance of nine sub-10B parameter language models across three legal benchmarks (ContractNLI, CaseHOLD, and ECtHR) using five prompting strategies. The research finds that architectural efficiency, specifically Mixture-of-Experts (MoE) models like Qwen3-A3B, can match the performance of larger commercial models like GPT-4o-mini in legal reasoning. The study highlights that prompting strategies are highly task-dependent, with few-shot prompting being the most consistent, while chain-of-thought prompting significantly improves contract entailment but degrades multiple-choice legal reasoning.
Entities (6)
Relation Signals (4)
Chain-of-thought → degrades → CaseHOLD
confidence 95% · degrades CASEHOLD by 16.0 percentage points
Chain-of-thought → improves → ContractNLI
confidence 95% · It improves CONTRACTNLI by 8.5 percentage points
Few-shot → iseffectiveon → Legal Reasoning
confidence 90% · few-shot prompting emerges as the most consistently effective strategy
Qwen3-A3B → outperforms → GPT-4o-mini
confidence 90% · surpassing it on legal holding identification
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy. We evaluate whether sub-10B parameter models can serve as practical alternatives by testing nine models across three legal benchmarks (ContractNLI, CaseHOLD, and ECtHR) using five prompting strategies (direct, chain-of-thought, few-shot, BM25 RAG, and dense RAG). Across 405 experiments with three random seeds per configuration, we find that a Mixture-of-Experts model activating only 3B parameters matches GPT-4o-mini in mean accuracy while surpassing it on legal holding identification, and that architecture and training quality matter more than raw parameter count. Our largest model (9B parameters) performs worst overall. Chain-of-thought prompting proves sharply task-dependent, improving contract entailment but degrading multiple-choice legal reasoning, while few-shot prompting emerges as the most consistently effective strategy. Comparing BM25 and dense retrieval for RAG, we find near-identical results, suggesting the bottleneck lies in the language model's utilization of retrieved context rather than retrieval quality. All experiments were conducted via cloud inference APIs at a total cost of $62, demonstrating that rigorous LLM evaluation is accessible without dedicated GPU infrastructure.
Tags
Links
- Source: https://arxiv.org/abs/2603.25944v1
- Canonical: https://arxiv.org/abs/2603.25944v1
Trouble viewing inline? Open PDF directly →
Full Text
48,953 characters extracted from source content.
Expand or collapse full text
Preprint. Under review. Can Small Models Reason About Legal Documents? A Comparative Study Snehit Vaddi Independent Researcher vaddisnehit@gmail.com Abstract Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy. We evaluate whether sub-10B parameter models can serve as practical alterna- tives by testing nine models across three legal benchmarks (ContractNLI, CaseHOLD, and ECtHR) using five prompting strategies (direct, chain-of- thought, few-shot, BM25 RAG, and dense RAG). Across 405 experiments with three random seeds per configuration, we find that a Mixture-of- Experts model activating only 3B parameters matches GPT-4o-mini in mean accuracy while surpassing it on legal holding identification, and that archi- tecture and training quality matter more than raw parameter count. Our largest model (9B parameters) performs worst overall. Chain-of-thought prompting proves sharply task-dependent, improving contract entailment but degrading multiple-choice legal reasoning, while few-shot prompting emerges as the most consistently effective strategy. Comparing BM25 and dense retrieval for RAG, we find near-identical results, suggesting the bot- tleneck lies in the language model’s utilization of retrieved context rather than retrieval quality. All experiments were conducted via cloud inference APIs at a total cost of $62, demonstrating that rigorous LLM evaluation is accessible without dedicated GPU infrastructure. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities on legal tasks, from passing the bar exam (Katz et al., 2024) to answering law school questions (Choi et al., 2023). Yet deploying frontier models for legal applications raises practical concerns: API costs scale with query volume, proprietary models introduce data privacy risks for sensitive legal documents, and latency requirements in production systems favor smaller, locally deployable models. Meanwhile, a new generation of sub-10B parameter models, including Mixture-of-Experts (MoE) architectures that activate only a fraction of their parameters per token (Qwen Team, 2025), promises competitive performance at dramatically lower cost. This motivates four research questions: RQ1 Can sub-10B models match commercial API models on legal reasoning tasks? RQ2 How does the effectiveness of prompting strategies vary across legal task types? RQ3 What is the minimum viable model size for legal reasoning, and does architecture (e.g., MoE) matter more than raw parameter count? RQ4 Does retrieval augmentation improve small-model legal reasoning, and does the retrieval method (sparse vs. dense) matter? Prior work has addressed aspects of these questions, but with important gaps. LEGALBENCH (Guha et al., 2023) tested 20 models across 162 tasks but focused on large models with a single prompting approach (RQ2 unaddressed). Arvin (2025) studied scaling effects on CASEHOLD but examined only one dataset and one prompting strategy (RQ2, RQ4 unaddressed). Chalkidis (2023) evaluated GPT-3.5 on LEXGLUE but tested a single model 1 arXiv:2603.25944v1 [cs.CL] 26 Mar 2026 Preprint. Under review. in zero-shot mode (RQ1, RQ3 unaddressed). Studies comparing small and large models (Pecher et al., 2025) have focused on general text classification rather than the specialized reasoning demands of legal text. These questions have not been jointly examined: how prompting strategies interact with model size and architecture across diverse legal tasks. We address this gap with a comprehensive evaluation of nine models (seven open-weight, two commercial API) across three legal benchmarks spanning distinct reasoning types: contract entailment (CONTRACTNLI; Koreeda & Manning 2021), legal holding identification (CASEHOLD; Zheng et al. 2021), and human rights violation prediction (ECTHR; Chalkidis et al. 2019; 2022). We evaluate five prompting strategies: direct, chain-of-thought, few-shot, BM25 retrieval-augmented generation, and dense retrieval-augmented generation. Our experimental design comprises 405 experiments (9 models×5 strategies×3 datasets× 3 seeds), evaluated on 500 stratified samples per dataset (225 for ECTHR) with bootstrap confidence intervals, at a total cost of $62 using cloud inference APIs (Together AI, OpenAI, Anthropic) without dedicated GPU infrastructure. Our study yields four key findings: 1. MoE efficiency for legal reasoning. Qwen3-A3B, a Mixture-of-Experts model activating only 3B of its 30B parameters per token, matches GPT-4o-mini in mean accuracy (46.5% vs. 47.2%) and surpasses it on CASEHOLD with few-shot prompting (71.2% vs. 67.9%). This demonstrates that architectural efficiency can substitute for raw parameter count in legal reasoning. 2. Task-dependent prompting effects. Chain-of-thought prompting is not universally beneficial for legal tasks. It improves CONTRACTNLI by 8.5 percentage points (where logical entailment reasoning is required) but degrades CASEHOLD by 16.0 percentage points (where verbose reasoning disrupts answer selection) and hurts ECTHR multi-label classification. Few-shot prompting is the most consistently effective strategy across all tasks and models. 3.Architecture matters more than parameter count. Nemotron-9B (9B parameters) is our worst performer (17.7% mean accuracy), underperforming models one-third its size. This finding challenges the assumption that larger models always perform better and highlights the importance of training methodology and architecture design. 4.Retrieval method is secondary to retrieval utility. Our ablation comparing BM25 (sparse) and dense retrieval (sentence-transformers) finds no significant difference between the two methods. RAG helps CONTRACTNLI but hurts CASEHOLD regardless of retrieval method, suggesting that retrieval utility is task-dependent rather than retrieval-method- dependent. These findings provide actionable guidance for legal AI practitioners selecting models and strategies for deployment, and contribute to the broader understanding of how prompting strategies interact with model architecture on domain-specific reasoning tasks. 1 2 Related Work Legal NLP benchmarks. Standardized benchmarks have consolidated evaluation across legal reasoning types, from document-level entailment (Koreeda & Manning, 2021) and legal holding identification (Zheng et al., 2021) to multi-task evaluation spanning diverse jurisdictions (Chalkidis et al., 2022; Guha et al., 2023; Fei et al., 2024). While these benchmarks established strong fine-tuned baselines, the advent of LLMs raises the question of whether prompting alone can achieve competitive performance. Chalkidis (2023) showed that zero- shot GPT-3.5 underperforms fine-tuned Legal-BERT on all LEXGLUE tasks, and Arvin (2025) evaluated eight LLMs on CASEHOLD with a memorization test via citation anonymization. However, these studies target either single datasets or single prompting approaches. 1 Code and experimental results will be released upon publication. We note concurrent work on test-time scaling for legal reasoning (Hu et al., 2025) and legal RAG evaluation (Park et al., 2025), which examine complementary aspects of LLM-based legal reasoning. 2 Preprint. Under review. Table 1: Comparison with closely related work. † Includes both BM25 and dense retrieval. StudyModels Tasks Strat. RAG Seeds Chalkidis (2023)171×1 Guha et al. (2023)201621×1 Arvin (2025)811×1 Pecher et al. (2025)6+4+2×3+ Ours935 † ✓3 Small language models. Architectural innovations have driven rapid progress in the sub-10B regime: sliding-window attention (Jiang et al., 2023), scaled instruction-tuning at the 8B tier (Llama Team, Meta AI, 2024), and Mixture-of-Experts variants that achieve dense- model quality with fewer active parameters (Yang et al., 2024; Qwen Team, 2025). Whether such efficiency translates to competitive performance on specialized legal reasoning tasks is an open question. Domain-adapted models like Legal-BERT (Chalkidis et al., 2020) and SaulLM (Colombo et al., 2024a;b) show fine-tuning benefits, while Pecher et al. (2025) found fine-tuned small models match zero-shot large LLMs on general text classification. We investigate what general-purpose small models can achieve through prompting alone. Prompting strategies.Chain-of-thought prompting (Wei et al., 2022) showed that interme- diate reasoning steps improve LLM performance, but a meta-analysis by Sprague et al. (2024) found benefits concentrated in mathematical and symbolic reasoning. Retrieval-augmented generation (Lewis et al., 2020) offers a complementary approach, particularly relevant given that LLMs hallucinate on 58–88% of legal queries (Dahl et al., 2024; Magesh et al., 2025). Yet legal-domain retrieval proves challenging for both sparse and dense methods (Pipitone & Alami, 2024). We systematically evaluate direct, CoT, few-shot, and RAG strategies (including both BM25 and dense retrieval) across three legal tasks and nine models. Table 1 summarizes how our study compares to closely related work. 3 Methodology We evaluate nine language models across three legal benchmarks using five prompting strategies, totaling 405 experiments. The following subsections describe the models (§3.1), datasets (§3.2), prompting strategies (§3.3), and evaluation protocol (§3.4). 3.1 Models We select nine models spanning three categories: seven open-weight models (3B–9B param- eters) served via Together AI, and two commercial API models (GPT-4o-mini via OpenAI, Claude 3.5 Haiku via Anthropic). We select models based on cloud API availability, recency (all released 2023–2025), and coverage of the 3B–9B parameter range across distinct model families. Table 2 summarizes the models and their characteristics. The open-weight models span six families: Llama 3.2/3.1 (Llama Team, Meta AI, 2024), Qwen 2.5/3 (Yang et al., 2024; Qwen Team, 2025), Mistral (Jiang et al., 2023), Gemma (Gemma Team, Google DeepMind, 2025), and Nemotron (Muralidharan et al., 2024). Qwen3-A3B is a Mixture-of-Experts model activating 3B of 30B total parameters per token. GPT-4o-mini (OpenAI, 2024) and Claude 3.5 Haiku (Anthropic, 2024) serve as commercial API base- lines. All models are accessed via cloud inference APIs (Together AI, OpenAI, Anthropic) without dedicated GPU infrastructure, eliminating hardware-dependent confounds and ensuring reproducibility. We enforce a uniform context window of 8,192 tokens to ensure fair comparison. 3 Preprint. Under review. ModelArch.ParamsProviderCtx. Llama-3.2-3BDense3BTogether128K Gemma-3n-E4BDense4B eff Together32K Qwen2.5-7BDense7BTogether128K Mistral-7BDense7BTogether32K Llama-3.1-8BDense8BTogether128K Nemotron-9BDense9BTogether32K Qwen3-A3BMoE3B a Together128K GPT-4o-miniUndiscl.Undiscl.OpenAI128K Claude 3.5 HaikuUndiscl.Undiscl.Anthropic200K Table 2: Models evaluated in our study, spanning 3B–9B parameters for open-weight models and two commercial API services. All models are instruction-tuned variants ac- cessed via cloud APIs. All context windows are capped at 8,192 tokens in our experiments. eff Gemma-3n-E4B uses net-efficient architecture with 4B effective parameters. a Qwen3-A3B is a Mixture-of-Experts model activating 3B of 30B total parameters per token. DatasetTask TypeLabelsNJuris.Chance CONTRACTNLINLI3500US33.3% CASEHOLDMC (5-way)5500US20.0% ECTHR Task AMulti-label11 arts.225EUvaries Table 3: Dataset characteristics.Nis the number of stratified test samples per experiment. ECTHR Task A yields 225 samples due to the high number of unique multi-label combina- tions (51) under stratified sampling. Chance denotes random baseline accuracy; ECTHR chance varies by article combination. 3.2 Datasets We evaluate on three English-language legal benchmarks that span distinct reasoning types, label structures, and jurisdictions. Table 3 summarizes their characteristics. CONTRACTNLI. This benchmark (Koreeda & Manning, 2021) tests natural language inference on contract clauses: given a contract clause (premise) and a hypothesis about its content, models must classify the relationship as Entailment, Contradiction, or Not Mentioned. CASEHOLD.This benchmark (Zheng et al., 2021) tests legal holding identification: given a court opinion excerpt with a masked citation, models select the correct legal holding from five options (A–E). This requires understanding precedent relationships and distinguishing holdings from dicta. ECTHR Task A.This benchmark (Chalkidis et al., 2019; 2022) is a multi-label classification task: given the facts of a European Court of Human Rights case, models predict which articles of the Convention were violated (from 11 possible articles, or none). This multi- label setting is inherently more challenging than the single-label tasks above, as models must predict the exact subset of violated articles from among 2 11 =2,048 possible label combinations. Sampling. For each experiment, we draw a stratified random sample from the test split, with the seed controlling sample selection. CONTRACTNLI and CASEHOLD yield 500 samples (3 and 5 classes, respectively), balancing statistical power with API cost. ECTHR yields 225 samples because its 51 unique multi-label combinations limit stratified allocation to⌊500/51⌋ =9 samples per combination, with many combinations having fewer than 9 test examples. 4 Preprint. Under review. 3.3 Prompting Strategies We evaluate five prompting strategies that represent the primary approaches available to practitioners deploying LLMs without fine-tuning. Direct (zero-shot).The model receives the legal text, a question, and a format instruction specifying the expected answer form. This serves as the baseline measuring raw model capability. Chain-of-Thought (CoT).Following Wei et al. (2022), we augment the direct prompt with an explicit reasoning scaffold: “Think through this step by step: (1) identify key legal concepts, (2) analyze their relation to the question, (3) consider relevant principles, (4) provide your answer.” We allocate 512 output tokens for CoT (vs. 128 for direct) to accommodate the reasoning chain. Few-shot (3-shot). Following the in-context learning paradigm (Brown et al., 2020), we prepend three labeled examples drawn from the training split via stratified sampling, ordered by increasing document length. We use the same three examples for all queries within a dataset-seed combination, ensuring consistency across models. Drawing examples from the training split (rather than the validation or test split) avoids data leakage. BM25 RAG. We retrieve the top-3 passages from the training split using BM25 sparse retrieval (Robertson & Zaragoza, 2009) and prepend them as “reference material” before the input document. The prompt instructs the model to use both the retrieved context and the primary document. We truncate retrieved passages that exceed the remaining context budget after accounting for the input document and output allocation. Dense RAG. We replace BM25 with dense retrieval usingall-MiniLM-L6-v2(Reimers & Gurevych, 2019), keeping all other settings identical to BM25 RAG. Both methods use the same prompt template, corpus, and context budget; see §4.5 for the controlled comparison. Full prompt templates for all strategies are provided in Appendix B. 3.4 Evaluation Protocol Answer extraction. Since all models produce free-form text, we apply a four-stage ex- traction pipeline: (1) exact match against valid labels, (2) normalized keyword matching (case-insensitive), (3) positionally first valid label if multiple are present, (4) parseerror if no valid label is found. Parse errors are counted as incorrect predictions. We report parse error rates per model as a measure of instruction-following capability. Metrics.Our primary metric is accuracy (exact match). For CONTRACTNLI, we addition- ally report macro F1 across the three classes. For ECTHR, we report both subset accuracy (exact match on the full predicted article set) and per-article macro F1, as subset accuracy pe- nalizes partially correct predictions. We do not compute F1 for CASEHOLD, as its balanced 5-way classification makes accuracy and F1 equivalent. Statistical analysis.We repeat each model-strategy-dataset configuration with three seeds (42, 123, 456) that control the stratified data sampling from the test split. All inference uses greedy decoding (temperature=0) to ensure deterministic outputs, so the seeds induce variance solely through test set composition rather than model stochasticity. We report mean accuracy across seeds with 95% bootstrap confidence intervals computed by resampling the Nindividual predictions within each experiment (10,000 bootstrap iterations, pooled across seeds). For key pairwise comparisons, we use the paired bootstrap test (Berg-Kirkpatrick et al., 2012) and McNemar ’s test with Bonferroni correction. We report effect sizes as Cohen’s d. Experimental scale. The full evaluation comprises 405 experiments: 324 in the main evaluation (9 models×4 strategies×3 datasets×3 seeds) and 81 in the dense RAG 5 Preprint. Under review. ModelParamsDirectFew-ShotCoTRAG BM25 RAG Dense Mean GPT-4o-miniAPI.498.499.425.471.470.472 Qwen3-A3B3B ∗ .503.521.393.453.454.465 Qwen2.5-7B7B.452.468.325.445.440.426 Mistral-7B-v0.37B.447.440.336.427.411.412 Llama-3.1-8B8B.439.425.366.381.381.398 Gemma-3n-E4B4B † .374.440.316.363.357.370 Claude 3.5 HaikuAPI.358.339.445.313.308.353 Llama-3.2-3B3B.287.303.272.300.300.292 Nemotron-9B9B.111.139.251.189.197.177 Table 4: Mean accuracy across 3 datasets and 3 seeds (9 values per cell). Models sorted by mean accuracy. Best per-strategy value in bold, second-bestunderlined. ∗ 3B active parameters (MoE). † 4B effective parameters. Dashed line separates models above and below the 0.35 threshold. retrieval comparison (9 models×3 datasets×3 seeds). We conduct all inference using cloud inference APIs (Together AI, OpenAI, Anthropic) without dedicated GPU infrastructure, at a total cost of US$62, totaling approximately 165,375 API calls. We set maximum generation length per strategy: 128 tokens (direct, few-shot), 256 tokens (RAG), and 512 tokens (CoT). 4 Results We present results from 405 experiments spanning 9 models (see Table 2), 5 prompting strategies, 3 legal benchmarks, and 3 random seeds, totaling 165,375 individual predictions. All reported values are means across 3 seeds; 95% bootstrap confidence intervals (10,000 resamples) are provided for key comparisons. 4.1 Overall Model Performance Table 4 presents the main results matrix: accuracy by model and prompting strategy, averaged across all three datasets and seeds. We find that GPT-4o-mini achieves the highest overall accuracy (47.2%), closely followed by Qwen3-A3B (46.5%), a mixture-of-experts model with only 3B active parameters. The gap between these two is not statistically significant when pooling across all conditions (p =0.19, paired bootstrap). Notably, Qwen3-A3B outperforms all dense 7–8B models (Qwen2.5-7B, Mistral-7B, Llama-3.1-8B), suggesting that sparse MoE architectures offer a favorable efficiency–performance tradeoff for legal reasoning. Nemotron-9B, despite having the largest nominal parameter count (9B), ranks last at 17.7%. This underscores how architecture and training regime matter more than raw parameter count for legal NLU tasks. We next examine how these aggregate patterns break down across individual datasets. 4.2 Per-Dataset Results We observe that performance varies dramatically across datasets, reflecting the distinct cognitive demands of each task (Table 5). For reference, random baselines yield 33.3% on CONTRACTNLI (3 classes) and 20.0% on CASEHOLD (5 choices). CONTRACTNLI. This is the most tractable task: six of nine models exceed 65% accuracy, with Qwen3-A3B achieving the highest score (77.5%). The three-class NLI format aligns well with the instruction-following capabilities of modern LLMs. CASEHOLD.This task shows wider model separation. GPT-4o-mini leads at 62.2%, with Llama-3.1-8B (57.8%), Qwen3-A3B (55.8%), and Qwen2.5-7B (54.4%) forming a competitive 6 Preprint. Under review. ModelParamsCONTRACTNLICASEHOLDECTHRMean GPT-4o-miniAPI.766[.756, .776].622 [.611, .633].029 [.024, .035].472 Qwen3-A3B3B ∗ .775 [.765, .784].558 [.548, .569].063 [.055, .071].465 Qwen2.5-7B7B.683 [.672, .693].544 [.533, .555].051[.044, .059].426 Mistral-7B7B.686 [.676, .697].524 [.513, .535].027 [.021, .032].412 Llama-3.1-8B8B.578 [.567, .589].578[.566, .589].039 [.033, .046].398 Gemma-3n-E4B4B † .669 [.658, .680].411 [.400, .422].029 [.024, .035].370 Claude 3.5 HaikuAPI.672 [.664, .690].344 [.336, .385].042 [.035, .050].353 Llama-3.2-3B3B.409 [.398, .420].451 [.439, .462].017 [.013, .022].292 Nemotron-9B9B.305 [.294, .315].204 [.195, .213].023 [.019, .028].177 Table 5: Accuracy by model and dataset, averaged across 5 strategies and 3 seeds (15 values per cell). 95% bootstrap CIs in brackets (pooling all predictions per model-dataset combination). Best in bold, second-best underlined. StrategyCNLICHECTHRMean Direct.570.537.049.385 Few-Shot.608.532.051.397 CoT.655.377.011.348 RAG BM25 .627 .456.031.371 RAG Dense .619.450.036.369 Table 6: Mean accuracy by strategy and dataset, averaged across 9 models and 3 seeds (27 values per cell). CNLI = CONTRACTNLI, CH = CASEHOLD. Best in bold, second-best underlined. middle tier. The five-choice multiple-choice format reveals a clear accuracy gap between stronger and weaker models. ECTHR (multi-label classification).This task proves near-impossible for all models under prompting alone. Qwen3-A3B achieves the best result at 6.3% subset accuracy, meaning fewer than 1 in 15 predictions exactly match the full set of violated articles. We attribute this to the inherent difficulty of mapping free-text reasoning to fine-grained multi-label predictions over 11 ECHR articles. Given these dataset-level differences, we turn to how prompting strategies interact with task type. 4.3 Prompting Strategy Analysis Table 6 reveals that prompting strategy effectiveness is strongly task-dependent. Few-shot is the most consistent strategy. We find that across tasks, few-shot achieves the highest overall mean (39.7%) and never ranks worst on any dataset. It provides mod- erate gains on CONTRACTNLI (+3.8 percentage points (p) over direct) without the sharp CASEHOLD penalties of CoT. Chain-of-thought is strongly task-dependent.CoT achieves the highest CONTRACTNLI accuracy (65.5%) but the lowest CASEHOLD accuracy (37.7%), a reversal of 28p. This interaction is analyzed in detail in Section 4.4. RAG provides modest benefits on NLI but hurts multiple-choice.Both BM25 and dense retrieval improve over direct on CONTRACTNLI (+4.9–5.7p) but degrade CASEHOLD accuracy by 8.1–8.7p, suggesting retrieved context introduces noise for tasks requiring precise citation matching. 7 Preprint. Under review. ModelCNLICASEHOLD CoT− DirCoT− Dir Nemotron-9B+43.8 ∗ +0.6 Claude 3.5 Haiku+14.2 ∗ +19.2 ∗ Llama-3.2-3B+14.0 ∗ −21.0 ∗ Gemma-3n-E4B+4.8 ∗ −17.8 ∗ Qwen2.5-7B+2.4 ∗ −37.2 ∗ Qwen3-A3B+1.0 −32.0 ∗ GPT-4o-mini+0.0 −18.6 ∗ Mistral-7B−4.2 ∗ −25.2 ∗ Llama-3.1-8B −9.0 ∗ −8.8 ∗ Table 7: Accuracy change from direct to CoT on seed 42 (percentage points,n=500 per condition). CNLI = CONTRACTNLI. Paired bootstrap significance test: ∗ p<0.001, ∗ p< 0.01, ∗ p< 0.05. Results are consistent across seeds (see Appendix). 4.4 Chain-of-Thought Effects by Task Our most striking finding is the divergent effect of chain-of-thought prompting across tasks. Table 7 shows the accuracy change when switching from direct to CoT prompting. CoT universally harms CASEHOLD. We observe that seven of nine models show sta- tistically significant accuracy drops (p<0.001) when using CoT on CASEHOLD, with degradations ranging from−8.8p (Llama-3.1-8B) to−37.2p (Qwen2.5-7B). The exceptions are Claude 3.5 Haiku, which is the only model where CoT significantly improves CASEHOLD performance (+19.2p,p<0.001), and Nemotron (+0.6p, not significant), which is already at floor performance. We hypothesize that Haiku’s instruction-tuning enables it to maintain answer format fidelity during extended reasoning, while other models lose track of the multiple-choice format during CoT generation. CoT helps weak models on NLI but not strong ones.On CONTRACTNLI, CoT produces dramatic gains for Nemotron (+43.8p), which suffers from severe parse errors under direct prompting. CoT’s structured output effectively rescues instruction-following failures. However, we find that for models already performing well (GPT-4o-mini, Qwen3-A3B), CoT provides no benefit. For Mistral-7B and Llama-3.1-8B, CoT actually hurts NLI performance (p< 0.05). Having established the task-dependent nature of CoT, we examine whether retrieval method choice similarly affects performance. 4.5 BM25 vs. Dense Retrieval Our controlled ablation comparing BM25 (sparse) and dense retrieval (all-MiniLM-L6-v2) yields a striking null result: mean accuracy differs by just 0.3 p (37.1% vs. 36.8%), with BM25 slightly ahead on CONTRACTNLI (−0.8 p) and CASEHOLD (−0.6 p) while dense retrieval marginally leads on ECTHR (+0.5 p). Out of 27 paired comparisons, only 1 reaches significance atp<0.01, consistent with the expected false positive rate. This suggests BM25’s lexical matching is as effective as learned dense representations for legal passage retrieval. Detailed per-dataset model×strategy breakdowns are provided in Appendix C. Parse errors. Parse error rates (outputs unmappable to valid labels) vary substantially: Nemotron-9B produces 16.4% unparseable outputs on average (reaching 29.8% on CON- TRACTNLI), explaining much of its poor performance, while GPT-4o-mini and Qwen2.5-7B achieve near-zero rates. Full breakdowns are in Appendix D. 8 Preprint. Under review. 4.6 Comparison to Published Baselines Our best prompted results remain below published fine-tuned baselines: Legal-BERT achieves 75.3% on CASEHOLD with fine-tuning (Chalkidis et al., 2020) vs. our 71.2% (Qwen3-A3B few-shot); SaulLM-7B reaches 83.4% on CONTRACTNLI (Colombo et al., 2024a) vs. our 79.9% (Haiku CoT); and fine-tuned LEXGLUE models achieve macro-F1 above 60% on ECTHR (Chalkidis et al., 2022) vs. our 11.9% subset accuracy. For the two classification tasks, these gaps of 3.5–4.1 p quantify the cost of prompting-only evaluation relative to domain-adapted fine-tuning. 5 Conclusion We presented a systematic evaluation of nine language models across three legal reasoning tasks using five prompting strategies, comprising 405 experiments at a total cost of $62. Our central finding is that open-weight small models can match commercial APIs on legal reasoning: Qwen3-A3B (3B active parameters) matches GPT-4o-mini in mean accuracy (46.5% vs. 47.2%) and surpasses it on CASEHOLD with few-shot prompting (71.2% vs. 67.9%), demonstrating that architectural efficiency can substitute for raw parameter count. Moreover, prompting strategy effectiveness is strongly task-dependent: chain-of-thought helps contract entailment (+8.5 p) but hurts legal holding identification (−16.0 p) and multi-label classification (−3.8 p). Few-shot prompting emerges as the most consistently effective strategy. We also find that architecture and training quality matter more than scale: a 9B dense model (Nemotron) performs worst overall while a 3B-active MoE model outperforms all dense models up to 8B, and BM25 and dense retrieval yield near-identical results (0.3 p mean difference), suggesting the bottleneck lies in the LLM’s utilization of retrieved context rather than retrieval quality. Practical guidance. For practitioners, we recommend: (1) Qwen3-A3B as the best cost- performance option, matching commercial APIs at open-source inference costs; (2) few-shot prompting as the default strategy, with CoT reserved for entailment tasks; (3) BM25 for RAG, as it matches dense retrieval without requiring an embedding model or vector database; and (4) prioritizing well-trained 3–8B models over larger but poorly calibrated alternatives. Future Work. Several directions merit investigation. Comparing prompted general- purpose models against fine-tuned and domain-adapted alternatives (e.g., SaulLM; Colombo et al. 2024a) would clarify when fine-tuning is worth the additional investment. Expanding to additional legal tasks, including statutory reasoning (Blair-Stanek et al., 2023), legal summarization, and judgment prediction, would test generalizability, as would multilingual evaluation using LEXTREME (Niklaus et al., 2023) to assess whether our findings transfer across legal systems and languages. Evaluating models with longer context windows on full-length legal documents (rather than our 8K-truncated inputs) may also reveal different model rankings, particularly for ECTHR cases that average 4.5K tokens. Taken together, our results show that careful model and strategy selection enables open-weight small models to approach commercial API performance on legal reasoning tasks at a fraction of the cost. Limitations Prompting-only scope. We evaluate only prompting-based approaches without fine- tuning; parameter-efficient methods such as LoRA (Hu et al., 2022) substantially outperform prompted models on legal benchmarks (Colombo et al., 2024a), so our results represent a lower bound. We also use a single prompt template per strategy without per-model optimization (Kojima et al., 2022). Limited scope. Our three English-language benchmarks cover contract law (US), case law (US), and human rights (European) but cannot represent all legal reasoning types. We do not evaluate generative tasks (summarization, drafting), non-English jurisdictions, or 9 Preprint. Under review. civil law systems. The ECTHR subset of 225 samples also limits statistical power on that benchmark. Data contamination. We cannot rule out overlap between evaluation data and model training corpora, particularly for CASEHOLD (public judicial opinions) and CONTRACTNLI (SEC filings). Contamination detection is infeasible for closed-weight API models where training data composition is undisclosed. Reproducibility constraints.All models are evaluated via cloud API endpoints, introduc- ing dependence on provider-specific serving configurations (quantization, batching) that may differ from local deployments. We use greedy decoding (temperature=0) and three data-sampling seeds, but API providers may update model weights without notice. We release our evaluation harness to facilitate replication. Ethics Statement Data. We use three publicly available research datasets: CONTRACTNLI (Koreeda & Manning, 2021), CASEHOLD (Zheng et al., 2021), and ECTHR (Chalkidis et al., 2019), derived from published legal documents (SEC filings, judicial opinions, and court rulings, respectively). These datasets do not contain private personal information, and their use for research purposes is consistent with their intended distribution. Cost transparency and accessibility.The total computational cost of our 405 experiments was US$61.88, processed entirely through commercial cloud APIs without dedicated GPU infrastructure. We report detailed per-model and per-provider cost breakdowns to enable other researchers to estimate budgets for similar studies. However, we acknowledge that API-based evaluation creates a dependency on commercial providers, which may present access barriers for researchers in resource-constrained settings or regions with limited API availability. The cost disparity across providers (e.g., $0.81 total for Gemma-3n-E4B vs. $13.43 for Claude 3.5 Haiku) also means that model selection in practice may be influenced by budget constraints rather than purely by task performance. Because we conducted all experiments via cloud APIs with no model training, the direct environmental footprint is limited to inference-time energy consumption. Intended use. We intend this study to inform the NLP and legal AI communities about the capabilities of small language models on legal reasoning tasks. By demonstrating that sub-10B models can approach commercial API-level performance on structured legal tasks, we hope to support efforts toward more accessible and cost-efficient legal NLP tools. Our results should not be interpreted as endorsements of any model for deployment in legal practice. Legal applications of language models carry significant risks, including potential for incorrect legal analysis, hallucinated citations, and encoded biases, and require careful human oversight by qualified legal professionals. Bias considerations. Language models may encode biases present in their training data, including biases related to jurisdiction, legal tradition, socioeconomic factors, and protected characteristics. The ECTHR dataset, which concerns human rights violations, is particularly sensitive in this regard. We evaluate models on classification accuracy but do not audit for systematic biases in model outputs across demographic or jurisdictional dimensions, which we leave as important future work. LLM Disclosure Language model assistance was used during the drafting and editing process; all content, analysis, and experimental results have been verified for accuracy by the authors. The experimental infrastructure, data collection, and statistical analysis were designed and executed by the authors. 10 Preprint. Under review. Reproducibility Statement All experiments use cloud inference APIs with greedy decoding (temperature=0) and specific model identifiers listed in Table 2. Each configuration is evaluated with three ran- dom seeds (42, 123, 456) controlling stratified data sampling, with 95% bootstrap confidence intervals (10,000 resamples). Prompt templates are in Appendix B. Our evaluation harness and analysis scripts will be released upon publication. The total cost of US$62 (Appendix A) makes full replication accessible. References Anthropic. Claude 3.5 Haiku model card. Anthropic Model Card, 2024.https://w. anthropic.com/news/claude-3-5-haiku. Chuck Arvin. Identifying legal holdings with LLMs: A systematic study of performance, scale, and memorization. In Proceedings of the Twenty-first International Conference on Artificial Intelligence and Law (ICAIL). ACM, 2025. Taylor Berg-Kirkpatrick, David Burkett, and Dan Klein. An empirical investigation of statistical significance in NLP. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, p. 995–1005, Jeju Island, Korea, July 2012. Association for Computational Linguistics. Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. Can GPT-3 perform statutory reasoning? In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law (ICAIL). ACM, 2023. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 (NeurIPS), 2020. Ilias Chalkidis. ChatGPT may pass the bar exam soon, but has a long way to go for the LexGLUE benchmark. arXiv preprint arXiv:2304.12202, 2023. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural legal judgment predic- tion in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 4317–4323. Association for Computational Linguistics, 2019. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, p. 2898–2904. Association for Computational Linguistics, 2020. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4310–4330. Association for Computational Linguistics, 2022. Jonathan H. Choi, Kristin E. Hickman, Amy Monahan, and Daniel Schwarcz. ChatGPT goes to law school. Journal of Legal Education, 71(3):387–400, 2023. Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera L ́ ucia Raposo, Sofia Morgado, and Michael Desa. SaulLM-7B: A pioneering large language model for law. arXiv preprint arXiv:2403.03883, 2024a. Pierre Colombo et al. SaulLM-54B & SaulLM-141B: Scaling up domain adaptation for the legal domain. In Advances in Neural Information Processing Systems 37 (NeurIPS), 2024b. 11 Preprint. Under review. Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1): 64–93, 2024. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. LawBench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024. Gemma Team, Google DeepMind. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. Gemma 3n is a variant optimized for on-device inference; seehttps://ai.google. dev/gemma/docs/gemma-3n. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Re, Adam Chilton, et al. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neural Information Processing Systems 36 (NeurIPS): Datasets and Benchmarks Track, 2023. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. Yinghao Hu, Yaoyao Yu, Leilei Gan, Bin Wei, Kun Kuang, and Fei Wu. Evaluating test-time scaling LLMs for legal reasoning: OpenAI o1, DeepSeek-R1, and beyond. In Findings of the Association for Computational Linguistics: EMNLP 2025, p. 13759–13781. Association for Computational Linguistics, 2025. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270):20230254, 2024. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Process- ing Systems 35 (NeurIPS), 2022. Yuta Koreeda and Christopher D. Manning. ContractNLI: A dataset for document-level natural language inference for contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, p. 1907–1919. Association for Computational Linguistics, 2021. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ̈ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ̈ aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33 (NeurIPS), 2020. Llama Team, Meta AI. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, and Daniel E. Ho. Hallucination-free? Assessing the reliability of leading AI legal research tools. Journal of Empirical Legal Studies, 22:216–242, 2025. Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. In Advances in Neural Information Processing Systems 37 (NeurIPS), 2024. Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stuermer, and Ilias Chalkidis. LEXTREME: A multi-lingual and multi-task benchmark for the legal domain. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics, 2023. 12 Preprint. Under review. OpenAI. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. Minhu Park, Hongseok Oh, Eunkyung Choi, and Wonseok Hwang. LRAGE: Legal retrieval augmented generation evaluation tool. arXiv preprint arXiv:2504.01840, 2025. Branislav Pecher, Ivan Srba, and Maria Bielikova. Comparing specialised small and general large language models on text classification: 100 labelled samples to achieve break-even performance. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2025. Nicholas Pipitone and Ghita Houir Alami. LegalBench-RAG: A benchmark for retrieval- augmented generation in the legal domain. arXiv preprint arXiv:2408.10343, 2024. Qwen Team. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics. Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. Zayne Sprague, Fei Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12183, 2024. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35 (NeurIPS), 2022. An Yang et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024. Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. When does pretraining help? Assessing self-supervised learning for law and the CaseHOLD dataset of 53,000+ legal holdings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law (ICAIL), p. 159–168. ACM, 2021. A Experimental Infrastructure and Cost Table 8 provides detailed per-model inference cost and latency breakdowns for all 405 experiments across the three API providers. All models are accessed via cloud API endpoints: Together AI 2 serves the seven open- weight models, the OpenAI API serves GPT-4o-mini, and the Anthropic API serves Claude 3.5 Haiku. The total cost of US$61.88 covers approximately 165,375 individual API calls processing 314.9M input tokens and generating 26.7M output tokens. Together AI accounts for 56% of the total cost ($34.62 for 315 experiments), followed by Anthropic ($13.43 for 45 experiments) and OpenAI ($5.84 for 45 experiments). We use separate thread pools per API provider to maximize throughput while respecting rate limits: 29 concurrent threads for Together AI, 15 for OpenAI, and 3 for Anthropic. The execution framework is resume-safe, skipping completed experiments based on output file existence, which proved essential when Anthropic credit depletion required re-running 23 Haiku experiments with a fresh API key. Dense retrieval embeddings are computed using theall-MiniLM-L6-v2sentence- transformer model and cached to disk, adding negligible cost to the retrieval ablation experiments. 2 https://w.together.ai 13 Preprint. Under review. ModelProviderCostMed.P95 (USD)Lat.Lat. Haiku 3.5Anthropic$13.434.49s12.85s Qwen2.5-7BTogether$11.051.98s5.52s Qwen3-A3BTogether$9.562.10s4.32s Mistral-7BTogether$9.352.26s3.83s Llama-3.1-8BTogether$6.511.08s1.68s GPT-4o-miniOpenAI$5.842.78s6.59s Nemotron-9BTogether$3.202.60s3.28s Llama-3.2-3BTogether$2.133.10s8.61s Gemma-3n-E4BTogether$0.815.46s9.99s Total$61.88 Table 8: Per-model inference cost and latency across all 405 experiments (324 main + 81 ablation). Median and P95 latencies are averaged across all configurations per model. Models sorted by total cost. B Prompting Strategy Templates Figure 1 provides a visual overview of the five prompting strategies evaluated in this study. Each strategy augments the base prompt with different context components before sending it to the model. The direct strategy serves as the baseline, while the remaining strategies add reasoning scaffolds (CoT), in-context examples (few-shot), or retrieved passages (BM25 and dense RAG). Five Prompting Strategies Each strategy augments the base prompt with different context before sending to the LLM Direct (Zero-shot) 128 max output tokens Legal Text Question Format Instruction Baseline: raw model capability Chain-of-Thought (CoT) 512 max output tokens Legal Text Question Reasoning Scaffold Format Instruction Adds step-by-step reasoning instructions Few-shot (3-shot) 128 max output tokens 3 Labeled Examples Legal Text Question Format Instruction Prepends 3 training examples (stratified) BM25 RAG 256 max output tokens BM25 Top-3 Passages Legal Text Question Format Instruction Retrieves 3 passages via sparse retrieval Dense RAG 256 max output tokens Dense Top-3 Passages Legal Text Question Format Instruction Retrieves 3 passages via all-MiniLM-L6-v2 LLM (9 models) temp = 0 8K context Answer Component Legend: Input (legal text / question) In-context examples Reasoning scaffold BM25 retrieved passagesDense retrieved passages Format instruction Figure 1: Overview of the five prompting strategies. Each row shows the prompt compo- nents assembled for a given strategy. All strategies share the core components (legal text, question, format instruction) but differ in the additional context provided. Output token budgets vary by strategy: 128 tokens (direct, few-shot), 256 tokens (RAG), and 512 tokens (CoT). 14 Preprint. Under review. C Detailed Per-Dataset Results Tables 9–11 provide the full model×strategy accuracy for each dataset. Model names are abbreviated; see Table 2 for full names. ModelDirFS CoT R B R D GPT-4o-mini.761.791 .771 .754 .755 Qwen3-A3B.759 .767.776.787 .785 Haiku.657 .618 .799 .638 .645 Mistral-7B.675 .678 .659 .723 .696 Qwen2.5-7B.657 .697 .680 .688 .691 Gemma-3n-E4B .605 .719 .687 .678 .655 Llama-3.1-8B.641 .608 .544 .558 .539 Llama-3.2-3B.273 .421 .432 .474 .447 Nemotron.101 .175 .544 .343 .360 Table 9: CONTRACTNLI accuracy (mean over 3 seeds). Dir = Direct, FS = Few-Shot, R B = RAG BM25 , R D = RAG Dense . Best in bold, second-best underlined. ModelDirFS CoT R B R D Qwen3-A3B.692 .712 .373 .498 .513 GPT-4o-mini.681 .679.499 .634 .615 Qwen2.5-7B.649 .649 .277 .583.563 Mistral-7B.606 .619 .342 .540 .511 Llama-3.1-8B.618 .595 .535.561 .579 Llama-3.2-3B.577 .482 .371 .407 .415 Haiku.343 .281 .537 .285 .275 Gemma-3n-E4B .464 .575 .261 .381 .374 Nemotron.203 .199 .201 .211 .205 Table 10: CASEHOLD accuracy (mean over 3 seeds). Best in bold, second-best underlined. Note the consistent CoT degradation across models, with Haiku as the sole exception. ModelDirFS CoT R B R D Haiku.073 .119 .000 .016 .004 Qwen3-A3B.059 .085.031 .074 .064 Llama-3.1-8B.058 .073 .018 .024 .024 Qwen2.5-7B.050 .059 .019.062.065 Mistral-7B.061 .022 .006 .019 .025 Gemma-3n-E4B .052 .025 .000 .028 .042 GPT-4o-mini.050 .028 .004 .024 .039 Nemotron.028 .042 .007 .013 .027 Llama-3.2-3B.010 .006 .013 .019 .037 Table 11: ECTHR subset accuracy (mean over 3 seeds,n=225 per experiment). Best in bold, second-bestunderlined. Haiku few-shot achieves 11.9%, the only configuration exceeding 10%. D Parse Error Analysis Table 12 reports the proportion of model outputs that could not be mapped to valid answer labels. Nemotron-9B exhibits the highest parse error rates (16.4% overall, reaching 29.8% on CONTRACTNLI), explaining much of its poor performance. Haiku shows elevated CON- TRACTNLI parse errors (13.2%). GPT-4o-mini and Qwen2.5-7B achieve near-zero parse error rates across all conditions, indicating strong instruction-following capabilities. 15 Preprint. Under review. ModelCNLICHECTHRMean GPT-4o-mini.000.000.001.000 Qwen2.5-7B.000.000.001.000 Mistral-7B.003.000.002.002 Llama-3.1-8B.003.000.007.003 Llama-3.2-3B.001.002.010.004 Qwen3-A3B.001.004.035.013 Gemma-3n-E4B.001.005.079.028 Haiku.132.006.038.059 Nemotron.298 † .004.189 † .164 † Table 12: Mean parse error rate by model and dataset (proportion of unparseable outputs). Worst (highest) per column in bold. † Rates exceeding 10%. 16