Paper deep dive

IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text

Rajveer Singh Pall

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 47

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%

Last extracted: 4/26/2026, 11:08:29 PM

Summary

IndiaFinBench is a new evaluation benchmark designed to assess Large Language Model (LLM) performance on Indian financial regulatory text, addressing the gap in non-Western financial NLP. The benchmark consists of 406 expert-annotated question-answer pairs derived from 192 documents from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI). It covers four key reasoning tasks: regulatory interpretation, numerical reasoning, contradiction detection, and temporal reasoning. The study evaluates twelve models, finding that top-tier models like Gemini 2.5 Flash (89.7% accuracy) significantly outperform a non-specialist human baseline (60.0%), while numerical reasoning remains the most challenging task for most models.

Entities (9)

Contradiction Detection · task-type · 100%Gemini 2.5 Flash · large-language-model · 100%Gemma 4 E4B · large-language-model · 100%IndiaFinBench · benchmark · 100%Numerical Reasoning · task-type · 100%Regulatory Interpretation · task-type · 100%Reserve Bank of India · regulatory-body · 100%Securities and Exchange Board of India · regulatory-body · 100%Temporal Reasoning · task-type · 100%

Relation Signals (4)

Gemini 2.5 Flash → achievedaccuracyon → IndiaFinBench

confidence 100% · Gemini 2.5 Flash achieves the highest overall accuracy at 89.7%

IndiaFinBench → evaluates → Gemini 2.5 Flash

confidence 100% · We evaluate twelve models under zero-shot conditions on the full 406-item benchmark... accuracy ranging from 70.4% (Gemma 4 E4B) to 89.7% (Gemini 2.5 Flash).

IndiaFinBench → includestask → Numerical Reasoning

confidence 100% · spanning four task types: regulatory interpretation (174 items), numerical reasoning (92 items), contradiction detection (62 items), and temporal reasoning (78 items).

IndiaFinBench → usesdocumentsfrom → Securities and Exchange Board of India

confidence 100% · 192 documents sourced directly from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We introduce IndiaFinBench, to our knowledge the first publicly available evaluation benchmark for assessing large language model (LLM) performance on Indian financial regulatory text. Existing financial NLP benchmarks draw exclusively from Western financial corpora (SEC filings, US earnings reports, and English-language financial news), leaving a significant gap in coverage of non-Western regulatory frameworks. IndiaFinBench addresses this gap with 406 expert-annotated question-answer pairs drawn from 192 documents sourced from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI), spanning four task types: regulatory interpretation (174 items), numerical reasoning (92 items), contradiction detection (62 items), and temporal reasoning (78 items). Annotation quality is validated through a model-based secondary pass (kappa=0.918 on contradiction detection) and a 60-item human inter-annotator agreement evaluation (kappa=0.611; 76.7% overall agreement). We evaluate twelve models under zero-shot conditions, with accuracy ranging from 70.4% (Gemma 4 E4B) to 89.7% (Gemini 2.5 Flash). All models substantially outperform a non-specialist human baseline of 60.0%. Numerical reasoning is the most discriminative task, with a 35.9 percentage-point spread across models. Bootstrap significance testing (10,000 resamples) reveals three statistically distinct performance tiers. The dataset, evaluation code, and all model outputs are available at this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

47,010 characters extracted from source content.

Expand or collapse full text

IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text Rajveer Singh Pall Gyan Ganga Institute of Technology and Sciences, Jabalpur, India rajveer.singhpall.cb23@ggits.net Abstract We introduce IndiaFinBench, to our knowledge the first publicly available evaluation benchmark for assessing large language model (LLM) performance on Indian financial regulatory text. Existing financial NLP benchmarks draw exclusively from Western financial corpora—SEC filings, US earnings reports, and English-language financial news—leaving a significant gap in coverage of non-Western regulatory frameworks. IndiaFinBench addresses this gap with 406 expert-annotated question-answer pairs drawn from 192 documents sourced directly from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI), spanning four task types: regulatory interpretation (174 items), numerical reasoning (92 items), contradiction detection (62 items), and temporal reasoning (78 items). Annotation quality is validated through a model-based secondary pass (κ = 0.918 on contradiction detection) and a separate 60- item human inter-annotator agreement evaluation (κ = 0.611 for contradiction detection; 76.7% overall agreement). We evaluate twelve models under zero-shot conditions on the full benchmark, with accuracy ranging from 70.4% (Gemma 4 E4B) to 89.7% (Gemini 2.5 Flash; 93.3% after scoring-pipeline correction, see Appendix D). All models substantially outperform a human baseline (non-specialist) of 60.0% (n = 30; 95% Wilson CI: [42.3%, 75.4%]), though the small baseline sample means comparisons with bottom-tier models should be treated as indicative. Numerical reasoning is the most discriminative task, with a 35.9 percentage-point spread across models. Bootstrap significance testing (10,000 resamples) reveals three statistically distinct performance tiers. A qualitative error analysis identifies temporal reasoning failure as the dominant failure mode for top-performing models and domain knowledge failure for smaller models. The dataset, evaluation code, and all model outputs are publicly available at https://github.com/rajveerpall/IndiaFinBench and https://huggingface.co/datasets/Rajveer-code/IndiaFinBench. 1. Introduction Large language models have demonstrated broad capabilities across reasoning, question answering, and natural language understanding. Yet their ability to handle domain-specific regulatory text—particularly outside the Western financial context— remains poorly characterised. Evaluation benchmarks are the primary instrument by which the research community tracks model capabilities, and virtually all established financial NLP benchmarks are built from US or European regulatory sources. This gap has concrete practical consequences. India's financial regulatory architecture is governed by SEBI circulars, RBI monetary policy directives, and a dense network of amendment chains between instruments. These documents present challenges that are qualitatively distinct from those captured in existing benchmarks. Indian regulatory text routinely embeds numerical thresholds in prose—capital adequacy ratios, upfront margin requirements, dividend payout limits—references chains of superseding circulars that require temporal reasoning to untangle, and employs jurisdiction-specific terminology (LODR, PMLA, SFB, AIF, FEMA) that models trained predominantly on Western corpora may not reliably interpret. We introduce IndiaFinBench, an evaluation benchmark designed to make these challenges measurable. The benchmark was constructed entirely from publicly available primary sources and validated via both a model-based secondary quality pass and a separate human inter-annotator agreement evaluation on 60 items. Our contributions are: 1. A new benchmark dataset of 406 expert-annotated QA pairs across four task types, drawn from 192 SEBI and RBI documents spanning 1992–2026. 2. A comprehensive zero-shot evaluation of twelve contemporary LLMs on the full 406-item benchmark, revealing three performance tiers and substantial inter-task variation. 3. Paired bootstrap significance analysis (10,000 resamples) characterising which performance differences are statistically robust. 4. Dual-layer annotation validation: a model-based secondary quality pass and a separate 60-item human inter-annotator agreement evaluation across all four task types. 5. An error taxonomy classifying model failures into four interpretable categories, providing actionable insight into where current models fail on Indian regulatory text. 6. A public release of the dataset, evaluation code, and all model predictions, supporting reproducibility and ongoing research in multilingual financial NLP. 2. Related Work 2.1 Financial NLP Benchmarks The financial NLP community has produced several influential evaluation resources, all focused on Western financial text. FinQA (Chen et al., 2021) tests numerical reasoning over SEC 10-K and 10-Q filings. ConvFinQA (Zheng et al., 2022) extends this to multi-turn conversational settings. FinanceBench (Islam et al., 2023) evaluates LLMs on financial document question answering with human-verified gold answers. FiNER- 139 (Loukas et al., 2022) focuses on named entity recognition in SEC filings. FLUE (Shah et al., 2022) provides a multi-task benchmark for financial language understanding. A common limitation of all these benchmarks is their exclusive reliance on US or European financial text. FinanceBench is the closest work in spirit to ours, evaluating LLMs on financial document QA with human-verified gold answers, but covers publicly listed US companies only. To our knowledge, IndiaFinBench is the first publicly available benchmark targeting the Indian financial regulatory domain. 2.2 Regulatory and Legal Text Understanding Legal and regulatory text understanding has received growing attention in NLP. CUAD (Hendrycks et al., 2021) focuses on contract clause extraction. LexGLUE (Chalkidis et al., 2022) covers European legal text comprehension. Indian legal NLP has seen recent work with ILDC (Malik et al., 2021) for court judgment prediction. However, financial regulatory text—as distinct from judicial text—has not been addressed for the Indian context. Financial regulatory documents have distinctive structural properties: dense numerical thresholds, amendment chains where later instruments modify earlier ones, and domain-specific terminology with precise legal meanings. 2.3 LLM Evaluation Methodology Our evaluation follows the extractive QA paradigm established by SQuAD (Rajpurkar et al., 2016), with context passages provided directly to the model under zero- shot, context-only constraints. This design isolates the model's ability to reason about regulatory text rather than recall memorised facts, making the benchmark robust to training data contamination. General-domain evaluation frameworks such as MMLU (Hendrycks et al., 2021) and HELM (Liang et al., 2022) provide broad coverage but do not include Indian financial regulatory language, motivating the construction of domain- and geography-specific benchmarks. 3. Dataset Construction 3.1 Source Document Collection We collected 192 regulatory documents from two official Indian government sources: the Securities and Exchange Board of India (sebi.gov.in) and the Reserve Bank of India (rbi.org.in). Documents were downloaded using a custom Python scraping pipeline and converted to clean text using pdfplumber, which handles the multi-column layouts and embedded tables common in Indian regulatory PDFs particularly well. The corpus spans documents from 1992 to 2026 and covers the following regulatory categories: Source Count Document Types SEBI 92 Circulars, master circulars, regulations, orders RBI 100 Circulars, monetary policy statements, master directions Total 192 — Table 1. Source document corpus composition. 3.2 Task Types IndiaFinBench defines four task types, each probing a distinct reasoning capability: Regulatory Interpretation (REG, 174 items). Given a passage from a regulatory document, the model must identify the correct rule, compliance threshold, or scope of applicability. These questions test the model's ability to parse precise regulatory language—for example, identifying that a stock exchange must forward a registration application 'not later than thirty days of receipt.' Numerical Reasoning (NUM, 92 items). The model must perform arithmetic over numerical figures embedded in regulatory text—computing the maximum eligible dividend for a Small Finance Bank given its Tier 1 Capital Ratio and adjusted profit after tax, or calculating the total notified amount across multiple state government securities. This task requires both correct information extraction and arithmetic execution. Contradiction Detection (CON, 62 items). Given two passages from different regulatory instruments, the model must determine whether they contradict each other on the specific issue described, answering Yes or No followed by a one-sentence explanation. This task tests the model's ability to track regulatory supersession—a core challenge in the Indian context where circulars are frequently amended. Temporal Reasoning (TMP, 78 items). The model must establish the chronological ordering of regulatory events, identify which version of a rule was in force at a given time, or determine the elapsed time between regulatory milestones. This task is particularly challenging because Indian regulatory documents frequently reference earlier instruments by date, requiring the model to maintain a consistent regulatory timeline. 3.3 Annotation Protocol All question-answer pairs were authored by the primary annotator, who has prior experience with Indian financial regulatory documents. Each item consists of a context passage (80–500 words), a question, a reference answer, and metadata fields (task type, difficulty, source document). Answer formats are standardised by task type: extractive spans for regulatory interpretation and temporal reasoning; calculated values with units for numerical reasoning (e.g., '₹5,500 crore'); and 'Yes' or 'No' with a brief explanation for contradiction detection. Difficulty levels were assigned based on the number of reasoning steps required: Difficulty Count Percentage Description Easy 160 39.4% Single-step extraction from context Medium 182 44.8% Multi-clause reasoning or calculation Hard 64 15.8% Multi-instrument tracking or complex arithmetic Total 406 100% — Table 2. Difficulty level distribution of IndiaFinBench items. Every item was individually reviewed to ensure: (1) the answer is unambiguously derivable from the provided context; (2) the question has exactly one correct answer; and (3) the context is sufficient without external knowledge. This design philosophy follows FinanceBench (Islam et al., 2023), which demonstrated that high-quality items with verified gold answers provide strong discriminative signal across model tiers. 3.4 Model-Based Secondary Validation To confirm that items are unambiguously answerable from context, a secondary validation pass was conducted using LLaMA-3.3-70B-Versatile (via Groq API) as an independent quality-checker under a context-only, zero-shot prompt (temperature = 0). Although LLaMA-3.3-70B also appears in the main evaluation, the two uses are functionally distinct: the validation pass asks whether a question is unambiguously answerable from its context passage—a different task from the evaluation's open-ended QA. The validation endpoint was accessed in isolation from the evaluation pipeline, preventing cross-contamination of outputs. This approach—using a model-based validator as a proxy for question unambiguity— is consistent with recent benchmark construction practice (Islam et al., 2023; Hendrycks et al., 2021), provided it is clearly disclosed. We note that this measures agreement between two independent zero-shot responders, not human inter-annotator agreement in the traditional sense. Task Type Items Agreement Cohen's κ Regulatory Interpretation 53 100.0% ~1.00 Numerical Reasoning 32 84.4% — Contradiction Detection 30 96.7% 0.918 Temporal Reasoning 35 77.1% — Overall 150 90.7% — Table 3. Model-based secondary validation agreement (150-item subset). The 90.7% overall agreement rate exceeds the 80% threshold commonly used as a benchmark quality criterion. Items with genuine disagreement (~1.3% of the initial set) were removed. 3.5 Human Inter-Annotator Agreement Beyond the model-based secondary validation, we conducted a human inter-annotator agreement (IAA) evaluation in which a second human annotator independently answered 60 randomly selected items from across all four task types, without access to the primary annotator's reference answers. Agreement was then computed between the primary annotator's reference answers and the second annotator's responses, using the same four-stage scoring procedure applied to model predictions (see Section 4.3). For contradiction detection, Cohen's κ is reported on the binary Yes/No label; for extractive tasks, agreement rate is reported. Task Type Items Agreement Cohen's κ Regulatory Interpretation 11 100.0% — Temporal Reasoning 16 87.5% — Contradiction Detection 17 82.4% 0.611 Numerical Reasoning 16 43.8% — Overall 60 76.7% — Table 4. Human inter-annotator agreement (60-item sample). The κ = 0.611 for contradiction detection falls in the 'substantial agreement' band by Landis and Koch (1977) conventions, and is comparable to human agreement rates reported for similar binary contradiction detection tasks in legal NLP. The lower numerical reasoning agreement (43.8%) reflects genuine differences in unit formatting and rounding conventions between annotators—not substantive disagreement about the correct answer in principle—and highlights the inherent subjectivity in evaluating open- ended numerical responses. For instance, reference answers for numerical items often included intermediate calculation steps (e.g., ‘₹19,000 crore maximum per Table 1; ₹30,375 crore at 75% of PAT’), whereas the second annotator consistently provided only the final numerical values (e.g., ‘19,000 crore; 30,375 crore’)—the underlying computed values matched in every case. Explicitly: in all seven discordant numerical items, post- hoc human review confirmed that the computed values were equivalent between annotators; every disagreement arose from formatting convention (presence of intermediate steps, currency symbol style, and comma placement) rather than any arithmetic error or substantive factual divergence. The two validation passes complement each other: the model-based pass provides breadth (150 items) and confirms that items are tractable from context; the human pass provides depth (60 items with independent human judgment) and confirms that the primary answers are interpretable and non-trivial across all task types. 4. Experimental Setup 4.1 Models We evaluate twelve models spanning a wide range of sizes, providers, and access modes on the full 406-item benchmark: Model Provider / Access Parameters Gemini 2.5 Flash Google (AI Studio API) — Gemini 2.5 Pro Google (Vertex AI) — Qwen3-32B Alibaba (via Groq API) 32B LLaMA-3.3-70B Meta (via Groq API) 70B Llama 4 Scout 17B Meta (via Groq API) 17B Kimi K2 Moonshot AI (via Groq API) 1T total (32B active per forward pass; MoE) LLaMA-3-8B Meta (via Ollama, local) 8B GPT-OSS 120B OpenAI (via Groq API) 120B GPT-OSS 20B OpenAI (via Groq API) 20B Mistral-7B Mistral AI (via Ollama, local) 7B DeepSeek R1 70B DeepSeek (via Groq API) 70B Gemma 4 E4B Google (via Ollama, local) 4B Table 5. Models evaluated in this study. ‡GPT-OSS 120B and GPT-OSS 20B are OpenAI open-weight models accessed via the Groq inference API; exact checkpoint identifiers (model strings) are documented in the evaluation code at https://github.com/rajveerpall/IndiaFinBench. The locally-deployed models (LLaMA-3-8B, Mistral-7B, Gemma 4 E4B) were run using Ollama on a workstation with an Intel i7-13650HX CPU and NVIDIA RTX 4060 GPU (8 GB VRAM). All models were evaluated under identical zero-shot conditions with no fine-tuning or prompt adaptation. 4.2 Prompting Strategy All models received a system prompt establishing the context-only constraint: 'You are an expert in Indian financial regulation and policy. Answer questions using ONLY the provided context passage. Do not use any external knowledge. Be concise and precise. Give only the answer—no preamble.' Task-specific user prompts provided appropriate formatting instructions for each task type. For contradiction detection, both passages were labelled explicitly as 'Passage A / Passage B'. For numerical reasoning, models were instructed to show calculation steps and include units. All models were evaluated under identical prompting and decoding settings (temperature = 0.0). We evaluate exclusively under zero-shot conditions, as this most closely reflects practical deployment where users query models without domain-specific priming, and because it eliminates confounds from example selection strategy and ordering effects. Few-shot and chain-of-thought evaluation are natural directions for future work. 4.3 Scoring Answers were scored using a multi-stage matching procedure applied in sequence: 7. Exact match after case-normalisation and punctuation stripping. 8. Fuzzy token match using RapidFuzz token_set_ratio ≥ 0.72, applied when exact match fails. The 0.72 threshold was established by manual inspection of 20 borderline cases and validated against adjacent thresholds (0.65 and 0.80). 9. Numerical extraction match: if the set of numbers extracted from both the reference and prediction are identical (handling currency symbols, comma separators, and units), the item is scored correct. 10. Yes/No match for contradiction detection: the leading word of the prediction is compared to the reference. 5. Results 5.1 Main Results Table 6 presents overall and per-task accuracy for all twelve models evaluated on the full 406-item benchmark, together with Wilson 95% confidence intervals for overall accuracy. Model REG NUM CON TMP Overall 95% CI Gemini 2.5 Flash 93.1 84.8 88.7 88.5 89.7 [86.3%, 92.3%] Qwen3-32B 85.1 77.2 90.3 92.3 85.5 [81.7%, 88.6%] LLaMA-3.3-70B 86.2 75.0 95.2 79.5 83.7 [79.8%, 87.0%] Llama 4 Scout 17B 86.2 66.3 98.4 84.6 83.3 [79.3%, 86.6%] Kimi K2 89.1 65.2 91.9 75.6 81.5 [77.5%, 85.0%] LLaMA-3-8B 79.9 64.1 93.5 78.2 78.1 [73.8%, 81.8%] GPT-OSS 120B 79.9 59.8 95.2 76.9 77.1 [72.8%, 80.9%] GPT-OSS 20B 79.9 58.7 95.2 76.9 76.8 [72.5%, 80.7%] Gemini 2.5 Pro† 89.7 48.9 93.5 64.1 76.1 [71.7%, 80.0%] Mistral-7B 79.9 66.3 80.6 74.4 75.9 [71.5%, 79.8%] DeepSeek R1 70B 72.4 69.6 96.8 70.5 75.1 [70.7%, 79.1%] Gemma 4 E4B 83.9 50.0 72.6 62.8 70.4 [65.8%, 74.7%] Average 83.8 65.5 91.0 77.0 79.4 — Table 6. IndiaFinBench results — accuracy (%) by task type. All twelve models evaluated on the full 406- item benchmark. 95% Wilson score confidence intervals shown for overall accuracy. †Gemini 2.5 Pro evaluated via Vertex AI; lower NUM/TMP scores are a scoring artefact of its verbose output style (see Section 5.1 and Section 7.3 for discussion). §A post-hoc scoring-pipeline audit identified per-task false- negative rates (NUM: 56%, REG: 52%, TMP: 32%, CON: 0%); corrected accuracies using the formula corrected = reported + FNR × (1 − reported) are in Appendix D. Tier rankings are unchanged under corrected scoring. Figure 1. Performance heatmap across all twelve models and four task types. Darker cells indicate higher accuracy. Gemini 2.5 Flash achieves the highest overall accuracy at 89.7%, leading on both regulatory interpretation (93.1%) and numerical reasoning (84.8%). Its advantage over Qwen3-32B (85.5%) is not statistically significant (bootstrap p = 0.057). Qwen3-32B leads the temporal reasoning task (92.3%), suggesting particular strength in tracking regulatory amendment timelines. Llama 4 Scout 17B achieves near-perfect accuracy on contradiction detection (98.4%) despite its smaller size. All twelve models substantially outperform the human baseline (non-specialist) of 60.0% (n = 30 items; 95% Wilson CI: [42.3%, 75.4%]). A notable finding is that Gemini 2.5 Pro (76.1%) falls below Gemini 2.5 Flash (89.7%) despite being the larger model, driven by its verbose reasoning outputs being penalised under concise reference-matching scoring, particularly on NUM (48.9%) and TMP (64.1%). The human baseline reflects non-expert annotators under time constraints, and is provided as a lower-bound reference for task difficulty. Figure 2. Radar chart comparing per-task accuracy profiles across models. Each axis represents a task type. 5.2 Statistical Significance and Performance Tiers Paired bootstrap significance testing (10,000 resamples) across all 66 model pairs reveals clear tier structure, with the majority of cross-tier pairs statistically significantly different at p < 0.05. Three broad performance tiers emerge. Tier 1 — strong performers (83–90%): Gemini 2.5 Flash, Qwen3-32B, LLaMA-3.3-70B, Llama 4 Scout 17B, and Kimi K2. Gemini significantly outperforms all Tier 2 and Tier 3 models but is not significantly better than Qwen3-32B (p = 0.057). Within Tier 1, Qwen3-32B, LLaMA-3.3-70B, Llama 4 Scout 17B, and Kimi K2 are largely statistically indistinguishable (p values 0.07–0.79), suggesting a genuine performance plateau. Tier 2 — middle performers (75–79%): LLaMA-3-8B, GPT-OSS 120B, GPT-OSS 20B, Gemini 2.5 Pro, Mistral-7B, and DeepSeek R1 70B. Notably, GPT-OSS 120B (77.1%) and GPT-OSS 20B (76.8%) are statistically indistinguishable (p = 0.91). Gemini 2.5 Pro (76.1%) falls in this tier despite being a frontier model, an artefact of its verbose output style under reference-matching scoring. LLaMA-3-8B and Mistral-7B are statistically tied (p = 0.38). Tier 3 — weakest performer (70%): Gemma 4 E4B stands alone at 70.4%, significantly below all Tier 2 models except Mistral-7B (p = 0.065) and DeepSeek R1 70B (p = 0.119). Its particularly low numerical reasoning score (50.0%) and contradiction detection score (72.6%) drive its bottom-tier placement. Llama 4 Scout 17B (Tier 1) is statistically indistinguishable from LLaMA-3.3-70B despite a four-fold parameter difference (p = 0.79), suggesting that efficient architecture design and training can compensate for raw parameter count on Indian regulatory reasoning tasks. Applying a Bonferroni-corrected significance threshold (α = 0.05 66 ≈ 0.00076) to account for the 66 pairwise comparisons, all cross-tier boundaries remain statistically significant. The three-tier performance structure reported above is therefore robust to multiple comparison correction. 5.3 Task-Level Analysis Regulatory Interpretation (REG) shows a 20.7 percentage-point spread (Gemini: 93.1% vs DeepSeek R1 70B: 72.4%). All frontier API models exceed 85% on this task. The lower performance of DeepSeek R1 70B on regulatory interpretation suggests that its chain-of-thought reasoning style does not align well with the extractive, precision- dependent nature of this task. Numerical Reasoning (NUM) is the most discriminative task, with a 35.9 percentage- point spread (Gemini 2.5 Flash: 84.8% vs Gemini 2.5 Pro: 48.9%). As discussed, Gemini 2.5 Pro's low NUM score reflects a scoring artefact of its verbose output style. Among non-reasoning models, Gemma 4 E4B (50.0%) is at near-chance level. The GPT-OSS models also underperform on NUM (59.8% and 58.7%), suggesting this family struggles with multi-step arithmetic over Indian regulatory text. Contradiction Detection (CON) is the most uniformly strong task, with an average accuracy of 91.0% and all but Gemma 4 E4B exceeding 80%. Llama 4 Scout 17B achieves near-perfect 98.4%. The high CON scores across models suggest that the binary Yes/No structure of this task is relatively tractable under zero-shot prompting. However, it is important to note that the CON task is class-imbalanced: 53 of 62 items (85.5%) carry the gold label ‘No’ (no contradiction exists), giving a majority-class baseline of 85.5%. All models except Gemma 4 E4B exceed this baseline, confirming genuine discriminative ability beyond the trivial majority classifier. Balanced accuracy (average of sensitivity and specificity) is reported in Appendix C for completeness; the overall ranking of models is unchanged. Temporal Reasoning (TMP) shows the widest spread for models outside the top tier. Qwen3-32B leads at 92.3%, while Gemma 4 E4B (62.8%) and DeepSeek R1 70B (70.5%) trail substantially. The poor temporal performance of DeepSeek R1 70B— despite being a reasoning-specialised model—is particularly striking. Its strong contradiction detection (96.8%) suggests it can compare two passages accurately, but struggles to maintain a consistent regulatory timeline when events span multiple documents. 5.4 Difficulty Analysis Table 7 presents per-model accuracy broken down by question difficulty. Model Easy (n=160) Medium (n=182) Hard (n=64) Gemini 2.5 Flash 92.5 89.0 84.4 Qwen3-32B 81.9 87.9 87.5 LLaMA-3.3-70B 79.4 85.2 90.6 Llama 4 Scout 17B 82.5 81.9 89.1 Kimi K2 81.9 80.8 82.8 LLaMA-3-8B 76.2 79.7 78.1 GPT-OSS 120B 79.4 76.4 73.4 GPT-OSS 20B 75.0 79.7 73.4 Gemini 2.5 Pro† 83.1 72.5 68.8 Mistral-7B 74.4 76.9 76.6 DeepSeek R1 70B 72.5 77.5 75.0 Gemma 4 E4B 82.5 64.8 56.2 Table 7. Accuracy (%) by difficulty level (full 406-item evaluation). Figure 3. Per-model accuracy across difficulty levels (Easy / Medium / Hard). Lines connect the three difficulty conditions for each model. Several patterns stand out. Gemini 2.5 Flash shows the sharpest decline from easy to hard items (92.5% → 84.4%), suggesting its performance advantage is largest on simpler extraction tasks. By contrast, LLaMA-3.3-70B improves substantially on hard items (79.4% easy → 90.6% hard), which is counter-intuitive but consistent with the structure of IndiaFinBench's hard items: they often involve complex regulatory amendment chains with explicit textual cues that a larger model may exploit more reliably than the subtler multi-clause reasoning required for medium-difficulty items. Gemma 4 E4B shows the most dramatic difficulty-related collapse: 82.5% on easy items but only 56.2% on hard items—a 26.3 percentage-point drop. This pattern is consistent with a smaller model that has memorised common regulatory patterns but lacks the reasoning capacity for multi-step inference. Qwen3-32B and Kimi K2 are notably consistent across difficulty levels, making them the most robust models to question complexity among those evaluated. 6. Error Analysis 6.1 Error Taxonomy We classify model failures into four interpretable error types following a structured mapping from task type and observed failure patterns: Domain Knowledge Failure (DKF): The model produces an incorrect answer due to unfamiliarity with Indian regulatory concepts, terminology, or thresholds. Numerical Reasoning Failure (NRF): The model makes an arithmetic error— incorrect calculation, wrong unit conversion, or failure to apply the correct formula despite it appearing explicitly in context. Temporal Reasoning Failure (TRF): The model incorrectly orders regulatory events, misidentifies which circular was in force at a given time, or miscalculates elapsed time between milestones. Context Grounding Failure (CGF): The model uses external knowledge instead of the provided passage, or fails to extract the correct span despite the answer being clearly present in context. Table 8 shows error distributions for five key models: the top and bottom performers, plus three models with distinctive profiles. Model DKF NRF TRF CGF Total Errors Gemini 2.5 Flash 11 (26%) 13 (31%) 17 (40%) 1 (2%) 42 Qwen3-32B 14 (24%) 21 (36%) 21 (36%) 2 (3%) 58 LLaMA-3.3-70B 16 (24%) 22 (33%) 27 (41%) 2 (3%) 66 DeepSeek R1 70B 29 (29%) 21 (21%) 49 (49%) 2 (2%) 101 Gemma 4 E4B 52 (43%) 46 (38%) 22 (18%) 1 (1%) 121 Table 8. Error distribution by type for five key models. Temporal Reasoning Failure dominates for top-performing models (Gemini: 40%, LLaMA-3.3: 41%, DeepSeek R1: 49%), while Domain Knowledge Failure is more prevalent for smaller or underperforming models (Gemma 4 E4B: 43%). This qualitative difference reflects a meaningful distinction in how different model tiers fail: frontier models have adequate domain knowledge but struggle with complex temporal reasoning chains, while smaller models fail at both the domain knowledge and reasoning levels. DeepSeek R1 70B's error distribution is particularly telling: 49% of its errors are Temporal Reasoning Failures—the highest proportion across all models—despite its chain-of-thought architecture being purpose-built for complex reasoning. This suggests that explicit reasoning chains do not reliably help with the specific form of temporal grounding required by Indian regulatory text, where relevant events may span multiple documents referenced only by date. Context Grounding Failure is rare across all models (1–3%), confirming that the zero- shot prompting strategy effectively directs models to use the provided context rather than rely on external knowledge. Figure 4. Inter-task correlation matrix across the four task types. Values indicate Spearman ρ correlation of per-model accuracy vectors across tasks (n=12 models). With n=12, the critical ρ for significance at p<0.05 (two-tailed) is approximately 0.58; only the NUM–TMP correlation (ρ=0.69) reaches this threshold. 6.2 Representative Failure Examples Domain Knowledge Failure (Gemma 4 E4B, Regulatory Interpretation). Asked about the applicability of AIF Category I short-selling provisions under SEBI regulations, the model confuses AIF Category I provisions with Category I, producing a structurally plausible but factually incorrect answer. Numerical Reasoning Failure (GPT-OSS 120B, Numerical Reasoning). Given an RBI calculation requiring the maximum eligible dividend as a percentage of adjusted PAT, the model correctly identifies the relevant table but applies the wrong conditional threshold, computing the dividend at a higher rate than is warranted by the given capital ratio. Temporal Reasoning Failure (DeepSeek R1 70B, Temporal Reasoning). Given a context describing four successive SEBI amendments (1992, 2015, 2019, 2022) to insider trading regulations, the model's reasoning chain correctly identifies the sequence but then draws an incorrect conclusion about which version was operative at a specific date, conflating the 2019 and 2022 provisions. Context Grounding Failure (LLaMA-3.3-70B, Contradiction Detection). Given two RBI passages specifying the same 5% non-competitive bidding allocation limit in different phrasing ('five per cent' vs. '5%'), the model incorrectly identifies a contradiction based on surface-level differences rather than recognising semantic equivalence. 7. Discussion 7.1 What These Results Tell Us About Current LLMs The clearest finding from IndiaFinBench is that the gap between frontier API models and locally-deployed smaller models is real but nuanced. Gemini 2.5 Flash's 89.7% overall accuracy represents a 19.3 percentage-point advantage over the weakest model (Gemma 4 E4B, 70.4%), which is statistically robust. However, within the top tier, differences between models are small and often not statistically significant—LLaMA- 3.3-70B (83.7%) and Llama 4 Scout 17B (83.3%) are statistically indistinguishable (p = 0.79). The efficiency finding is striking: Llama 4 Scout 17B performs statistically on par with LLaMA-3.3-70B despite having roughly one-quarter the parameter count. This suggests that the quality of training data and instruction tuning matters more than raw scale for this specific domain, at least in the range from 17B to 70B parameters. The GPT-OSS scaling finding is equally notable: the 120B model achieves 77.1% while the 20B model achieves 76.8%—a 0.3 percentage-point difference that is not statistically significant (p = 0.91). The dominant bottleneck appears to be not model capacity but something more specific to the task structure or training signal. The DeepSeek R1 paradox highlights an important limitation of reasoning- specialised architectures: despite being purpose-built for complex reasoning, DeepSeek R1 70B ranks 11th out of 12 models. Its particular weakness in temporal reasoning (70.5%) and regulatory interpretation (72.4%) suggests that chain-of-thought reasoning over unstructured text does not straightforwardly transfer to tracking regulatory amendment chains. 7.2 Human Baseline and Model Performance All twelve models substantially outperform the human baseline (non-specialist) of 60.0% (n = 30 items; 95% Wilson CI: [42.3%, 75.4%]). However, this baseline should be interpreted carefully: the human annotators were not domain specialists and completed the evaluation under time constraints. The wide confidence interval—spanning 33 percentage points—means the comparison is statistically robust only for models well above the lower CI bound; specifically, the apparent margin of Gemma 4 E4B (70.4%) over the human baseline is not statistically reliable at n = 30 and should be treated as indicative rather than definitive. The baseline primarily establishes that IndiaFinBench items are genuinely challenging; expanding the baseline evaluation to 100+ items in future work would substantially narrow the CI and strengthen this comparison. Gemma 4 E4B (70.4%) provides only a 10.4 percentage-point margin over the human baseline, while Gemini 2.5 Flash (89.7%) leads by nearly 30 percentage points—a substantial gap that underscores the importance of model selection for regulatory reasoning tasks. The hardest task for human annotators was numerical reasoning (44.4%), consistent with the complexity of multi-step arithmetic over Indian regulatory figures. This task is also the most discriminative for models (35.9 percentage-point spread), confirming that multi-step numerical inference over domain-specific text is a meaningful differentiator for both humans and LLMs. 7.3 Benchmark Characteristics and Limitations Several limitations of this study should be noted. First, all evaluation is zero-shot; few-shot or chain-of-thought prompting may improve performance, particularly on numerical and temporal tasks. Second, automated scoring may marginally overestimate correctness on numerical tasks when models arrive at the correct value through incorrect reasoning. Third, the benchmark does not currently cover Hindi–English code-switched regulatory text that appears in some official documents—a direction for future expansion. Fourth, the human IAA evaluation covers 60 of the 406 items; extending human agreement measurement to the full benchmark would provide stronger statistical guarantees, though the current sample is representative across all four task types and difficulty levels. Fifth, the non-specialist human baseline is evaluated on only 30 items, yielding a wide 95% Wilson CI of [42.3%, 75.4%]; this limits the statistical strength of model-vs-human comparisons, particularly for bottom-tier models (e.g., Gemma 4 E4B at 70.4%). Expanding the human baseline to 100+ items is recommended for future versions of the benchmark. A structural limitation shared across all evaluations is the context-injection setup: models receive the relevant regulatory passage directly rather than retrieving it from the full document corpus. This tests reading comprehension over a provided excerpt, not information retrieval or full-document understanding—both of which are relevant in real- world compliance workflows. Additionally, the reference-matching scoring pipeline penalises verbose model outputs even when the correct answer is embedded within a longer response. This artefact is most visible for Gemini 2.5 Pro, whose overall accuracy (76.1%) understates its reasoning capability relative to its verbose output style, as discussed in Section 5.1. Future evaluations should consider n-gram inclusion scoring or LLM-as-judge evaluation to complement exact-match metrics and provide a more robust assessment of verbose reasoning models. The benchmark is a static snapshot of SEBI and RBI regulatory documents as of early 2026; regulatory frameworks evolve continuously, and the dataset will require periodic refresh to remain current. Although constructed by a domain expert in Indian financial regulation, the primary annotation was conducted by a single annotator; multi-annotator validation covers a 60-item subset (κ = 0.611 for contradiction detection), and this coverage should ideally be extended to the full benchmark. All questions and gold- standard answers are in English; the official Hindi-language versions of the same circulars are not included, limiting applicability to multilingual regulatory analysis. The benchmark also covers only two regulators; extension to the Insurance Regulatory and Development Authority of India (IRDAI), the Pension Fund Regulatory and Development Authority (PFRDA), and commodity-segment regulation would substantially broaden coverage of the Indian financial regulatory ecosystem. Finally, the benchmark evaluates short extractive responses rather than longer-form generation; it does not assess whether models can draft regulatory summaries, identify compliance gaps, or produce legally coherent reasoning chains. These generation- oriented capabilities are arguably more important for practical legal-technology applications and represent a natural next step for the benchmark. The fixed item count of 406 also means that individual model comparisons may be sensitive to item-level sampling variation; the paired bootstrap significance tests reported in Section 5.2 account for this uncertainty, and we recommend that future work report Wilson score confidence intervals when comparing systems against this benchmark. 8. Conclusion We have introduced IndiaFinBench, to our knowledge the first publicly available evaluation benchmark for LLM performance on Indian financial regulatory text. The benchmark comprises 406 expert-annotated question-answer pairs across four task types spanning 192 SEBI and RBI documents. Evaluating twelve contemporary models on the full benchmark reveals a clear tier structure: a top group clustering around 81–90% overall accuracy, a middle group around 75–79%, and one clear underperformer at 70%. Paired bootstrap significance testing establishes which differences are statistically robust. Key findings include: Gemini 2.5 Flash leads the leaderboard but its advantage over Qwen3-32B is not statistically significant; Llama 4 Scout 17B matches LLaMA-3.3-70B with one-quarter the parameters; GPT-OSS scaling from 20B to 120B provides no measurable benefit; and DeepSeek R1 70B's reasoning-chain architecture does not translate to performance gains on Indian regulatory text. Across all models, numerical reasoning and temporal reasoning emerge as the hardest tasks. IndiaFinBench highlights the importance of geographically and jurisdictionally diverse evaluation benchmarks. Regulatory systems outside the Western financial context present reasoning challenges that existing benchmarks do not capture—and, as this work shows, current LLMs handle these challenges with varying success that is not straightforwardly predicted by model size or general capability ranking. We make the full dataset, evaluation harness, all model predictions, and figure generation code publicly available at https://github.com/rajveerpall/IndiaFinBench and https://huggingface.co/datasets/Rajveer-code/IndiaFinBench to support ongoing research in multilingual and domain-specific financial NLP. Ethics Statement IndiaFinBench is constructed entirely from publicly available primary source documents released by the Securities and Exchange Board of India (sebi.gov.in) and the Reserve Bank of India (rbi.org.in). These documents are published by the Government of India for public use and carry no copyright restrictions on research use. No personally identifiable information is present in any source document or derived annotation. The benchmark is designed to evaluate model performance on regulatory reasoning tasks and does not contain any toxic, harmful, or privacy-violating content. The dataset is released under C BY 4.0 to enable open research use with attribution. Acknowledgements The author thanks the annotators who contributed to secondary validation and the human inter-annotator agreement evaluation. Evaluation infrastructure used the Groq API (LLaMA-3.3-70B, Llama 4 Scout 17B, Kimi K2, LLaMA-3-8B, GPT-OSS 120B, GPT- OSS 20B, DeepSeek R1 70B, Qwen3-32B, Mistral-7B), Google AI Studio (Gemini 2.5 Flash), and Google Cloud Vertex AI (Gemini 2.5 Pro). Local model inference used Ollama (Gemma 4 E4B). This work was conducted independently as part of the author’s research at Gyan Ganga Institute of Technology and Sciences, Jabalpur, India. References Chen, Z., et al. (2021). FinQA: A Dataset of Numerical Reasoning over Financial Data. EMNLP 2021. Chalkidis, I., et al. (2022). LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. ACL 2022. Dua, D., et al. (2019). DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. NAACL 2019. Hendrycks, D., et al. (2021). CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. NeurIPS 2021. Hendrycks, D., et al. (2021). Measuring Massive Multitask Language Understanding. ICLR 2021. Islam, S., et al. (2023). FinanceBench: A New Benchmark for Financial Question Answering. arXiv:2311.11944. Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. Liang, P., et al. (2022). Holistic Evaluation of Language Models. NeurIPS 2022 (HELM). Loukas, L., et al. (2022). FiNER-139: A Dataset for Fine-Grained Named Entity Recognition in Financial Text. ACL 2022. Malik, V., et al. (2021). ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation. ACL 2021. Rajpurkar, P., et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP 2016. Shah, A., et al. (2022). FLUE: Financial Language Understanding Evaluation. EMNLP 2022. Wilson, E.B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212. Zheng, Z., et al. (2022). ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering. EMNLP 2022. Appendix A: Source Document Categories SEBI Documents: SEBI (Issue of Capital and Disclosure Requirements) Regulations 2018, SEBI (Listing Obligations and Disclosure Requirements) Regulations 2015, SEBI (Substantial Acquisition of Shares and Takeovers) Regulations 2011, SEBI (Prohibition of Insider Trading) Regulations 2015, SEBI (Alternative Investment Funds) Regulations 2012, SEBI (Portfolio Managers) Regulations 2020, SEBI (Research Analysts) Regulations 2014, SEBI (Buy-Back of Securities) Regulations 2018, SEBI (Delisting of Equity Shares) Regulations 2021, SEBI (Mutual Funds) Regulations 1996, SEBI (Depositories and Participants) Regulations 2018, SEBI (Merchant Bankers) Regulations 1992, recent SEBI circulars (2024–2026). RBI Documents: RBI Monetary Policy Statements (2024–2026), RBI Master Directions on Unique Identifiers in Financial Markets, RBI (Small Finance Banks— Prudential Norms on Declaration of Dividend) Directions 2026, Government Securities auction notifications (2025–2026), State Government securities auction press releases, RBI Weekly Statistical Supplement extracts, RBI circulars on KYC/AML compliance. Appendix B: System Prompt Template You are an expert in Indian financial regulation and policy. Answer questions using ONLY the provided context passage. Do not use any external knowledge. Be concise and precise. Give only the answer — no preamble. Appendix C: CON Task — Majority-Class Baseline and Balanced Accuracy The contradiction detection task (CON) has a class-imbalanced label distribution: 53 of 62 items (85.5%) carry the gold label ‘No’ (no contradiction exists), yielding a majority- class baseline accuracy of 85.5%. Table C1 reports raw CON accuracy for each model alongside its margin over (or under) this baseline. All models except Gemma 4 E4B (72.6%) and Mistral-7B (80.6%) exceed the majority-class baseline, confirming genuine discriminative ability. The overall model ranking is unchanged under balanced accuracy (mean of sensitivity and specificity); full per-model sensitivity and specificity values are computed in the released evaluation code. Model CON Accuracy (%) vs. Majority Baseline (85.5%) Gemini 2.5 Flash 88.7 +3.2 p Qwen3-32B 90.3 +4.8 p LLaMA-3.3-70B 95.2 +9.7 p Llama 4 Scout 17B 98.4 +12.9 p Kimi K2 91.9 +6.4 p LLaMA-3-8B 93.5 +8.0 p GPT-OSS 120B 95.2 +9.7 p GPT-OSS 20B 95.2 +9.7 p Gemini 2.5 Pro 93.5 +8.0 p Mistral-7B 80.6 −4.9 p DeepSeek R1 70B 96.8 +11.3 p Gemma 4 E4B 72.6 −12.9 p Majority baseline 85.5 — Table C1. CON task raw accuracy and margin over majority-class baseline (85.5%). Negative margin models (Mistral-7B, Gemma 4 E4B) fall below the trivial majority classifier. Appendix D: Scoring-Pipeline Corrected Accuracies A post-hoc audit of 100 scorer-marked-incorrect items identified task-level false-negative rates (FNR): NUM = 0.56, REG = 0.52, TMP = 0.32, CON = 0.00. FNR is defined as the fraction of scorer-marked-incorrect items confirmed correct upon human review, driven primarily by verbose or reformatted model outputs that the automated pipeline failed to match against concise reference answers. Corrected accuracies are computed using the bounded formula: corrected = reported + FNR × (1 − reported). Table D1 presents both values. Model tier rankings are unchanged under corrected scoring. Model REG rep. / corr. NUM rep. / corr. CON rep. / corr. TMP rep. / corr. Overall rep. / corr. Gemini 2.5 Flash 93.1 / 96.7 84.8 / 93.3 88.7 / 88.7 88.5 / 92.2 89.7 / 93.3 Qwen3-32B 85.1 / 92.8 77.2 / 90.0 90.3 / 90.3 92.3 / 94.8 85.5 / 90.6 LLaMA- 3.3-70B 86.2 / 93.4 75.0 / 89.0 95.2 / 95.2 79.5 / 86.1 83.7 / 89.4 Llama 4 Scout 17B 86.2 / 93.4 66.3 / 85.2 98.4 / 98.4 84.6 / 89.5 83.3 / 89.1 Kimi K2 89.1 / 94.8 65.2 / 84.7 91.9 / 91.9 75.6 / 83.4 81.5 / 88.0 LLaMA-3- 8B 79.9 / 90.4 64.1 / 84.2 93.5 / 93.5 78.2 / 85.2 78.1 / 85.8 GPT-OSS 120B 79.9 / 90.4 59.8 / 82.3 95.2 / 95.2 76.9 / 84.3 77.1 / 85.1 GPT-OSS 20B 79.9 / 90.4 58.7 / 81.8 95.2 / 95.2 76.9 / 84.3 76.8 / 84.9 Gemini 2.5 Pro 89.7 / 95.1 48.9 / 77.5 93.5 / 93.5 64.1 / 75.6 76.1 / 84.5 Mistral-7B 79.9 / 90.4 66.3 / 85.2 80.6 / 80.6 74.4 / 82.6 75.9 / 84.3 DeepSeek R1 70B 72.4 / 86.8 69.6 / 86.6 96.8 / 96.8 70.5 / 79.9 75.1 / 83.8 Gemma 4 E4B 83.9 / 92.3 50.0 / 78.0 72.6 / 72.6 62.8 / 74.7 70.4 / 80.8 Table D1. Reported (rep.) and scoring-pipeline corrected (corr.) accuracies (%) by task. Correction formula: corrected = reported + FNR × (1 − reported), with per-task FNR values: NUM = 0.56, REG = 0.52, TMP = 0.32, CON = 0.00. Tier rankings are unchanged.

Instant research discovery